search

lopuhin / transformer-lm

Transformer language model (GPT-2) with sentencepiece tokenizer
164 stars 47 forks source link

Plans for transformer-xl? #13

Closed gooofy closed 5 years ago

gooofy commented 5 years ago

You seem to have done quite some work on sentencepiece encoding and inference in your transformer-xl repo. As I have been experimenting with that model lately I might be able to contribute to that effort - so I was wondering what your long term plans are here: are you planning to keep transformer-xl separate or add it to this repo?

lopuhin commented 5 years ago

@gooofy transformer-xl didn't work that well for me for text generation, and for language modeling I was more interested in short texts (one sentence), and I don't have plans to add it here. Also maybe the best long-term solution would be to use https://github.com/huggingface/pytorch-transformers as now they have implementations of all major models, and it's only a matter of changing how tokenization is done (and making sure they have correct initialization for training from scratch).

gooofy commented 5 years ago

ah, pytorch-transformers, of course! - thanks for the quick reply, I will look into their implemenation then :)