seq2seq model with transformer

[ghost]

Hi I am urgently looking for a sequence to sequence model with transformer with script to finetuning and training, I appreciate telling me which of the implementations in this repo could do a sequence to sequence model? thanks Best regards Julia

[LysandreJik]

Are you looking for LSTM/RNN-based seq2seq architectures or Transformer-based architectures? This repository does not host any LSTM/RNN architectures.

You can find information on all our (transformer) models here, and examples using them here.

[ghost]

I am looking for transformer based, pretrained model, I am not sure which of the implemented models in this repo I can use for seq2seq model? thanks for your help

[LysandreJik]

The models hosted on this repo unfortunately probably cannot be used in a traditional sequence-to-sequence manner like translation (if that's what you have in mind).

[ghost]

yes, exactly, I am looking for such models, even gpt model cannot be used for this purpose? or gpt2 by conditioning? Are you aware of clean implementation for seq2seq model with any of these pretrained models hosted in your repo? thanks.

On Thu, Aug 8, 2019 at 5:10 PM Lysandre Debut notifications@github.com wrote:

The models hosted on this repo unfortunately probably cannot be used in a traditional sequence-to-sequence manner like translation (if that's what you have in mind).

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/huggingface/pytorch-transformers/issues/988?email_source=notifications&email_token=AM3GZM7GUYVJHJ6YKKQK3PDQDQZM5A5CNFSM4IKIFMC2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD335YCY#issuecomment-519560203, or mute the thread https://github.com/notifications/unsubscribe-auth/AM3GZM2PJO6J63NC4DPTFW3QDQZM5ANCNFSM4IKIFMCQ .

[julien-c]

Hi @juliahane, maybe take a look at fairseq

[ghost]

Hi Thanks, Do you mind also suggest me a good implementation with lstm for seq2seq model, I need some implementation with high quality of decoding, thanks.

On Thu, Aug 8, 2019 at 6:52 PM Julien Chaumond notifications@github.com wrote:

Hi @juliahane https://github.com/juliahane, maybe take a look at fairseq

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/huggingface/pytorch-transformers/issues/988?email_source=notifications&email_token=AM3GZMZG265MHPY53WI2HNTQDRFLDA5CNFSM4IKIFMC2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD34HSYQ#issuecomment-519600482, or mute the thread https://github.com/notifications/unsubscribe-auth/AM3GZMYF5YYWWJMDVOXH3LTQDRFLDANCNFSM4IKIFMCQ .

[ghost]

Hi I found FairSeq implementation not really clean and modular code. Are you aware of more work which extend BERT, GPT, ... to a language model with decoder? thanks Julia

On Thu, Aug 8, 2019 at 9:07 PM julia hane juliahane123@gmail.com wrote:

Hi Thanks, Do you mind also suggest me a good implementation with lstm for seq2seq model, I need some implementation with high quality of decoding, thanks.

On Thu, Aug 8, 2019 at 6:52 PM Julien Chaumond notifications@github.com wrote:

Hi @juliahane https://github.com/juliahane, maybe take a look at fairseq

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/huggingface/pytorch-transformers/issues/988?email_source=notifications&email_token=AM3GZMZG265MHPY53WI2HNTQDRFLDA5CNFSM4IKIFMC2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD34HSYQ#issuecomment-519600482, or mute the thread https://github.com/notifications/unsubscribe-auth/AM3GZMYF5YYWWJMDVOXH3LTQDRFLDANCNFSM4IKIFMCQ .

[stefan-it]

Then you should have a look at the "Cross-lingual Language Model Pretraining" from Lample and Conneau: https://arxiv.org/abs/1901.07291

Implementation of supervised and unsupervised NMT can be found here: https://github.com/facebookresearch/XLM#iii-applications-supervised--unsupervised-mt :)

[ghost]

Hi thanks a lot. I was wondering if you could also suggest me a good implementation for seq2seq with LSTMs in pytorch with good accuracy. I have a deadline and I cannot find any, I really appreciate your help. thanks Julia

On Thu, Aug 8, 2019 at 11:41 PM Stefan Schweter notifications@github.com wrote:

Then you should have a look at the "Cross-lingual Language Model Pretraining" from Lample and Conneau: https://arxiv.org/abs/1901.07291

Implementation of supervised and unsupervised NMT can be found here: https://github.com/facebookresearch/XLM#iii-applications-supervised--unsupervised-mt :)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/huggingface/pytorch-transformers/issues/988?email_source=notifications&email_token=AM3GZM3URUZ6GWYI7A4WPFTQDSHKBA5CNFSM4IKIFMC2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD347NXQ#issuecomment-519698142, or mute the thread https://github.com/notifications/unsubscribe-auth/AM3GZM2PRHPLQBS7XBGCUBLQDSHKBANCNFSM4IKIFMCQ .

[LysandreJik]

Hey Julia, without a specific task in mind I can't think of anything relevant, but browsing paperswithcode.com with a seq2seq search yields quite a few interesting results.

[ghost]

Hi My task is a autoencoding text. So encoding and decoding it in one language. Thanks

On Wed, Aug 14, 2019, 5:18 PM Lysandre Debut notifications@github.com wrote:

Hey Julia, without a specific task in mind I can't think of anything relevant, but browsing paperswithcode.com with a seq2seq search https://paperswithcode.com/search?q_meta=&q=seq2seq yields quite a few interesting results.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/huggingface/pytorch-transformers/issues/988?email_source=notifications&email_token=AM3GZMZ3PDZUSW7QWNQV3CDQEQO2XA5CNFSM4IKIFMC2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4JEKTA#issuecomment-521291084, or mute the thread https://github.com/notifications/unsubscribe-auth/AM3GZM62USCZO4Z5UGUTOBTQEQO2XANCNFSM4IKIFMCQ .

[ghost]

I was wondering if you could tell me which of these are a fast sequence to sequence implementation, this is really hard for me to figure out which one to use. thanks

On Wed, Aug 14, 2019 at 5:19 PM julia hane juliahane123@gmail.com wrote:

Hi My task is a autoencoding text. So encoding and decoding it in one language. Thanks

On Wed, Aug 14, 2019, 5:18 PM Lysandre Debut notifications@github.com wrote:

Hey Julia, without a specific task in mind I can't think of anything relevant, but browsing paperswithcode.com with a seq2seq search https://paperswithcode.com/search?q_meta=&q=seq2seq yields quite a few interesting results.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/huggingface/pytorch-transformers/issues/988?email_source=notifications&email_token=AM3GZMZ3PDZUSW7QWNQV3CDQEQO2XA5CNFSM4IKIFMC2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4JEKTA#issuecomment-521291084, or mute the thread https://github.com/notifications/unsubscribe-auth/AM3GZM62USCZO4Z5UGUTOBTQEQO2XANCNFSM4IKIFMCQ .

[ghost]

I did checked this implementations you sent me, I honestly cannot find a single good seq2seq one with lstm, and I really appreciate your help

On Wed, Aug 14, 2019 at 5:39 PM julia hane juliahane123@gmail.com wrote:

I was wondering if you could tell me which of these are a fast sequence to sequence implementation, this is really hard for me to figure out which one to use. thanks

On Wed, Aug 14, 2019 at 5:19 PM julia hane juliahane123@gmail.com wrote:

Hi My task is a autoencoding text. So encoding and decoding it in one language. Thanks

On Wed, Aug 14, 2019, 5:18 PM Lysandre Debut notifications@github.com wrote:

Hey Julia, without a specific task in mind I can't think of anything relevant, but browsing paperswithcode.com with a seq2seq search https://paperswithcode.com/search?q_meta=&q=seq2seq yields quite a few interesting results.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/huggingface/pytorch-transformers/issues/988?email_source=notifications&email_token=AM3GZMZ3PDZUSW7QWNQV3CDQEQO2XA5CNFSM4IKIFMC2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4JEKTA#issuecomment-521291084, or mute the thread https://github.com/notifications/unsubscribe-auth/AM3GZM62USCZO4Z5UGUTOBTQEQO2XANCNFSM4IKIFMCQ .

[stefan-it]

@juliahane fairseq has an example of how to use a LSTM (encoder & decoder) for a seq2seq model:

https://fairseq.readthedocs.io/en/latest/tutorial_simple_lstm.html

Additionally, you could also check out Joey NMT, which has a very nice and clear codebase:

https://github.com/joeynmt/joeynmt

[ghost]

Hi Thanks, Fairseq to me is not following a good coding practice although Facebook has published it, but the second one looks much better, thank you.

I was wondering if you could tell me if torchtext is faster than using dataloader in pytorch for seq2seq applications? I wonder how torchtext impact the speed and if this is really better than dataloader

thanks

On Fri, Aug 16, 2019 at 1:42 PM Stefan Schweter notifications@github.com wrote:

@juliahane https://github.com/juliahane fairseq has an example of how to use a LSTM (encoder & decoder) for a seq2seq model:

https://fairseq.readthedocs.io/en/latest/tutorial_simple_lstm.html

Additionally, you could also check out Joey NMT, which has a very nice and clear codebase:

https://github.com/joeynmt/joeynmt

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/huggingface/pytorch-transformers/issues/988?email_source=notifications&email_token=AM3GZMZGVNRAT6FJLOEZAB3QE2HB3A5CNFSM4IKIFMC2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4OMY7A#issuecomment-521981052, or mute the thread https://github.com/notifications/unsubscribe-auth/AM3GZM3YNYMZ32LXZCCPSA3QE2HB3ANCNFSM4IKIFMCQ .

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[thomwolf]

Merging with #1506