matt-peters
commented
2 years ago Hello, we have a complete working end-to-end example of training a summarization model included as part of our recent NAACL 2021 tutorial on long document NLP: https://github.com/allenai/naacl2021-longdoc-tutorial
We don't have a simple example that uses a model without pretraining, but it should be easy to adapt the code that does use a pretrained model as the process would be the same, except for loading the pretrained checkpoint weights.
Hello Is there a simple example (not from pretrained models) that shows how to just define enc and dec for seq2seq learning in the longformer? The inputs can be also just synthetic and random like
x=torch.rand(1,128,100)? I am specifically looking for an example like the one that pytorch has for its transformers encoder decoder, to show me how can I separately define encoder and decoder. thanks