Currently the repo is focused on control tasks. This is an issue so we can work on including pure text based task and use autoregressive pre-training on the same.
This likely means :
Extending the code to work with text (TextTokenizer, Prediction, etc) [Note GPT-2 is already existing so we don't have to write the model]
Using the current workflow of the code alongwith conventions
Make the code compatible to train both control and text tasks together by merging the data batches
Currently the repo is focused on control tasks. This is an issue so we can work on including pure text based task and use autoregressive pre-training on the same.
This likely means :
As for how this could be done, I wrote a doc couple of weeks back explaining different parts of the repo and how and what are the places where code changes are to be made : https://docs.google.com/document/d/16rNVqRucDMz4yqiYJz8NZFPHMGvzO0emb0KkxjAlfXo/edit#heading=h.huk7442i8c1n