Closed SaraBakic closed 2 years ago
Hi, I notice that this PR contains a lot of files for data pre-processing. These files already exist in the GPT folder. Can you remove these files and state that the user should use a symbolic link in your README.md
.
Hi, yes, the preprocessing structure is the same but it is adapted for T5 (everything related to tokenization is adapted to T5 rather than GPT). Some of the files are identical to ones in GPT folder (e.g. in tools/download). Should I only remove the folders that contain all files that already exist in the GPT folder and leave those that have something new due to dependencies or just remove all files that are identical to GPT?
Hi, yes, the preprocessing structure is the same but it is adapted for T5 (everything related to tokenization is adapted to T5 rather than GPT). Some of the files are identical to ones in GPT folder (e.g. in tools/download). Should I only remove the folders that contain all files that already exist in the GPT folder and leave those that have something new due to dependencies or just remove all files that are identical to GPT?
Hey, I noticed your great job on T5 example. You can just reuse the GPT code as much as possible.
Also, can you provide train logs with different parallelism strategies?
This PR is closed due to inactivity.
A pipeline for training T5 for a language modeling objective is created. It is done following the GPT pipeline from the examples