Fix various small problems - Githubissues

bigscience-workshop / Megatron-DeepSpeed

Ongoing research training transformer language models at scale, including: BERT & GPT-2

Other

1.3k stars 211 forks source link

Fix various small problems #367

Open janEbert opened 1 year ago

janEbert commented 1 year ago

Problems:

BERT/T5 dataset handles already corrupted indices incorrectly.
GPT tokenizer vocab size does not include special tokens.
In Eval Harness evaluation, the last token of the target is unconditionally truncated. This results in wrong evaluation metrics and is actually not such a small problem.

This PR fixes these issues. Since the changes are so small, I didn't bother creating 3 PRs.