Fix unnatural tokenizations if possible

OpenGPTX / lm-evaluation-harness

A framework for few-shot evaluation of autoregressive language models.

MIT License

8 stars 8 forks source link

Open KlaudiaTH opened 10 months ago

KlaudiaTH commented 10 months ago

This generically fixes unnatural tokenizations where it is possible and makes the is_continuation flag obsolete.