Tokenizers inconsistency

exo-explore / exo

Run your own AI cluster at home with everyday devices 📱💻 🖥️⌚

GNU General Public License v3.0

6.56k stars 342 forks source link

Closed AlexCheema closed 3 weeks ago

AlexCheema commented 3 weeks ago

see test ./test/test_tokenizers.py we explicitly ignore some tokenisers because they fail the test. this needs to be fixed.