Fix tokenization freeze

AlignmentResearch / tuned-lens

Tools for understanding how transformer predictions are built layer-by-layer

https://tuned-lens.readthedocs.io/en/latest/

MIT License

432 stars 47 forks source link

Closed levmckinney closed 1 year ago

levmckinney commented 1 year ago

When running the eval or train commands on main the program appears to freeze

python3 -m tuned_lens eval EleutherAI/pythia-410m-deduped-v0 test.jsonl --lens pythia-410m-deduped-v0

I believe this is some kind of deadlock, but I'm not completely sure.