search

helpmefindaname / transformer-smaller-training-vocab

Temporary remove unused tokens during training to save ram and speed.
https://helpmefindaname.github.io/transformer-smaller-training-vocab/
MIT License
20 stars 2 forks source link

Python 3.12 Support #12

Closed b3n4kh closed 3 months ago

b3n4kh commented 3 months ago

With the currently pinned sentencepiece version, this project can not be run on python 3.12

https://github.com/helpmefindaname/transformer-smaller-training-vocab/blob/90a2d498559505dd3e73a35f3fa1620792f82827/pyproject.toml#L15C1

helpmefindaname commented 3 months ago

Hi @b3n4kh and thank you for your report.

I have updated the dependencies and the testing to ensure python 3.12 compatibility. Version 0.4.0 now has active python 3.12 support