OpenNMT / Tokenizer

Fast and customizable text tokenization library with BPE and SentencePiece support
https://opennmt.net/
MIT License
276 stars 69 forks source link

pyonmttok installation fails while installing OpenNMT-py with python 3.12 #329

Open ramanirudh opened 3 months ago

ramanirudh commented 3 months ago

Had difficulty installing pyonmttok on Redhat 9.4 with python 3.12, while all other dependencies have been resolved smoothly. The error goes like (similar to https://github.com/OpenNMT/OpenNMT-py/issues/1770 ): No local packages or working download links found for pyonmttok<2,>=1.37 . error: Could not find suitable distribution for Requirement.parse('pyonmttok').

Figured out hard way that (https://pypi.org/simple/pyonmttok/) does not have the required .whl file for python 3.12. Installation was successful with python 3.11.

This is to request you to mention in Readme.md the requirement of python<3.12

royshil commented 2 months ago

no whl but you can build from source

roughly

$ cmake .. -DICU_ROOT="..." -DCMAKE_INSTALL_PREFIX="./dist" # best use abs paths
$ cmake --build . --config Release
$ cmake --install .
$ cd ../bindings/python
$ pip install wheel setuptools --upgrade # for good measure
$ $env:TOKENIZER_ROOT="../../build/dist" # use abs path here
$ pip install .

that'll pip install for 3.12