Closed thfrkielikone closed 1 month ago
svirpioj
commented
2 months ago This does not look straightforward - possible the issue with M-series Macs is similar? I tried for some time to get something working for that, but did not succeed. So any help is gladly taken.
And in a wider sense: would it be possible to have a pre-built wheel for this package to make installation even easier?
There is a number of wheels available at PyPI, but I guess they are not working in your platform then?
svirpioj
commented
2 months ago at import time opus-fast-mosestokenizer fails to find its dll / so files. This may be a failure caused by my hacky way of compiling the package.
A quick comment on this: I found that the problem seems to be in all systems when the package is installed from source (pip install --no-binary :all: opus-fast-mosestokenizer). When installed from a wheel, the library is found.
I couldn't so far find a solution for this (other than setting LD_LIBRARY_PATH).
thfrkielikone
commented
2 months ago My platform is a mac, but it's running an x86_64 fedora image and the hardware isn't arm-based either: I don't think there not being a wheel is related to that. The version of Python it is running is 3.12, for which there is no wheel (at a glance): is that the reason it decides to build from source? The error does look very similar to the one in the issue you linked.
svirpioj
commented
2 months ago The version of Python it is running is 3.12, for which there is no wheel (at a glance): is that the reason it decides to build from source?
Yes, there was indeed no Python 3.12 wheels before, which forced installing from source. I added them now to a new version 0.0.8.6. I think it's working now?
thfrkielikone
commented
2 months ago This looks like its working now, thank you very much for your time.
docker run -it fedora:40
[root@57390d4cfe46 /]# python3 -m ensurepip
Looking in links: /tmp/tmpw85dicm7
Processing /tmp/tmpw85dicm7/pip-23.3.2-py3-none-any.whl
Installing collected packages: pip
Successfully installed pip-23.3.2
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
[root@57390d4cfe46 /]# pip3 install opus-fast-mosestokenizer
Collecting opus-fast-mosestokenizer
Downloading opus_fast_mosestokenizer-0.0.8.6-cp312-cp312-manylinux1_x86_64.whl.metadata (3.6 kB)
Downloading opus_fast_mosestokenizer-0.0.8.6-cp312-cp312-manylinux1_x86_64.whl (838 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 838.3/838.3 kB 2.2 MB/s eta 0:00:00
Installing collected packages: opus-fast-mosestokenizer
Successfully installed opus-fast-mosestokenizer-0.0.8.6
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
[notice] A new release of pip is available: 23.3.2 -> 24.1.1
[notice] To update, run: python3 -m pip install --upgrade pip
[root@57390d4cfe46 /]# python3
Python 3.12.2 (main, Feb 21 2024, 00:00:00) [GCC 14.0.1 20240217 (Red Hat 14.0.1-0)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import mosestokenizer
>>> m = mosestokenizer.MosesTokenizer()
>>> m.tokenize("Kissa kävelee puussa")
['Kissa', 'kävelee', 'puussa']
>>>(Building from source is still similarly broken: no surprises there.)
svirpioj
commented
1 month ago I found a hack to make also the source installs work. As far as I was able to understand, the problem was that source install uses shared libraries (which makes sense), and then the main lib (libmosestokenizer-dev.so) wasn't found by the python interface lib (_mosestokenizer.<xyz>.so) unless it was in LD_LIBRARY_PATH. Loading the main library with ctypes.cdll.LoadLibrary beforehand did the trick. Closing this now.
If I run the following to try to install opus-fast-mosestokenizer, I get various errors.
The pybind11 python package doesn't get correctly pulled in. I get this error:
After installing pybind with
pip3 install pybind11, I still get the same error. I can get around it by specifying where cmake should look for pybind, but it feels really hacky:At this point, opus-fast-mosestokenizer installs. Running it is still problematic:
If I add it to LD_LIBRARY_PATH, it runs:
My issue here is then:
And in a wider sense: would it be possible to have a pre-built wheel for this package to make installation even easier?