search

google / sentencepiece

Unsupervised text tokenizer for Neural Network-based text generation.
Apache License 2.0
10.28k stars 1.18k forks source link

Python from source on armv7l raises ' undefined symbol: __atomic_fetch_add_8 ' #865

Open FrancescoScandiffio opened 1 year ago

FrancescoScandiffio commented 1 year ago
Device Raspberry Pi 3B revision 2
Architecture armv7l
OS Raspberry OS Lite (32 bit)
Python 3.9.2
GCC 10.2.1
Error ImportError at _sentencepiece.cpython-39-arm-linux-gnueabihf.so: undefined symbol: __atomic_fetch_add_8

Hello,

I would like to use the sentencepiece python package on a Raspberry Pi 3B with 32 bit Raspi OS but I am experiencing some difficulties with the import. The available wheels are not compatible with my setup so I followed your guide to build the package from source (at https://github.com/google/sentencepiece/blob/master/python/README.md). The installation ends successfully, but when I try to import it raises an ImportError:

Python 3.9.2 (default, Mar 12 2021, 04:06:34) 
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sentencepiece
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/pi/.local/lib/python3.9/site-packages/sentencepiece/__init__.py", line 13, in <module>
    from . import _sentencepiece
ImportError: /home/pi/.local/lib/python3.9/site-packages/sentencepiece/_sentencepiece.cpython-39-arm-linux-gnueabihf.so: undefined symbol: __atomic_fetch_add_8

Can you help me with this issue? Should I manually link some library or add specific flags to the cmake?

Thank you in advance for any assistance you can provide.

Log, from the end of make install

 Install the project...
-- Install configuration: ""
-- Installing: /home/pi/environments/sentencepiece/build/root/lib/pkgconfig/sentencepiece.pc
-- Installing: /home/pi/environments/sentencepiece/build/root/lib/libsentencepiece.a
-- Installing: /home/pi/environments/sentencepiece/build/root/lib/libsentencepiece_train.a
-- Installing: /home/pi/environments/sentencepiece/build/root/bin/spm_encode
-- Installing: /home/pi/environments/sentencepiece/build/root/bin/spm_decode
-- Installing: /home/pi/environments/sentencepiece/build/root/bin/spm_normalize
-- Installing: /home/pi/environments/sentencepiece/build/root/bin/spm_train
-- Installing: /home/pi/environments/sentencepiece/build/root/bin/spm_export_vocab
-- Installing: /home/pi/environments/sentencepiece/build/root/include/sentencepiece_trainer.h
-- Installing: /home/pi/environments/sentencepiece/build/root/include/sentencepiece_processor.h

/usr/lib/python3/dist-packages/setuptools/dist.py:461: UserWarning: Normalizing '0.2.00' to '0.2.0'
  warnings.warn(tmpl.format(**locals()))
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-armv7l-3.9
creating build/lib.linux-armv7l-3.9/sentencepiece
copying src/sentencepiece/__init__.py -> build/lib.linux-armv7l-3.9/sentencepiece
copying src/sentencepiece/_version.py -> build/lib.linux-armv7l-3.9/sentencepiece
copying src/sentencepiece/sentencepiece_model_pb2.py -> build/lib.linux-armv7l-3.9/sentencepiece
copying src/sentencepiece/sentencepiece_pb2.py -> build/lib.linux-armv7l-3.9/sentencepiece
running build_ext
## cflags=-std=c++17 -I../build/root/include -Wl,-strip-all
## libs=../build/root/lib/libsentencepiece.a ../build/root/lib/libsentencepiece_train.a -Wl,-strip-all
building 'sentencepiece._sentencepiece' extension
creating build/temp.linux-armv7l-3.9
creating build/temp.linux-armv7l-3.9/src
creating build/temp.linux-armv7l-3.9/src/sentencepiece
arm-linux-gnueabihf-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -ffile-prefix-map=/python3.9-3.9.2=. -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -ffile-prefix-map=/python3.9-3.9.2=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.9 -c src/sentencepiece/sentencepiece_wrap.cxx -o build/temp.linux-armv7l-3.9/src/sentencepiece/sentencepiece_wrap.o -std=c++17 -I../build/root/include -Wl,-strip-all
arm-linux-gnueabihf-g++ -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fwrapv -O2 -Wl,-z,relro -g -fwrapv -O2 -g -ffile-prefix-map=/python3.9-3.9.2=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-armv7l-3.9/src/sentencepiece/sentencepiece_wrap.o -o build/lib.linux-armv7l-3.9/sentencepiece/_sentencepiece.cpython-39-arm-linux-gnueabihf.so ../build/root/lib/libsentencepiece.a ../build/root/lib/libsentencepiece_train.a -Wl,-strip-all
installing to build/bdist.linux-armv7l/wheel
running install
running install_lib
creating build/bdist.linux-armv7l
creating build/bdist.linux-armv7l/wheel
creating build/bdist.linux-armv7l/wheel/sentencepiece
copying build/lib.linux-armv7l-3.9/sentencepiece/__init__.py -> build/bdist.linux-armv7l/wheel/sentencepiece
copying build/lib.linux-armv7l-3.9/sentencepiece/_version.py -> build/bdist.linux-armv7l/wheel/sentencepiece
copying build/lib.linux-armv7l-3.9/sentencepiece/sentencepiece_model_pb2.py -> build/bdist.linux-armv7l/wheel/sentencepiece
copying build/lib.linux-armv7l-3.9/sentencepiece/_sentencepiece.cpython-39-arm-linux-gnueabihf.so -> build/bdist.linux-armv7l/wheel/sentencepiece
copying build/lib.linux-armv7l-3.9/sentencepiece/sentencepiece_pb2.py -> build/bdist.linux-armv7l/wheel/sentencepiece
running install_egg_info
running egg_info
creating src/sentencepiece.egg-info
writing src/sentencepiece.egg-info/PKG-INFO
writing dependency_links to src/sentencepiece.egg-info/dependency_links.txt
writing top-level names to src/sentencepiece.egg-info/top_level.txt
writing manifest file 'src/sentencepiece.egg-info/SOURCES.txt'
reading manifest file 'src/sentencepiece.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching '*' under directory 'sentencepiece'
warning: no files found matching 'VERSION.*'
writing manifest file 'src/sentencepiece.egg-info/SOURCES.txt'
Copying src/sentencepiece.egg-info to build/bdist.linux-armv7l/wheel/sentencepiece-0.2.0.egg-info
running install_scripts
creating build/bdist.linux-armv7l/wheel/sentencepiece-0.2.0.dist-info/WHEEL
creating 'dist/sentencepiece-0.2.0-cp39-cp39-linux_armv7l.whl' and adding 'build/bdist.linux-armv7l/wheel' to it
adding 'sentencepiece/__init__.py'
adding 'sentencepiece/_sentencepiece.cpython-39-arm-linux-gnueabihf.so'
adding 'sentencepiece/_version.py'
adding 'sentencepiece/sentencepiece_model_pb2.py'
adding 'sentencepiece/sentencepiece_pb2.py'
adding 'sentencepiece-0.2.0.dist-info/METADATA'
adding 'sentencepiece-0.2.0.dist-info/WHEEL'
adding 'sentencepiece-0.2.0.dist-info/top_level.txt'
adding 'sentencepiece-0.2.0.dist-info/RECORD'
removing build/bdist.linux-armv7l/wheel

Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Processing ./dist/sentencepiece-0.2.0-cp39-cp39-linux_armv7l.whl
Installing collected packages: sentencepiece
Successfully installed sentencepiece-0.2.0
brandonwheat commented 3 months ago

Did you ever find a solution to this?

FrancescoScandiffio commented 3 months ago

Did you ever find a solution to this?

Hello @brandonwheat, no I never found any solution. Actually, I didn't even look for it because it was just a feasibility study for a hobby project.