Undefined Symbol

[norasyeezys]

_pynini.cpython-312-x86_64-linux-gnu.so: undefined symbol: FST_FLAGS_fst_align

[kylebgorman]

You're going to need to write a longer description than that.

[norasyeezys]

You're going to need to write a longer description than that.

Traceback (most recent call last): File "/.../pyninitest.py", line 1, in import pynini File "/.../pynini/init.py", line 1, in from _pynini import * ImportError: /.../pynini-2.1.5/_pynini.cpython-312-x86_64-linux-gnu.so: undefined symbol: FST_FLAGS_fst_align

[kylebgorman]

I can't replicate, sorry. I don't know how you built the library and its dependencies, and I am only inferring plattform from the subsrings in the .so name, but I know that Pynini 2.1.5 on CPython 3.12 on x86_64 Linux does not give me this error.

The current version is 2.1.6. I think you are probably using an older version of Pynini built against an even older version of OpenFst. Pynini 2.1.6 is always built against OpenFst 1.8.3. There are precompiled versions available for your platform anyways, from PyPI (pip install --only-binary :all: pynini) and also via Conda-Forge (conda install -c conda-forge pynini). Given how long it takes to build and given that it's not working why don't you try one of those first?

I must say I find the way you reported this bug to be rude. I'd ask you to clearly state more clearly what you did both in terms of compilation, and what platform you're working on etc. We're all volunteers here and a bug request is an implicit request for help.

[norasyeezys]

I can't replicate, sorry. I don't know how you built the library and its dependencies, and I am only inferring plattform from the subsrings in the .so name, but I know that Pynini 2.1.5 on CPython 3.12 on x86_64 Linux does not give me this error.

The current version is 2.1.6. I think you are probably using an older version of Pynini built against an even older version of OpenFst. Pynini 2.1.6 is always built against OpenFst 1.8.3. There are precompiled versions available for your platform anyways, from PyPI (pip install --only-binary :all: pynini) and also via Conda-Forge (conda install -c conda-forge pynini). Given how long it takes to build and given that it's not working why don't you try one of those first?

I must say I find the way you reported this bug to be rude. I'd ask you to clearly state more clearly what you did both in terms of compilation, and what platform you're working on etc. We're all volunteers here and a bug request is an implicit request for help.

My apologies. It wasn't my intention to be rude, so I will clarify a bit better.

I installed openfst version 1.8.2. I tried to install an older version of openfst (1.6.9) however both times didn't let me get the python extension (I was using 3.12, and they didn't let me get python because my version wasn't above 3.6, even thought it clearly was)

My OS is ubuntu 24.04. I used the little bit older version of Pynini compiled from source because I needed that specific version to run NVidia's NeMo.

This error was in a very simple program hence why the error was only in an import.

[kylebgorman]

If you don't want to use the newest version, try this combination: OpenFst 1.8.2.post1 and Pynini 2.1.5.post2. It's too far long ago for me to remember, but the "post" versions might fix the issue you're dealing with. There is a pre-compiled Pynini for 2.1.5* out there too, though Python 3.12 didn't exist at the time so it doesn't support that, sorry to say.

One thing that may help you is that you don't need to --enable-python when building OpenFst for Pynini support, actually. (All you need to do is --enable-grm.) Pynini ships with a identical copy of pywrapfst and the Python-specific extension compilation framework Pynini can use is much more sophisticated than autoconf (it isn't plagued by the rather stupid bug, outside my control, that autoconf uses lexicographic comparison for version strings and "3.6" is lexicographically higher than "3.12"). Good luck there.

[Atmelfan]

I also get this error even when building openfst 1.8.3 directly from source (with --enable-grm) and installing pynini from github. For reference (Ubuntu 20.04, Aarch64):

wget  --no-check-certificate https://www.openfst.org/twiki/pub/FST/FstDownload/openfst-1.8.3.tar.gz
cd openfst-1.8.3
./configure --enable-grm
make -j8
sudo make install
pip install --upgrade git+https://github.com/kylebgorman/pynini.git

In python:

nvidia@ubuntu:/tmp/openfst-1.8.3$ python
Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pynini
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/nvidia/.local/lib/python3.10/site-packages/pynini/__init__.py", line 1, in <module>
    from _pynini import *
ImportError: /home/nvidia/.local/lib/python3.10/site-packages/_pynini.cpython-310-aarch64-linux-gnu.so: undefined symbol: FST_FLAGS_fst_error_fatal
>>> 

[kylebgorman]

@Atmelfan I just attempted to replicate this on x86_64 Ubuntu. (I added in decompressing the OpenFst tarball but didn't do anything else different.) However the import statement worked out of the box without error so I have failed. I am not sure, but I have a hypothesis that in your environment it may have linked against the "wrong" OpenFst installation (there would be additional copies of a file with a name like libfst-VERSION.so) since you mentioned earlier that you were experimenting with earlier versions and this symbol didn't exist in some earlier versions. I don't know how to quickly test that other than for you to repeat these steps in a completely hermetic environment and see if that solves anything.

[Atmelfan]

Ok, so it seems I have a libfst.so.8 in /usr/lib while the newer libfst.so.26 was installed in /usr/local/lib... Trying to get pip to use the local version instead.

[Atmelfan]

Success, specifying build options during pip install works.

For anyone else in the future who might find this. This is how I got it working:

cd /tmp

# Download, build, and install openfst 1.8.3
wget  --no-check-certificate https://www.openfst.org/twiki/pub/FST/FstDownload/openfst-1.8.3.tar.gz
tar -xzvf file.tar.gz
cd openfst-1.8.3
./configure --enable-grm
make -j8
sudo make install

# Install pynini 
pip install \
    --config-settings="--build-option=build_ext" \
    --config-settings="--build-option=--library-dirs=/usr/local/include/" \
    --config-settings="--build-option=--library-dirs=/usr/local/lib" \
    --upgrade \
    --verbose \
    git+https://github.com/kylebgorman/pynini.git

DenverCoder9@ubuntu:/tmp/openfst-1.8.3$ python
Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pynini
>>> 

Edit: Below might also be necessary whenever running python for it to fins the right .so.

export LD_LIBRARY_PATH="/usr/local/lib/fst/:/usr/local/lib/:$LD_LIBRARY_PATH"

[kylebgorman]

Okay, I'll close this since this is WAI.