ImportError [...]/lib/python3.10/site-packages/gluonnlp/data/fast_bert_tokenizer.cpython-310-x86_64-linux-gnu.so: undefined symbol: _PyGen_Send

[MBrede]

I hope this is not an incredibly dumb issue, but googling did not produce any useable results. I installed gluonnlp 0.10.0 using pip on a Ubuntu 22.04.1 LTS server under Python 3.10.6. When I try to import gluon, I get the following error:

Traceback (most recent call last):
  File "[...]/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3460, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-4-e68c3b59001c>", line 1, in <module>
    import gluonnlp as nlp
  File "[...]/lib/python3.10/site-packages/gluonnlp/__init__.py", line 25, in <module>
    from . import data
  File "[...]/lib/python3.10/site-packages/gluonnlp/data/__init__.py", line 23, in <module>
    from . import (batchify, candidate_sampler, conll, corpora, dataloader,
  File "[...]/lib/python3.10/site-packages/gluonnlp/data/corpora/__init__.py", line 21, in <module>
    from . import (google_billion_word, large_text_compression_benchmark, wikitext)
  File "[...]/lib/python3.10/site-packages/gluonnlp/data/corpora/google_billion_word.py", line 34, in <module>
    from ...vocab import Vocab
  File "[...]/lib/python3.10/site-packages/gluonnlp/vocab/__init__.py", line 21, in <module>
    from . import bert, elmo, subwords, vocab
  File "[...]/lib/python3.10/site-packages/gluonnlp/vocab/bert.py", line 24, in <module>
    from ..data.transforms import SentencepieceTokenizer
  File "[...]/lib/python3.10/site-packages/gluonnlp/data/transforms.py", line 48, in <module>
    from .fast_bert_tokenizer import is_control, is_punctuation, is_whitespace
ImportError: [...]/lib/python3.10/site-packages/gluonnlp/data/fast_bert_tokenizer.cpython-310-x86_64-linux-gnu.so: undefined symbol: _PyGen_Send

pip list returns the following modules, if this information is helpfull:

Package                  Version
------------------------ -----------
asttokens                2.2.1
backcall                 0.2.0
certifi                  2022.12.7
charset-normalizer       3.0.1
click                    8.1.3
contourpy                1.0.7
cvxpy                    1.3.0
cycler                   0.11.0
Cython                   0.29.33
decorator                5.1.1
ecos                     2.0.12
executing                1.2.0
fasttext                 0.9.2
filelock                 3.9.0
fonttools                4.38.0
fst-pso                  1.8.1
FuzzyTM                  2.0.5
gensim                   4.3.0
germansentiment          1.1.0
gluonnlp                 0.10.0
graphviz                 0.8.4
h5py                     3.8.0
hdbscan                  0.8.29
huggingface-hub          0.12.1
idna                     3.4
ipython                  8.10.0
jedi                     0.18.2
joblib                   1.2.0
kaleido                  0.2.1
kiwisolver               1.4.4
llvmlite                 0.39.1
matplotlib               3.7.0
matplotlib-inline        0.1.6
miniful                  0.0.6
mxnet                    1.9.1
nltk                     3.8.1
numba                    0.56.4
numpy                    1.23.5
nvidia-cublas-cu11       11.10.3.66
nvidia-cuda-nvrtc-cu11   11.7.99
nvidia-cuda-runtime-cu11 11.7.99
nvidia-cudnn-cu11        8.5.0.96
osqp                     0.6.2.post8
packaging                23.0
pandas                   1.5.3
parso                    0.8.3
pexpect                  4.8.0
pickleshare              0.7.5
Pillow                   9.4.0
pip                      23.0.1
plotly                   5.13.0
polars                   0.16.7
prompt-toolkit           3.0.36
ptyprocess               0.7.0
pure-eval                0.2.2
pybind11                 2.10.3
pyFUME                   0.2.25
Pygments                 2.14.0
pynndescent              0.5.8
pyparsing                3.0.9
pyportfolioopt           1.5.4
python-dateutil          2.8.2
pytz                     2022.7.1
PyYAML                   6.0
qdldl                    0.1.5.post3
regex                    2022.10.31
requests                 2.28.2
scikit-learn             1.2.1
scipy                    1.10.1
scs                      3.2.2
sentence-transformers    2.2.2
sentencepiece            0.1.97
setuptools               64.0.2
simpful                  2.9.0
simple-elmo              0.9.1
six                      1.16.0
sklearn                  0.0.post1
smart-open               6.3.0
Snowball                 0.5.2
stack-data               0.6.2
tenacity                 8.2.1
threadpoolctl            3.1.0
tokenizers               0.13.2
top2vec                  1.0.28
torch                    1.13.1
torchvision              0.14.1
tqdm                     4.64.1
traitlets                5.9.0
transformers             4.26.1
typing                   3.6.6
typing_extensions        4.5.0
umap-learn               0.5.3
urllib3                  1.26.14
wcwidth                  0.2.6
wheel                    0.38.4
wordcloud                1.8.2.2

I tried reinstalling cython, gluonnlp and mxnet and would like to avoid using another python version if possible. I am not sure whether this is a gluon or a Cython issue, I hope this is the right place to ask! I did also post a question on StackOverflow regarding this issue.

[MBrede]

It was the old mxnet version, sorry for any inconvenience.

[djherbis]

@MBrede I'm running into this error as well. What version of mxnet are you using that fixed it? I've tried installing 1.9.1 but it didn't appear to resolve it in my env at least.

[MBrede]

@djherbis I installed the version mentioned in the README (python3 -m pip install -U --pre "mxnet>=2.0.0a") which did fix the aforementioned error but did end up in a new error (which I unfortunately do not remember). In the end, I wrote a module wrapping the models I wanted to use from other modules...

[djherbis]

Thanks, yeah I was unable to get it to work with that command either :(

For now I've pulled it from the Kaggle image for low use anyway.

Thanks for responding though!

[hankyuwon]

Downgrade python version