search

amir-zeldes / HebPipe

An NLP pipeline for Hebrew
Other
34 stars 9 forks source link

Errors running hebpipe #36

Closed colingoldberg closed 1 year ago

colingoldberg commented 1 year ago

On my MacBook Pro (Mojave, 10.14.6), after successfully installing hebpipe (Python 3.6.7, Pip 23.1), I get the following errors trying to run. (export1.csv contains Hebrew plus English words, separated by commas).

$ python3 -m hebpipe -l export1.csv

Running tasks:

o Automatic sentence splitting (neural) o Lemmatization

Traceback (most recent call last): File "", line 189, in _run_module_as_main File "", line 148, in _get_module_details File "", line 112, in _get_module_details File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/hebpipe/init.py", line 2, in run_hebpipe() File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/hebpipe/heb_pipe.py", line 867, in run_hebpipe tagger = FlairTagger() ^^^^^^^^^^^^^ File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/hebpipe/lib/flair_pos_tagger.py", line 30, in init self.model = SequenceTagger.load(model_dir + "heb.flair") ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/flair/nn.py", line 88, in load state = torch.load(f, map_location='cpu') ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/torch/serialization.py", line 809, in load return _load(opened_zipfile, map_location, pickle_module, **pickle_load_args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/torch/serialization.py", line 1172, in _load result = unpickler.load() ^^^^^^^^^^^^^^^^ File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/torch/serialization.py", line 1165, in find_class return super().find_class(mod_name, name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ModuleNotFoundError: No module named 'transformers.modeling_bert' Elapsed time: 0:00:00.926

I hope I have provided enough information.

Colin Goldberg

amir-zeldes commented 1 year ago

Hi and thanks for reporting - you say that this is python 3.6.7, but the error output indicates that you are running python 3.11. Is it possible that you are accidentally running this with a different python than the one for which the dependencies have been installed?

The error itself indicates an incompatible version of the transformers library. Please ensure that the dependencies in requirements.txt are correctly installed, and if it is still not working, please provide the output of:

python3 -m pip show transformers

amir-zeldes commented 1 year ago

PS: if you run hebpipe on Hebrew + English words without context, you are likely to get some strange results. The pretrained model has certainly seen some English words mixed in with Hebrew, but it expects running text and uses context to disambiguate grammatical categories, so while a CSV is valid input and should not crash, it will not work well.

colingoldberg commented 1 year ago

My apologies - I must have entered "python" instead of "python3" (I have both)

$ python3 --version Python 3.11.2

I don't see a requirements.txt file - I installed hebpipe with "pip3 install hebpipe", and run using "python3 -m hebpipe -l localfile.txt"

$ python3 -m pip show transformers Name: transformers Version: 4.28.1 Summary: State-of-the-art Machine Learning for JAX, PyTorch and TensorFlow Home-page: https://github.com/huggingface/transformers Author: The Hugging Face team (past and future) with the help of all our contributors (https://github.com/huggingface/transformers/graphs/contributors) Author-email: transformers@huggingface.co License: Apache 2.0 License Location: /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages Requires: filelock, huggingface-hub, numpy, packaging, pyyaml, regex, requests, tokenizers, tqdm Required-by: diaparser, flair

The output from "pip3 install hebpipe" was as follows:

Collecting hebpipe Downloading hebpipe-3.0.0.3.tar.gz (8.6 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8.6/8.6 MB 3.1 MB/s eta 0:00:00 Installing build dependencies ... done Getting requirements to build wheel ... done Preparing metadata (pyproject.toml) ... done Requirement already satisfied: requests in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from hebpipe) (2.28.2) Collecting numpy (from hebpipe) Downloading numpy-1.24.3-cp311-cp311-macosx_10_9_x86_64.whl (19.8 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 19.8/19.8 MB 18.4 MB/s eta 0:00:00 Collecting transformers==3.5.1 (from hebpipe) Downloading transformers-3.5.1-py3-none-any.whl (1.3 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.3/1.3 MB 14.7 MB/s eta 0:00:00 INFO: pip is looking at multiple versions of hebpipe to determine which version is compatible with other requirements. This could take a while. Collecting hebpipe Downloading hebpipe-3.0.0.2.tar.gz (8.6 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8.6/8.6 MB 4.7 MB/s eta 0:00:00 Installing build dependencies ... done Getting requirements to build wheel ... done Preparing metadata (pyproject.toml) ... done Downloading hebpipe-3.0.0.1.tar.gz (8.6 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8.6/8.6 MB 3.6 MB/s eta 0:00:00 Installing build dependencies ... done Getting requirements to build wheel ... done Preparing metadata (pyproject.toml) ... done Downloading hebpipe-3.0.0.0.tar.gz (8.6 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8.6/8.6 MB 5.0 MB/s eta 0:00:00 Installing build dependencies ... done Getting requirements to build wheel ... done Preparing metadata (pyproject.toml) ... done Downloading hebpipe-2.0.0.2.tar.gz (8.6 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8.6/8.6 MB 4.6 MB/s eta 0:00:00 Installing build dependencies ... done Getting requirements to build wheel ... done Preparing metadata (pyproject.toml) ... done Downloading hebpipe-2.0.0.1.tar.gz (8.6 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8.6/8.6 MB 4.7 MB/s eta 0:00:00 Installing build dependencies ... done Getting requirements to build wheel ... done Preparing metadata (pyproject.toml) ... done Collecting pandas (from hebpipe) Downloading pandas-2.0.1-cp311-cp311-macosx_10_9_x86_64.whl (11.6 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 11.6/11.6 MB 22.0 MB/s eta 0:00:00 Collecting scipy (from hebpipe) Downloading scipy-1.10.1-cp311-cp311-macosx_10_9_x86_64.whl (35.0 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 35.0/35.0 MB 18.2 MB/s eta 0:00:00 Collecting joblib (from hebpipe) Downloading joblib-1.2.0-py3-none-any.whl (297 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 298.0/298.0 kB 11.0 MB/s eta 0:00:00 Collecting xgboost==0.81 (from hebpipe) Downloading xgboost-0.81.tar.gz (636 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 636.3/636.3 kB 11.2 MB/s eta 0:00:00 Installing build dependencies ... done Getting requirements to build wheel ... done Preparing metadata (pyproject.toml) ... done Collecting rftokenizer (from hebpipe) Downloading rftokenizer-2.0.1.tar.gz (62.1 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.1/62.1 MB 3.3 MB/s eta 0:00:00 Installing build dependencies ... done Getting requirements to build wheel ... done Preparing metadata (pyproject.toml) ... done Collecting depedit (from hebpipe) Downloading depedit-3.2.1.0.zip (27 kB) Installing build dependencies ... done Getting requirements to build wheel ... done Preparing metadata (pyproject.toml) ... done Collecting xmltodict (from hebpipe) Downloading xmltodict-0.13.0-py2.py3-none-any.whl (10.0 kB) Collecting diaparser (from hebpipe) Downloading diaparser-1.1.3-py3-none-any.whl (69 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 69.7/69.7 kB 1.9 MB/s eta 0:00:00 Collecting flair==0.6.1 (from hebpipe) Downloading flair-0.6.1-py3-none-any.whl (331 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 331.5/331.5 kB 5.3 MB/s eta 0:00:00 Collecting stanza (from hebpipe) Downloading stanza-1.5.0-py3-none-any.whl (802 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 802.5/802.5 kB 10.5 MB/s eta 0:00:00 Collecting conllu (from hebpipe) Downloading conllu-4.5.2-py2.py3-none-any.whl (16 kB) Collecting python-dateutil>=2.6.1 (from flair==0.6.1->hebpipe) Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 247.7/247.7 kB 8.3 MB/s eta 0:00:00 Collecting torch>=1.1.0 (from flair==0.6.1->hebpipe) Downloading torch-2.0.0-cp311-none-macosx_10_9_x86_64.whl (139.5 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 139.5/139.5 MB 9.9 MB/s eta 0:00:00 Collecting gensim>=3.4.0 (from flair==0.6.1->hebpipe) Downloading gensim-4.3.1-cp311-cp311-macosx_10_9_x86_64.whl (24.0 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 24.0/24.0 MB 17.7 MB/s eta 0:00:00 Collecting pytest>=5.3.2 (from flair==0.6.1->hebpipe) Downloading pytest-7.3.1-py3-none-any.whl (320 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 320.5/320.5 kB 9.6 MB/s eta 0:00:00 Collecting tqdm>=4.26.0 (from flair==0.6.1->hebpipe) Downloading tqdm-4.65.0-py3-none-any.whl (77 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 77.1/77.1 kB 1.7 MB/s eta 0:00:00 Collecting segtok>=1.5.7 (from flair==0.6.1->hebpipe) Downloading segtok-1.5.11-py3-none-any.whl (24 kB) Collecting matplotlib>=2.2.3 (from flair==0.6.1->hebpipe) Downloading matplotlib-3.7.1-cp311-cp311-macosx_10_12_x86_64.whl (7.4 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.4/7.4 MB 22.9 MB/s eta 0:00:00 Collecting mpld3==0.3 (from flair==0.6.1->hebpipe) Downloading mpld3-0.3.tar.gz (788 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 788.5/788.5 kB 13.7 MB/s eta 0:00:00 Installing build dependencies ... done Getting requirements to build wheel ... done Preparing metadata (pyproject.toml) ... done Collecting scikit-learn>=0.21.3 (from flair==0.6.1->hebpipe) Downloading scikit_learn-1.2.2-cp311-cp311-macosx_10_9_x86_64.whl (9.0 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 9.0/9.0 MB 23.7 MB/s eta 0:00:00 Collecting sqlitedict>=1.6.0 (from flair==0.6.1->hebpipe) Downloading sqlitedict-2.1.0.tar.gz (21 kB) Installing build dependencies ... done Getting requirements to build wheel ... done Preparing metadata (pyproject.toml) ... done Collecting deprecated>=1.2.4 (from flair==0.6.1->hebpipe) Downloading Deprecated-1.2.13-py2.py3-none-any.whl (9.6 kB) Collecting hyperopt>=0.1.1 (from flair==0.6.1->hebpipe) Downloading hyperopt-0.2.7-py2.py3-none-any.whl (1.6 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.6/1.6 MB 23.3 MB/s eta 0:00:00 Collecting transformers>=3.0.0 (from flair==0.6.1->hebpipe) Downloading transformers-4.28.1-py3-none-any.whl (7.0 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.0/7.0 MB 23.6 MB/s eta 0:00:00 Collecting bpemb>=0.3.2 (from flair==0.6.1->hebpipe) Downloading bpemb-0.3.4-py3-none-any.whl (19 kB) Collecting regex (from flair==0.6.1->hebpipe) Downloading regex-2023.5.5-cp311-cp311-macosx_10_9_x86_64.whl (294 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 294.5/294.5 kB 7.7 MB/s eta 0:00:00 Collecting tabulate (from flair==0.6.1->hebpipe) Downloading tabulate-0.9.0-py3-none-any.whl (35 kB) Collecting langdetect (from flair==0.6.1->hebpipe) Downloading langdetect-1.0.9.tar.gz (981 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 981.5/981.5 kB 20.4 MB/s eta 0:00:00 Installing build dependencies ... done Getting requirements to build wheel ... done Preparing metadata (pyproject.toml) ... done Requirement already satisfied: lxml in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from flair==0.6.1->hebpipe) (4.9.2) Collecting ftfy (from flair==0.6.1->hebpipe) Downloading ftfy-6.1.1-py3-none-any.whl (53 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 53.1/53.1 kB 2.4 MB/s eta 0:00:00 Collecting sentencepiece!=0.1.92 (from flair==0.6.1->hebpipe) Downloading sentencepiece-0.1.99-cp311-cp311-macosx_10_9_x86_64.whl (1.2 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 20.2 MB/s eta 0:00:00 Collecting konoha<5.0.0,>=4.0.0 (from flair==0.6.1->hebpipe) Downloading konoha-4.6.5-py3-none-any.whl (20 kB) Collecting janome (from flair==0.6.1->hebpipe) Downloading Janome-0.4.2-py2.py3-none-any.whl (19.7 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 19.7/19.7 MB 22.1 MB/s eta 0:00:00 Collecting gdown (from flair==0.6.1->hebpipe) Downloading gdown-4.7.1-py3-none-any.whl (15 kB) Collecting six (from depedit->hebpipe) Downloading six-1.16.0-py2.py3-none-any.whl (11 kB) Collecting nltk (from diaparser->hebpipe) Downloading nltk-3.8.1-py3-none-any.whl (1.5 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.5/1.5 MB 19.4 MB/s eta 0:00:00 Collecting stanza (from hebpipe) Downloading stanza-1.2.3-py3-none-any.whl (342 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 342.1/342.1 kB 9.6 MB/s eta 0:00:00 Collecting protobuf (from stanza->hebpipe) Downloading protobuf-4.22.3-cp37-abi3-macosx_10_9_universal2.whl (397 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 397.2/397.2 kB 8.4 MB/s eta 0:00:00 Requirement already satisfied: pytz>=2020.1 in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from pandas->hebpipe) (2022.7.1) Collecting tzdata>=2022.1 (from pandas->hebpipe) Downloading tzdata-2023.3-py2.py3-none-any.whl (341 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 341.8/341.8 kB 11.9 MB/s eta 0:00:00 Collecting wrapt<2,>=1.10 (from deprecated>=1.2.4->flair==0.6.1->hebpipe) Downloading wrapt-1.15.0-cp311-cp311-macosx_10_9_x86_64.whl (35 kB) Collecting smart-open>=1.8.1 (from gensim>=3.4.0->flair==0.6.1->hebpipe) Downloading smart_open-6.3.0-py3-none-any.whl (56 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 56.8/56.8 kB 2.1 MB/s eta 0:00:00 Collecting networkx>=2.2 (from hyperopt>=0.1.1->flair==0.6.1->hebpipe) Downloading networkx-3.1-py3-none-any.whl (2.1 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.1/2.1 MB 19.4 MB/s eta 0:00:00 Collecting future (from hyperopt>=0.1.1->flair==0.6.1->hebpipe) Downloading future-0.18.3.tar.gz (840 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 840.9/840.9 kB 16.4 MB/s eta 0:00:00 Installing build dependencies ... done Getting requirements to build wheel ... done Preparing metadata (pyproject.toml) ... done Collecting cloudpickle (from hyperopt>=0.1.1->flair==0.6.1->hebpipe) Downloading cloudpickle-2.2.1-py3-none-any.whl (25 kB) Collecting py4j (from hyperopt>=0.1.1->flair==0.6.1->hebpipe) Downloading py4j-0.10.9.7-py2.py3-none-any.whl (200 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 200.5/200.5 kB 7.9 MB/s eta 0:00:00 Collecting importlib-metadata<4.0.0,>=3.7.0 (from konoha<5.0.0,>=4.0.0->flair==0.6.1->hebpipe) Downloading importlib_metadata-3.10.1-py3-none-any.whl (14 kB) Collecting overrides<4.0.0,>=3.0.0 (from konoha<5.0.0,>=4.0.0->flair==0.6.1->hebpipe) Downloading overrides-3.1.0.tar.gz (11 kB) Installing build dependencies ... done Getting requirements to build wheel ... done Preparing metadata (pyproject.toml) ... done Collecting contourpy>=1.0.1 (from matplotlib>=2.2.3->flair==0.6.1->hebpipe) Downloading contourpy-1.0.7-cp311-cp311-macosx_10_9_x86_64.whl (244 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 244.2/244.2 kB 9.3 MB/s eta 0:00:00 Collecting cycler>=0.10 (from matplotlib>=2.2.3->flair==0.6.1->hebpipe) Downloading cycler-0.11.0-py3-none-any.whl (6.4 kB) Collecting fonttools>=4.22.0 (from matplotlib>=2.2.3->flair==0.6.1->hebpipe) Downloading fonttools-4.39.3-py3-none-any.whl (1.0 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.0/1.0 MB 13.8 MB/s eta 0:00:00 Collecting kiwisolver>=1.0.1 (from matplotlib>=2.2.3->flair==0.6.1->hebpipe) Downloading kiwisolver-1.4.4-cp311-cp311-macosx_10_9_x86_64.whl (65 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 65.5/65.5 kB 1.9 MB/s eta 0:00:00 Collecting packaging>=20.0 (from matplotlib>=2.2.3->flair==0.6.1->hebpipe) Downloading packaging-23.1-py3-none-any.whl (48 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 48.9/48.9 kB 1.8 MB/s eta 0:00:00 Collecting pillow>=6.2.0 (from matplotlib>=2.2.3->flair==0.6.1->hebpipe) Downloading Pillow-9.5.0-cp311-cp311-macosx_10_10_x86_64.whl (3.4 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.4/3.4 MB 23.7 MB/s eta 0:00:00 Collecting pyparsing>=2.3.1 (from matplotlib>=2.2.3->flair==0.6.1->hebpipe) Downloading pyparsing-3.0.9-py3-none-any.whl (98 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 98.3/98.3 kB 3.3 MB/s eta 0:00:00 Collecting iniconfig (from pytest>=5.3.2->flair==0.6.1->hebpipe) Downloading iniconfig-2.0.0-py3-none-any.whl (5.9 kB) Collecting pluggy<2.0,>=0.12 (from pytest>=5.3.2->flair==0.6.1->hebpipe) Downloading pluggy-1.0.0-py2.py3-none-any.whl (13 kB) Requirement already satisfied: charset-normalizer<4,>=2 in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from requests->hebpipe) (3.1.0) Requirement already satisfied: idna<4,>=2.5 in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from requests->hebpipe) (3.4) Requirement already satisfied: urllib3<1.27,>=1.21.1 in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from requests->hebpipe) (1.26.15) Requirement already satisfied: certifi>=2017.4.17 in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from requests->hebpipe) (2022.12.7) Collecting threadpoolctl>=2.0.0 (from scikit-learn>=0.21.3->flair==0.6.1->hebpipe) Downloading threadpoolctl-3.1.0-py3-none-any.whl (14 kB) Collecting filelock (from torch>=1.1.0->flair==0.6.1->hebpipe) Downloading filelock-3.12.0-py3-none-any.whl (10 kB) Collecting typing-extensions (from torch>=1.1.0->flair==0.6.1->hebpipe) Downloading typing_extensions-4.5.0-py3-none-any.whl (27 kB) Collecting sympy (from torch>=1.1.0->flair==0.6.1->hebpipe) Downloading sympy-1.11.1-py3-none-any.whl (6.5 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.5/6.5 MB 22.6 MB/s eta 0:00:00 Collecting jinja2 (from torch>=1.1.0->flair==0.6.1->hebpipe) Downloading Jinja2-3.1.2-py3-none-any.whl (133 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 133.1/133.1 kB 4.9 MB/s eta 0:00:00 Collecting huggingface-hub<1.0,>=0.11.0 (from transformers>=3.0.0->flair==0.6.1->hebpipe) Downloading huggingface_hub-0.14.1-py3-none-any.whl (224 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 224.5/224.5 kB 9.1 MB/s eta 0:00:00 Collecting pyyaml>=5.1 (from transformers>=3.0.0->flair==0.6.1->hebpipe) Downloading PyYAML-6.0-cp311-cp311-macosx_10_9_x86_64.whl (188 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 188.6/188.6 kB 8.7 MB/s eta 0:00:00 Collecting tokenizers!=0.11.3,<0.14,>=0.11.1 (from transformers>=3.0.0->flair==0.6.1->hebpipe) Downloading tokenizers-0.13.3-cp311-cp311-macosx_10_11_universal2.whl (4.0 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.0/4.0 MB 21.2 MB/s eta 0:00:00 Collecting wcwidth>=0.2.5 (from ftfy->flair==0.6.1->hebpipe) Downloading wcwidth-0.2.6-py2.py3-none-any.whl (29 kB) Requirement already satisfied: beautifulsoup4 in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from gdown->flair==0.6.1->hebpipe) (4.12.2) Collecting click (from nltk->diaparser->hebpipe) Downloading click-8.1.3-py3-none-any.whl (96 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 96.6/96.6 kB 3.6 MB/s eta 0:00:00 Collecting fsspec (from huggingface-hub<1.0,>=0.11.0->transformers>=3.0.0->flair==0.6.1->hebpipe) Downloading fsspec-2023.4.0-py3-none-any.whl (153 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 154.0/154.0 kB 5.4 MB/s eta 0:00:00 Collecting zipp>=0.5 (from importlib-metadata<4.0.0,>=3.7.0->konoha<5.0.0,>=4.0.0->flair==0.6.1->hebpipe) Downloading zipp-3.15.0-py3-none-any.whl (6.8 kB) Requirement already satisfied: soupsieve>1.2 in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from beautifulsoup4->gdown->flair==0.6.1->hebpipe) (2.4.1) Collecting MarkupSafe>=2.0 (from jinja2->torch>=1.1.0->flair==0.6.1->hebpipe) Downloading MarkupSafe-2.1.2-cp311-cp311-macosx_10_9_x86_64.whl (13 kB) Collecting PySocks!=1.5.7,>=1.5.6 (from requests->hebpipe) Downloading PySocks-1.7.1-py3-none-any.whl (16 kB) Collecting mpmath>=0.19 (from sympy->torch>=1.1.0->flair==0.6.1->hebpipe) Downloading mpmath-1.3.0-py3-none-any.whl (536 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 536.2/536.2 kB 13.0 MB/s eta 0:00:00 Building wheels for collected packages: hebpipe, xgboost, mpld3, depedit, rftokenizer, sqlitedict, langdetect, overrides, future Building wheel for hebpipe (pyproject.toml) ... done Created wheel for hebpipe: filename=hebpipe-2.0.0.1-py3-none-any.whl size=8813734 sha256=ea3157c5f334b54541545cf2f91a6c33455576e8c6bc9a01fee736f3e7d40e2e Stored in directory: /Users/colingoldberg/Library/Caches/pip/wheels/7f/5a/8b/a1804c7b3334b267315bdbbee5098582cbb80083d7fd93b120 Building wheel for xgboost (pyproject.toml) ... done Created wheel for xgboost: filename=xgboost-0.81-cp311-cp311-macosx_10_14_universal2.whl size=1563746 sha256=6b8e57b431118bdf6b1619f617c1e98adb6ca0ba0fa2a2ebfabd29df4196973a Stored in directory: /Users/colingoldberg/Library/Caches/pip/wheels/e6/ef/7f/a3bf2bc317f4b716f65f538185a9ac649f3c29c2f3a58cd796 Building wheel for mpld3 (pyproject.toml) ... done Created wheel for mpld3: filename=mpld3-0.3-py3-none-any.whl size=116685 sha256=777ef6696c1357134344d3820df4fd64433c99e46b027a23b6b21948773e7fc2 Stored in directory: /Users/colingoldberg/Library/Caches/pip/wheels/bf/27/56/70e869d230f95edfe73551cca267e698178c0639fee888b0d3 Building wheel for depedit (pyproject.toml) ... done Created wheel for depedit: filename=depedit-3.2.1.0-py3-none-any.whl size=28561 sha256=391d4bc3b8fead8bdab21737c48f90f6d18b35deeff4ecd994820c5960a12cd1 Stored in directory: /Users/colingoldberg/Library/Caches/pip/wheels/4f/e9/dc/536c4be301dc6250030e0a652d04ca2f6c6c35f16a27d81029 Building wheel for rftokenizer (pyproject.toml) ... done Created wheel for rftokenizer: filename=rftokenizer-2.0.1-py3-none-any.whl size=62374621 sha256=59a94d92a34ff1b4905ae8138f4a7b647b764b2ef629d0348028134a48d3758f Stored in directory: /Users/colingoldberg/Library/Caches/pip/wheels/3b/3c/e8/a2af611bae4e997f61774d9fd5f370a8dc1c4845e91da1540f Building wheel for sqlitedict (pyproject.toml) ... done Created wheel for sqlitedict: filename=sqlitedict-2.1.0-py3-none-any.whl size=16863 sha256=31b41d0d7eeed43776e863c0ef955bbc6f93a15822247fb375bc02bbd96520a3 Stored in directory: /Users/colingoldberg/Library/Caches/pip/wheels/73/63/89/7210274f9b7fb033b8f22671f64c0e0b55083d30c3c046a3ff Building wheel for langdetect (pyproject.toml) ... done Created wheel for langdetect: filename=langdetect-1.0.9-py3-none-any.whl size=993224 sha256=0fda97225e570a637e84406a61900e6be6a71c96dd682400813285d779149056 Stored in directory: /Users/colingoldberg/Library/Caches/pip/wheels/0a/f2/b2/e5ca405801e05eb7c8ed5b3b4bcf1fcabcd6272c167640072e Building wheel for overrides (pyproject.toml) ... done Created wheel for overrides: filename=overrides-3.1.0-py3-none-any.whl size=10173 sha256=fc5674d78555770c56a6405203e27ed62ba223e22752b3b51787d97517da909a Stored in directory: /Users/colingoldberg/Library/Caches/pip/wheels/9c/ef/a2/1b900acdfd51243b15d61c4fdcd913447f901dbe6adf150eae Building wheel for future (pyproject.toml) ... done Created wheel for future: filename=future-0.18.3-py3-none-any.whl size=492022 sha256=0910f772bfa0ae966594499c9d79160cc5cad81cef5e26694704e037ea2d28c6 Stored in directory: /Users/colingoldberg/Library/Caches/pip/wheels/da/19/ca/9d8c44cd311a955509d7e13da3f0bea42400c469ef825b580b Successfully built hebpipe xgboost mpld3 depedit rftokenizer sqlitedict langdetect overrides future Installing collected packages: wcwidth, tokenizers, sqlitedict, sentencepiece, py4j, overrides, mpmath, mpld3, janome, zipp, xmltodict, wrapt, tzdata, typing-extensions, tqdm, threadpoolctl, tabulate, sympy, smart-open, six, regex, pyyaml, PySocks, pyparsing, protobuf, pluggy, pillow, packaging, numpy, networkx, MarkupSafe, kiwisolver, joblib, iniconfig, future, ftfy, fsspec, fonttools, filelock, cycler, conllu, cloudpickle, click, segtok, scipy, python-dateutil, pytest, nltk, langdetect, jinja2, importlib-metadata, huggingface-hub, deprecated, depedit, contourpy, xgboost, transformers, torch, scikit-learn, pandas, matplotlib, konoha, hyperopt, gensim, gdown, stanza, rftokenizer, bpemb, flair, diaparser, hebpipe Successfully installed MarkupSafe-2.1.2 PySocks-1.7.1 bpemb-0.3.4 click-8.1.3 cloudpickle-2.2.1 conllu-4.5.2 contourpy-1.0.7 cycler-0.11.0 depedit-3.2.1.0 deprecated-1.2.13 diaparser-1.1.3 filelock-3.12.0 flair-0.6.1 fonttools-4.39.3 fsspec-2023.4.0 ftfy-6.1.1 future-0.18.3 gdown-4.7.1 gensim-4.3.1 hebpipe-2.0.0.1 huggingface-hub-0.14.1 hyperopt-0.2.7 importlib-metadata-3.10.1 iniconfig-2.0.0 janome-0.4.2 jinja2-3.1.2 joblib-1.2.0 kiwisolver-1.4.4 konoha-4.6.5 langdetect-1.0.9 matplotlib-3.7.1 mpld3-0.3 mpmath-1.3.0 networkx-3.1 nltk-3.8.1 numpy-1.24.3 overrides-3.1.0 packaging-23.1 pandas-2.0.1 pillow-9.5.0 pluggy-1.0.0 protobuf-4.22.3 py4j-0.10.9.7 pyparsing-3.0.9 pytest-7.3.1 python-dateutil-2.8.2 pyyaml-6.0 regex-2023.5.5 rftokenizer-2.0.1 scikit-learn-1.2.2 scipy-1.10.1 segtok-1.5.11 sentencepiece-0.1.99 six-1.16.0 smart-open-6.3.0 sqlitedict-2.1.0 stanza-1.2.3 sympy-1.11.1 tabulate-0.9.0 threadpoolctl-3.1.0 tokenizers-0.13.3 torch-2.0.0 tqdm-4.65.0 transformers-4.28.1 typing-extensions-4.5.0 tzdata-2023.3 wcwidth-0.2.6 wrapt-1.15.0 xgboost-0.81 xmltodict-0.13.0 zipp-3.15.0

amir-zeldes commented 1 year ago

Right, in that case the problem is clear - the models were trained under transformers==3.5.1, and this is also specified in the requirements file here:

https://github.com/amir-zeldes/HebPipe/blob/master/requirements.txt#LL2C20-L2C20

I'm not sure if there is even a transformers wheel with that version for Python 3.11, which is very new. You could either compile 3.5.1 from source for Python 3.11, work with an older Python (I'm sure it was working on Python 3.8 and I believe also 3.9), or stay with the Python you have but retrain the models from scratch under your library versions.

Probably the last is the most difficult, and using an older Python release is the easiest. We may release an updated version of the model with transformers 4.X at some point, but not right away in any case.