Lemmatizer hangs

[jk-1]

My environment is Virtualbox virtual machine with Ubuntu 16.04, 12GB mem, 8 CPUs/cores.

I installed TurkuNLP with your instructions. During the installation I got following error:

ERROR: markdown 3.2 has requirement setuptools>=36, but you'll have setuptools 20.7.0 which is incompatible.
Installing collected packages: idna, chardet, certifi, urllib3, requests, numpy, six, keras-preprocessing, h5py, scipy, keras-applications, pyyaml, keras, kiwisolver, cycler, pyparsing,       python-dateutil, matplotlib, Werkzeug, click, itsdangerous, MarkupSafe, Jinja2, flask, ufal.udpipe, configargparse, gast, protobuf, grpcio, astor, termcolor, absl-py, markdown, 
tensorboard,    tensorflow, torch, tqdm, torchtext, pillow, torchvision
Successfully installed Jinja2-2.11.1 MarkupSafe-1.1.1 Werkzeug-1.0.0 absl-py-0.9.0 astor-0.8.1 certifi-2019.11.28 chardet-3.0.4 click-7.0 configargparse-1.0 cycler-0.10.0 flask-1.1.1 
gast-0.3.3 grpcio-1.27.1 h5py-2.10.0 idna-2.8 itsdangerous-1.1.0 keras-2.3.1 keras-applications-1.0.8 keras-preprocessing-1.1.0 kiwisolver-1.1.0 markdown-3.2 matplotlib-3.0.3 numpy-1.18.1
pillow-7.0.0 protobuf-3.11.3 pyparsing-2.4.6 python-dateutil-2.8.1 pyyaml-5.3 requests-2.22.0 scipy-1.4.1 six-1.14.0 tensorboard-1.12.2 tensorflow-1.12.2 termcolor-1.1.0 torch-0.4.1
torchtext-0.3.1 torchvision-0.2.1 tqdm-4.42.1 ufal.udpipe-1.2.0.3 urllib3-1.25.8

So it seems that markdown was installed ok despite the setuptools version error.

First run gave errors: CUDA driver version is insufficient for CUDA runtime version

I fixed this by reinstalling torch with no gpu settings

pip install torch==1.4.0+cpu torchvision==0.5.0+cpu -f https://download.pytorch.org/whl/torch_stable.html

After that few pipelines seem to work. However, pipelines with command:

  - lemmatizer_mod --model {thisdir}/Lemmatizer/lemmatizer.pt --replace_unk

hang.

When running full pipleline server with command

(venv-parser-neural) jk@mypc:~/Turku-neural-parser-pipeline$ python full_pipeline_server.py --port 7689 --conf models_fi_tdt/pipelines.yaml parse_plaintext

and client with client command:

 curl --request POST --header 'Content-Type: text/plain; charset=utf-8' --data-binary "Tämä on esimerkkilause. Minulla on koira. " http://localhost:7689

and with your original plain_text pipeline, the server hangs:

(venv-parser-neural) jk@mypc:~/Turku-neural-parser-pipeline$ python full_pipeline_server.py --port 7689 --conf models_fi_tdt/pipelines.yaml parse_plaintext
... output deleted ...
Initial # of tokens in buckets: 133
Final # of tokens in buckets: 133
Bucket parseset-0 is 5 x 6 x 9
Bucket parseset-1 is 4 x 9 x 9
Bucket parseset-2 is 4 x 12 x 9
Bucket parseset-3 is 3 x 14 x 9
Bucket parseset-4 is 3 x 16 x 9
 >>> 0/8 lemmas already filled before lemma cache module
 >>> 7/8 lemmatized with lemma cache

When running with modified pipeline

toke_parse_lemma:
  - tokenizer_udpipe_mod --model {thisdir}/Tokenizer/tokenizer.udpipe
  - parser_mod --model {thisdir}/Tagger
  - parser_mod --model {thisdir}/Parser --process_morpho
  - lemma_cache_mod --lemma_cache {thisdir}/Lemmatizer/big_lemma_cache.tsv

the same command, output is:

jk@mypc:~/Turku-neural-parser-pipeline/models_fi_tdt$ curl --request POST --header 'Content-Type: text/plain; charset=utf-8' --data-binary "Tämä on esimerkkilause. Minulla on koira. " \
http://localhost:7689
# newdoc
# newpar
# sent_id = 1
# text = Tämä on esimerkkilause.
1   Tämä    tämä    PRON    Pron    Case=Nom|Number=Sing|PronType=Dem   3   nsubj:cop   _   _
2   on  olla    AUX V   Mood=Ind|Number=Sing|Person=3|Tense=Pres|VerbForm=Fin|Voice=Act 3   cop _   _
3   esimerkkilause  _   NOUN    N   Case=Nom|Number=Sing    0   root    _   SpaceAfter=No
4   .   .   PUNCT   Punct   _   3   punct   _   _

# sent_id = 2
# text = Minulla on koira.
1   Minulla minä    PRON    Pron    Case=Ade|Number=Sing|Person=1|PronType=Prs  0   root    _   _
2   on  olla    AUX V   Mood=Ind|Number=Sing|Person=3|Tense=Pres|VerbForm=Fin|Voice=Act 1   cop:own _   _
3   koira   koira   NOUN    N   Case=Nom|Number=Sing    1   nsubj:cop   _   SpaceAfter=No
4   .   .   PUNCT   Punct   _   1   punct   _   SpacesAfter=\s\n

How to fix this issue? Is it possible that the setuptools version mismatch has something to do with this error in lemmatizer?

Thank you for your support.

[fginter]

Hi thanks for the detailed report. Try running the pipeline with --gpu -1 that should make sure no attempt at GPU is made. Do tell if this solved your problem. This is mentioned here http://turkunlp.org/Turku-neural-parser-pipeline/ but maybe we should make it more clear, how to separate the CPU and GPU stuff.

[fginter]

right, and you should not remove the lemmatizer_mod because that will only fall-back on lemma cache and unknown words won't get lemmatized, which is probably not what you want.

[jk-1]

Now running in non-server mode is working:

non-server-output.txt

Running in server mode gives following error:

server-output.txt

Client command was:

curl --request POST --header 'Content-Type: text/plain; charset=utf-8' --data-binary "Tämä on esimerkkilause. Minulla on koira. " http://localhost:7689

[fginter]

Never seen this error. :confused: the difference as far as I can tell is in the text. In the former case lemmatization happens from cache and the lemmatizer does not invoke, in the latter case the input is a little longer and the lemmatizer actually invokes and crashes. My bet is this must have something to do with versions. The lemmatizer is PyTorch, so I would look there first and try to downgrade torch few minor versions down. I will also do some tests myself, see if I can replicate the bug.

[jk-1]

You are right. Also non-server fails if the sentence is same/longer.

Here are my versions:

(venv-parser-neural) jk@mypc:~/Turku-neural-parser-pipeline$ pip list
Package             Version   
------------------- ----------
absl-py             0.9.0     
astor               0.8.1     
certifi             2019.11.28
chardet             3.0.4     
Click               7.0       
ConfigArgParse      1.0       
cycler              0.10.0    
Flask               1.1.1     
gast                0.3.3     
grpcio              1.27.1    
h5py                2.10.0    
idna                2.8       
itsdangerous        1.1.0     
Jinja2              2.11.1    
Keras               2.3.1     
Keras-Applications  1.0.8     
Keras-Preprocessing 1.1.0     
kiwisolver          1.1.0     
Markdown            3.2       
MarkupSafe          1.1.1     
matplotlib          3.0.3     
numpy               1.18.1    
Pillow              7.0.0     
pip                 20.0.2    
pkg-resources       0.0.0     
protobuf            3.11.3    
pyparsing           2.4.6     
python-dateutil     2.8.1     
PyYAML              5.3       
requests            2.22.0    
scipy               1.4.1     
setuptools          20.7.0    
six                 1.14.0    
tensorboard         1.12.2    
tensorflow          1.12.2    
termcolor           1.1.0     
torch               1.4.0+cpu 
torchtext           0.3.1     
torchvision         0.5.0+cpu 
tqdm                4.42.1    
ufal.udpipe         1.2.0.3   
urllib3             1.25.8    
Werkzeug            1.0.0     
wheel               0.34.2    
(venv-parser-neural) jk@mypc:~/Turku-neural-parser-pipeline$ python --version
Python 3.5.2
(venv-parser-neural) jk@mypc:~/Turku-neural-parser-pipeline$ pip3 --version
pip 20.0.2 from /home/jk/Turku-neural-parser-pipeline/venv-parser-neural/lib/python3.5/site-packages/pip (python 3.5)
(venv-parser-neural) jk@mypc:~/Turku-neural-parser-pipeline$ 

[jk-1]

As I said in the first message the initial install complained about the setuptools version:

ERROR: markdown 3.2 has requirement setuptools>=36, but you'll have setuptools 20.7.0 which is incompatible.

I managed to upgrade setup tools to 36.0.1, but the parsing error is still the same.

Versions now:

(venv-parser-neural) jk@mypc:~/Turku-neural-parser-pipeline$ pip list
Package             Version   
------------------- ----------
absl-py             0.9.0     
astor               0.8.1     
certifi             2019.11.28
chardet             3.0.4     
Click               7.0       
ConfigArgParse      1.0       
cycler              0.10.0    
Flask               1.1.1     
gast                0.3.3     
grpcio              1.27.1    
h5py                2.10.0    
idna                2.8       
itsdangerous        1.1.0     
Jinja2              2.11.1    
Keras               2.3.1     
Keras-Applications  1.0.8     
Keras-Preprocessing 1.1.0     
kiwisolver          1.1.0     
Markdown            3.2       
MarkupSafe          1.1.1     
matplotlib          3.0.3     
numpy               1.18.1    
Pillow              7.0.0     
pip                 20.0.2    
pkg-resources       0.0.0     
protobuf            3.11.3    
pyparsing           2.4.6     
python-dateutil     2.8.1     
PyYAML              5.3       
requests            2.22.0    
scipy               1.4.1     
setuptools          36.0.1    
six                 1.14.0    
tensorboard         1.12.2    
tensorflow          1.12.2    
termcolor           1.1.0     
torch               1.4.0+cpu 
torchtext           0.3.1     
torchvision         0.5.0+cpu 
tqdm                4.42.1    
ufal.udpipe         1.2.0.3   
urllib3             1.25.8    
Werkzeug            1.0.0     
wheel               0.34.2    
yolk3k              0.9       
(venv-parser-neural) jk@mypc:~/Turku-neural-parser-pipeline$ 

[haamis]

Hi,

Try downgrading your pytorch to 1.1.0. Apparently this mask inversion syntax was changed in 1.2.0. The requirements-cpu.txt file in this repository specifies torch==0.4.1 and your version is quite a lot newer than that.

[jk-1]

Reason for the torch 1.4 is that I got this line:

pip install torch==1.4.0+cpu torchvision==0.5.0+cpu -f https://download.pytorch.org/whl/torch_stable.html

from https://pytorch.org/ if you select package=pip, CUDA=none

I tried to downgrade to torch v 1.1.0:

pip install -Iv torch==1.1.0
...
ERROR: torchvision 0.5.0+cpu has requirement torch==1.4.0, but you'll have torch 1.1.0 which is incompatible.

Tried to downgrade torchvision:

pip install torch==1.1.0+cpu torchvision==0.4.0+cpu -f https://download.pytorch.org/whl/torch_stable.html
Looking in links: https://download.pytorch.org/whl/torch_stable.html
ERROR: Could not find a version that satisfies the requirement torch==1.1.0+cpu (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2, 0.1.6.post17, 0.1.6.post20, 0.3.0.post4, 0.3.1, 0.4.0,
0.4.1, 1.0.0, 1.0.1, 1.0.1.post2, 1.1.0, 1.2.0, 1.2.0+cpu, 1.2.0+cu92, 1.3.0, 1.3.0+cpu, 1.3.0+cu100, 1.3.0+cu92, 1.3.1, 1.3.1+cpu, 1.3.1+cu100, 1.3.1+cu92, 1.4.0, 1.4.0+cpu, 
1.4.0+cu100,1.4.0+cu92)
ERROR: No matching distribution found for torch==1.1.0+cpu

Then tried to downgrade only torchvision but it upgraded torch to 1.2 which caused parse error.

pip install torchvision==0.4.0+cpu -f https://download.pytorch.org/whl/torch_stable.html

Installing collected packages: torch, torchvision
  Attempting uninstall: torch
    Found existing installation: torch 1.1.0
    Uninstalling torch-1.1.0:
      Successfully uninstalled torch-1.1.0
  Attempting uninstall: torchvision
    Found existing installation: torchvision 0.5.0+cpu
    Uninstalling torchvision-0.5.0+cpu:
      Successfully uninstalled torchvision-0.5.0+cpu
Successfully installed torch-1.2.0+cu92 torchvision-0.4.0+cpu

Then tried non-cpu version of torch

pip install torch==1.1.0 torchvision==0.4.0+cpu -f https://download.pytorch.org/whl/torch_stable.html

ERROR: torchvision 0.4.0+cpu has requirement torch==1.2.0, but you'll have torch 1.1.0 which is incompatible.
Installing collected packages: torch
  Attempting uninstall: torch
    Found existing installation: torch 1.2.0+cu92
    Uninstalling torch-1.2.0+cu92:
      Successfully uninstalled torch-1.2.0+cu92
Successfully installed torch-1.1.0

So there is a version mismatch but still managed to install it. Versions are now:

torch               1.1.0     
torchtext           0.3.1     
torchvision         0.4.0+cpu 

Other pipelines seem to work except parse_conllu and parse_conllu_nolemmas, which hang.

Last thing which I tried was to downgrade torchvision to 0.2.1 first with cpu and then without it:

pip install torch==1.1.0 torchvision==0.2.1+cpu -f https://download.pytorch.org/whl/torch_stable.html
Looking in links: https://download.pytorch.org/whl/torch_stable.html
Requirement already satisfied: torch==1.1.0 in ./venv-parser-neural/lib/python3.5/site-packages (1.1.0)
ERROR: Could not find a version that satisfies the requirement torchvision==0.2.1+cpu (from versions: 0.1.6, 0.1.7, 0.1.8, 0.1.9, 0.2.0, 0.2.1, 0.2.2, 0.2.2.post2, 0.2.2.post3, 0.3.0,     0.4.0, 0.4.0+cpu, 0.4.0+cu92, 0.4.1, 0.4.1+cpu, 0.4.1+cu100, 0.4.1+cu92, 0.4.2, 0.4.2+cpu, 0.4.2+cu100, 0.4.2+cu92, 0.5.0, 0.5.0+cpu, 0.5.0+cu100, 0.5.0+cu92)
ERROR: No matching distribution found for torchvision==0.2.1+cpu

pip install torch==1.1.0 torchvision==0.2.1 -f https://download.pytorch.org/whl/torch_stable.html
Looking in links: https://download.pytorch.org/whl/torch_stable.html
Requirement already satisfied: torch==1.1.0 in ./venv-parser-neural/lib/python3.5/site-packages (1.1.0)
Collecting torchvision==0.2.1
  Using cached torchvision-0.2.1-py2.py3-none-any.whl (54 kB)
Requirement already satisfied: numpy in ./venv-parser-neural/lib/python3.5/site-packages (from torch==1.1.0) (1.18.1)
Requirement already satisfied: pillow>=4.1.1 in ./venv-parser-neural/lib/python3.5/site-packages (from torchvision==0.2.1) (7.0.0)
Requirement already satisfied: six in ./venv-parser-neural/lib/python3.5/site-packages (from torchvision==0.2.1) (1.14.0)
Installing collected packages: torchvision
  Attempting uninstall: torchvision
    Found existing installation: torchvision 0.4.0+cpu
    Uninstalling torchvision-0.4.0+cpu:
      Successfully uninstalled torchvision-0.4.0+cpu
Successfully installed torchvision-0.2.1

pip3 list
Package             Version   
------------------- ----------
absl-py             0.9.0     
astor               0.8.1     
certifi             2019.11.28
chardet             3.0.4     
Click               7.0       
ConfigArgParse      1.0       
cycler              0.10.0    
Flask               1.1.1     
gast                0.3.3     
grpcio              1.27.1    
h5py                2.10.0    
idna                2.8       
itsdangerous        1.1.0     
Jinja2              2.11.1    
Keras               2.3.1     
Keras-Applications  1.0.8     
Keras-Preprocessing 1.1.0     
kiwisolver          1.1.0     
Markdown            3.2       
MarkupSafe          1.1.1     
matplotlib          3.0.3     
numpy               1.18.1    
Pillow              7.0.0     
pip                 20.0.2    
pkg-resources       0.0.0     
protobuf            3.11.3    
pyparsing           2.4.6     
python-dateutil     2.8.1     
PyYAML              5.3       
requests            2.22.0    
scipy               1.4.1     
setuptools          36.0.1    
six                 1.14.0    
tensorboard         1.12.2    
tensorflow          1.12.2    
termcolor           1.1.0     
torch               1.1.0     
torchtext           0.3.1     
torchvision         0.2.1     
tqdm                4.42.1    
ufal.udpipe         1.2.0.3   
urllib3             1.25.8    
Werkzeug            1.0.0     
wheel               0.34.2    
yolk3k              0.9       

This did not change the parsing results. Other pipelines seem to work except parse_conllu and parse_conllu_nolemmas, which hang. Output is here: server-output-parse_conllu.txt

Client command is:

curl --request POST --header 'Content-Type: text/plain; charset=utf-8' --data-binary "Minulla on koira. Tämä on esimerkkilause." http://localhost:7689

As you can see there happens "IndexError: list assignment index out of range" during the parse. This occurs only when running parse_conllu and parse_conllu_nolemmas pipelines.

So the best versions I found are as in the original proposal except torch is 1.1.0 instead of 0.4.1. But even with these versions parse_conllu and parse_conllu_nolemmas pipelines hang.

[fginter]

Hi

I tried to replicate this now on a fresh CPU-only Ubuntu 16.04 server VM. This should replicate your environment exactly. I followed the exact install instructions as are on the parser webpage (the only difference being I upgraded pip when asked to do so). I also got the markdown error, but I ignored it. This comes from flask, but we don't use markdown anywhere so should be a non-issue. Of course such error sucks, but Ubuntu 16.04 is ancient and such incompatibilities are likely to arise with time.

Anyway, on this setup I tested both the stream mode and the server mode. Everything worked as advertised.

My opinion is that your current install is broken by the various manual package upgrades/downgrades you did. Luckily it's easy to wipe your virtualenv and start from scratch (or use the docker). So I suggest you start from scratch and simply pip install the requirements-cpu and run the parser with "--gpu -1" and this should work, since it works for me on the exact same system. Do upgrade pip when it asks.

[edit:] cross-versioning the various deep learning libraries is always a recipe for trouble, because for instance pickled models don't necessarily work, etc. So I would not be surprised if this boils down to running with the wrong torch. Of course we should eventually upgrade the parser to a newer torch, but I suspect this might actually mean re-training all the models so they get saved correctly. Fighting these version issues is no fun.

Best

Filip

[jk-1]

You have a point there. Maybe I come back to this when I have a new installed Ubuntu 18 (or 20) VM. Fortunately parse_plaintext seems to work fine.

Thank you for your support!

[fginter]

OK. Good. When in doubt, do run the docker version, that should work out of the box.