Closed jk-1 closed 4 years ago
Hi thanks for the detailed report. Try running the pipeline with --gpu -1
that should make sure no attempt at GPU is made. Do tell if this solved your problem. This is mentioned here http://turkunlp.org/Turku-neural-parser-pipeline/ but maybe we should make it more clear, how to separate the CPU and GPU stuff.
right, and you should not remove the lemmatizer_mod because that will only fall-back on lemma cache and unknown words won't get lemmatized, which is probably not what you want.
Now running in non-server mode is working:
Running in server mode gives following error:
Client command was:
curl --request POST --header 'Content-Type: text/plain; charset=utf-8' --data-binary "Tämä on esimerkkilause. Minulla on koira. " http://localhost:7689
Never seen this error. :confused: the difference as far as I can tell is in the text. In the former case lemmatization happens from cache and the lemmatizer does not invoke, in the latter case the input is a little longer and the lemmatizer actually invokes and crashes. My bet is this must have something to do with versions. The lemmatizer is PyTorch, so I would look there first and try to downgrade torch few minor versions down. I will also do some tests myself, see if I can replicate the bug.
You are right. Also non-server fails if the sentence is same/longer.
Here are my versions:
(venv-parser-neural) jk@mypc:~/Turku-neural-parser-pipeline$ pip list
Package Version
------------------- ----------
absl-py 0.9.0
astor 0.8.1
certifi 2019.11.28
chardet 3.0.4
Click 7.0
ConfigArgParse 1.0
cycler 0.10.0
Flask 1.1.1
gast 0.3.3
grpcio 1.27.1
h5py 2.10.0
idna 2.8
itsdangerous 1.1.0
Jinja2 2.11.1
Keras 2.3.1
Keras-Applications 1.0.8
Keras-Preprocessing 1.1.0
kiwisolver 1.1.0
Markdown 3.2
MarkupSafe 1.1.1
matplotlib 3.0.3
numpy 1.18.1
Pillow 7.0.0
pip 20.0.2
pkg-resources 0.0.0
protobuf 3.11.3
pyparsing 2.4.6
python-dateutil 2.8.1
PyYAML 5.3
requests 2.22.0
scipy 1.4.1
setuptools 20.7.0
six 1.14.0
tensorboard 1.12.2
tensorflow 1.12.2
termcolor 1.1.0
torch 1.4.0+cpu
torchtext 0.3.1
torchvision 0.5.0+cpu
tqdm 4.42.1
ufal.udpipe 1.2.0.3
urllib3 1.25.8
Werkzeug 1.0.0
wheel 0.34.2
(venv-parser-neural) jk@mypc:~/Turku-neural-parser-pipeline$ python --version
Python 3.5.2
(venv-parser-neural) jk@mypc:~/Turku-neural-parser-pipeline$ pip3 --version
pip 20.0.2 from /home/jk/Turku-neural-parser-pipeline/venv-parser-neural/lib/python3.5/site-packages/pip (python 3.5)
(venv-parser-neural) jk@mypc:~/Turku-neural-parser-pipeline$
As I said in the first message the initial install complained about the setuptools version:
ERROR: markdown 3.2 has requirement setuptools>=36, but you'll have setuptools 20.7.0 which is incompatible.
I managed to upgrade setup tools to 36.0.1, but the parsing error is still the same.
Versions now:
(venv-parser-neural) jk@mypc:~/Turku-neural-parser-pipeline$ pip list
Package Version
------------------- ----------
absl-py 0.9.0
astor 0.8.1
certifi 2019.11.28
chardet 3.0.4
Click 7.0
ConfigArgParse 1.0
cycler 0.10.0
Flask 1.1.1
gast 0.3.3
grpcio 1.27.1
h5py 2.10.0
idna 2.8
itsdangerous 1.1.0
Jinja2 2.11.1
Keras 2.3.1
Keras-Applications 1.0.8
Keras-Preprocessing 1.1.0
kiwisolver 1.1.0
Markdown 3.2
MarkupSafe 1.1.1
matplotlib 3.0.3
numpy 1.18.1
Pillow 7.0.0
pip 20.0.2
pkg-resources 0.0.0
protobuf 3.11.3
pyparsing 2.4.6
python-dateutil 2.8.1
PyYAML 5.3
requests 2.22.0
scipy 1.4.1
setuptools 36.0.1
six 1.14.0
tensorboard 1.12.2
tensorflow 1.12.2
termcolor 1.1.0
torch 1.4.0+cpu
torchtext 0.3.1
torchvision 0.5.0+cpu
tqdm 4.42.1
ufal.udpipe 1.2.0.3
urllib3 1.25.8
Werkzeug 1.0.0
wheel 0.34.2
yolk3k 0.9
(venv-parser-neural) jk@mypc:~/Turku-neural-parser-pipeline$
Hi,
Try downgrading your pytorch to 1.1.0. Apparently this mask inversion syntax was changed in 1.2.0. The requirements-cpu.txt file in this repository specifies torch==0.4.1
and your version is quite a lot newer than that.
Reason for the torch 1.4 is that I got this line:
pip install torch==1.4.0+cpu torchvision==0.5.0+cpu -f https://download.pytorch.org/whl/torch_stable.html
from https://pytorch.org/ if you select package=pip, CUDA=none
I tried to downgrade to torch v 1.1.0:
pip install -Iv torch==1.1.0
...
ERROR: torchvision 0.5.0+cpu has requirement torch==1.4.0, but you'll have torch 1.1.0 which is incompatible.
Tried to downgrade torchvision:
pip install torch==1.1.0+cpu torchvision==0.4.0+cpu -f https://download.pytorch.org/whl/torch_stable.html
Looking in links: https://download.pytorch.org/whl/torch_stable.html
ERROR: Could not find a version that satisfies the requirement torch==1.1.0+cpu (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2, 0.1.6.post17, 0.1.6.post20, 0.3.0.post4, 0.3.1, 0.4.0,
0.4.1, 1.0.0, 1.0.1, 1.0.1.post2, 1.1.0, 1.2.0, 1.2.0+cpu, 1.2.0+cu92, 1.3.0, 1.3.0+cpu, 1.3.0+cu100, 1.3.0+cu92, 1.3.1, 1.3.1+cpu, 1.3.1+cu100, 1.3.1+cu92, 1.4.0, 1.4.0+cpu,
1.4.0+cu100,1.4.0+cu92)
ERROR: No matching distribution found for torch==1.1.0+cpu
Then tried to downgrade only torchvision but it upgraded torch to 1.2 which caused parse error.
pip install torchvision==0.4.0+cpu -f https://download.pytorch.org/whl/torch_stable.html
Installing collected packages: torch, torchvision
Attempting uninstall: torch
Found existing installation: torch 1.1.0
Uninstalling torch-1.1.0:
Successfully uninstalled torch-1.1.0
Attempting uninstall: torchvision
Found existing installation: torchvision 0.5.0+cpu
Uninstalling torchvision-0.5.0+cpu:
Successfully uninstalled torchvision-0.5.0+cpu
Successfully installed torch-1.2.0+cu92 torchvision-0.4.0+cpu
Then tried non-cpu version of torch
pip install torch==1.1.0 torchvision==0.4.0+cpu -f https://download.pytorch.org/whl/torch_stable.html
ERROR: torchvision 0.4.0+cpu has requirement torch==1.2.0, but you'll have torch 1.1.0 which is incompatible.
Installing collected packages: torch
Attempting uninstall: torch
Found existing installation: torch 1.2.0+cu92
Uninstalling torch-1.2.0+cu92:
Successfully uninstalled torch-1.2.0+cu92
Successfully installed torch-1.1.0
So there is a version mismatch but still managed to install it. Versions are now:
torch 1.1.0
torchtext 0.3.1
torchvision 0.4.0+cpu
Other pipelines seem to work except parse_conllu and parse_conllu_nolemmas, which hang.
Last thing which I tried was to downgrade torchvision to 0.2.1 first with cpu and then without it:
pip install torch==1.1.0 torchvision==0.2.1+cpu -f https://download.pytorch.org/whl/torch_stable.html
Looking in links: https://download.pytorch.org/whl/torch_stable.html
Requirement already satisfied: torch==1.1.0 in ./venv-parser-neural/lib/python3.5/site-packages (1.1.0)
ERROR: Could not find a version that satisfies the requirement torchvision==0.2.1+cpu (from versions: 0.1.6, 0.1.7, 0.1.8, 0.1.9, 0.2.0, 0.2.1, 0.2.2, 0.2.2.post2, 0.2.2.post3, 0.3.0, 0.4.0, 0.4.0+cpu, 0.4.0+cu92, 0.4.1, 0.4.1+cpu, 0.4.1+cu100, 0.4.1+cu92, 0.4.2, 0.4.2+cpu, 0.4.2+cu100, 0.4.2+cu92, 0.5.0, 0.5.0+cpu, 0.5.0+cu100, 0.5.0+cu92)
ERROR: No matching distribution found for torchvision==0.2.1+cpu
pip install torch==1.1.0 torchvision==0.2.1 -f https://download.pytorch.org/whl/torch_stable.html
Looking in links: https://download.pytorch.org/whl/torch_stable.html
Requirement already satisfied: torch==1.1.0 in ./venv-parser-neural/lib/python3.5/site-packages (1.1.0)
Collecting torchvision==0.2.1
Using cached torchvision-0.2.1-py2.py3-none-any.whl (54 kB)
Requirement already satisfied: numpy in ./venv-parser-neural/lib/python3.5/site-packages (from torch==1.1.0) (1.18.1)
Requirement already satisfied: pillow>=4.1.1 in ./venv-parser-neural/lib/python3.5/site-packages (from torchvision==0.2.1) (7.0.0)
Requirement already satisfied: six in ./venv-parser-neural/lib/python3.5/site-packages (from torchvision==0.2.1) (1.14.0)
Installing collected packages: torchvision
Attempting uninstall: torchvision
Found existing installation: torchvision 0.4.0+cpu
Uninstalling torchvision-0.4.0+cpu:
Successfully uninstalled torchvision-0.4.0+cpu
Successfully installed torchvision-0.2.1
pip3 list
Package Version
------------------- ----------
absl-py 0.9.0
astor 0.8.1
certifi 2019.11.28
chardet 3.0.4
Click 7.0
ConfigArgParse 1.0
cycler 0.10.0
Flask 1.1.1
gast 0.3.3
grpcio 1.27.1
h5py 2.10.0
idna 2.8
itsdangerous 1.1.0
Jinja2 2.11.1
Keras 2.3.1
Keras-Applications 1.0.8
Keras-Preprocessing 1.1.0
kiwisolver 1.1.0
Markdown 3.2
MarkupSafe 1.1.1
matplotlib 3.0.3
numpy 1.18.1
Pillow 7.0.0
pip 20.0.2
pkg-resources 0.0.0
protobuf 3.11.3
pyparsing 2.4.6
python-dateutil 2.8.1
PyYAML 5.3
requests 2.22.0
scipy 1.4.1
setuptools 36.0.1
six 1.14.0
tensorboard 1.12.2
tensorflow 1.12.2
termcolor 1.1.0
torch 1.1.0
torchtext 0.3.1
torchvision 0.2.1
tqdm 4.42.1
ufal.udpipe 1.2.0.3
urllib3 1.25.8
Werkzeug 1.0.0
wheel 0.34.2
yolk3k 0.9
This did not change the parsing results. Other pipelines seem to work except parse_conllu and parse_conllu_nolemmas, which hang. Output is here: server-output-parse_conllu.txt
Client command is:
curl --request POST --header 'Content-Type: text/plain; charset=utf-8' --data-binary "Minulla on koira. Tämä on esimerkkilause." http://localhost:7689
As you can see there happens "IndexError: list assignment index out of range" during the parse. This occurs only when running parse_conllu and parse_conllu_nolemmas pipelines.
So the best versions I found are as in the original proposal except torch is 1.1.0 instead of 0.4.1. But even with these versions parse_conllu and parse_conllu_nolemmas pipelines hang.
Hi
I tried to replicate this now on a fresh CPU-only Ubuntu 16.04 server VM. This should replicate your environment exactly. I followed the exact install instructions as are on the parser webpage (the only difference being I upgraded pip when asked to do so). I also got the markdown error, but I ignored it. This comes from flask, but we don't use markdown anywhere so should be a non-issue. Of course such error sucks, but Ubuntu 16.04 is ancient and such incompatibilities are likely to arise with time.
Anyway, on this setup I tested both the stream mode and the server mode. Everything worked as advertised.
My opinion is that your current install is broken by the various manual package upgrades/downgrades you did. Luckily it's easy to wipe your virtualenv and start from scratch (or use the docker). So I suggest you start from scratch and simply pip install the requirements-cpu and run the parser with "--gpu -1" and this should work, since it works for me on the exact same system. Do upgrade pip when it asks.
[edit:] cross-versioning the various deep learning libraries is always a recipe for trouble, because for instance pickled models don't necessarily work, etc. So I would not be surprised if this boils down to running with the wrong torch. Of course we should eventually upgrade the parser to a newer torch, but I suspect this might actually mean re-training all the models so they get saved correctly. Fighting these version issues is no fun.
Best
Filip
You have a point there. Maybe I come back to this when I have a new installed Ubuntu 18 (or 20) VM. Fortunately parse_plaintext seems to work fine.
Thank you for your support!
OK. Good. When in doubt, do run the docker version, that should work out of the box.
My environment is Virtualbox virtual machine with Ubuntu 16.04, 12GB mem, 8 CPUs/cores.
I installed TurkuNLP with your instructions. During the installation I got following error:
So it seems that markdown was installed ok despite the setuptools version error.
First run gave errors: CUDA driver version is insufficient for CUDA runtime version
I fixed this by reinstalling torch with no gpu settings
After that few pipelines seem to work. However, pipelines with command:
hang.
When running full pipleline server with command
and client with client command:
and with your original plain_text pipeline, the server hangs:
When running with modified pipeline
the same command, output is:
How to fix this issue? Is it possible that the setuptools version mismatch has something to do with this error in lemmatizer?
Thank you for your support.