argosopentech / argos-translate

Open-source offline translation library written in Python
https://www.argosopentech.com
MIT License
3.82k stars 279 forks source link

Failed to install Argos-translate in Lubuntu 22.04. #313

Open user0022 opened 1 year ago

user0022 commented 1 year ago

Hello! For my first post I just want to say thanks for your work. I hope I can use this software for offline translation, helping people with limited connexion or avoiding a bit more Google and other bad guys would be cool. Sorry for my english I'm french-speaker, and I don't have much skills in computing.

* 1 or 2 months ago I had already tried to install Argos (following installation page here on Github) but for some reason, several times after downloading hundreds of MB it aborted, I don't remember the details but well I gave up. Yesterday I tried again and it seemed to work better.

First I put pip install argostranslate:

me@me-pc:~$ pip install argostranslate
Defaulting to user installation because normal site-packages is not writeable
Collecting argostranslate
  Downloading argostranslate-1.7.5-py3-none-any.whl (27 kB)
Collecting sentencepiece==0.1.96
  Downloading sentencepiece-0.1.96-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 1.4 MB/s eta 0:00:00
Collecting ctranslate2==2.24.0
  Downloading ctranslate2-2.24.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (18.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 18.3/18.3 MB 1.2 MB/s eta 0:00:00
Collecting stanza==1.1.1
  Downloading stanza-1.1.1-py3-none-any.whl (227 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 227.6/227.6 KB 632.8 kB/s eta 0:00:00
Requirement already satisfied: pyyaml<7,>=5.3 in /usr/lib/python3/dist-packages (from ctranslate2==2.24.0->argostranslate) (5.4.1)
Requirement already satisfied: numpy in /usr/lib/python3/dist-packages (from ctranslate2==2.24.0->argostranslate) (1.21.5)
Collecting torch>=1.3.0
  Downloading torch-1.13.1-cp310-cp310-manylinux1_x86_64.whl (887.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 887.5/887.5 MB ? eta 0:00:00
Requirement already satisfied: protobuf in /usr/lib/python3/dist-packages (from stanza==1.1.1->argostranslate) (3.12.4)
Collecting tqdm
  Downloading tqdm-4.64.1-py2.py3-none-any.whl (78 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 78.5/78.5 KB 614.4 kB/s eta 0:00:00
Requirement already satisfied: requests in /usr/lib/python3/dist-packages (from stanza==1.1.1->argostranslate) (2.25.1)
Collecting nvidia-cudnn-cu11==8.5.0.96
  Downloading nvidia_cudnn_cu11-8.5.0.96-2-py3-none-manylinux1_x86_64.whl (557.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 557.1/557.1 MB ? eta 0:00:00
Collecting typing-extensions
  Downloading typing_extensions-4.4.0-py3-none-any.whl (26 kB)
Collecting nvidia-cuda-runtime-cu11==11.7.99
  Downloading nvidia_cuda_runtime_cu11-11.7.99-py3-none-manylinux1_x86_64.whl (849 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 849.3/849.3 KB 44.6 kB/s eta 0:00:00
Collecting nvidia-cuda-nvrtc-cu11==11.7.99
  Downloading nvidia_cuda_nvrtc_cu11-11.7.99-2-py3-none-manylinux1_x86_64.whl (21.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 21.0/21.0 MB 987.4 kB/s eta 0:00:00
Collecting nvidia-cublas-cu11==11.10.3.66
  Downloading nvidia_cublas_cu11-11.10.3.66-py3-none-manylinux1_x86_64.whl (317.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 317.1/317.1 MB 515.3 kB/s eta 0:00:00
Requirement already satisfied: setuptools in /usr/lib/python3/dist-packages (from nvidia-cublas-cu11==11.10.3.66->torch>=1.3.0->stanza==1.1.1->argostranslate) (59.6.0)
Requirement already satisfied: wheel in /usr/lib/python3/dist-packages (from nvidia-cublas-cu11==11.10.3.66->torch>=1.3.0->stanza==1.1.1->argostranslate) (0.37.1)
Installing collected packages: sentencepiece, typing-extensions, tqdm, nvidia-cuda-runtime-cu11, nvidia-cuda-nvrtc-cu11, nvidia-cublas-cu11, ctranslate2, nvidia-cudnn-cu11, torch, stanza, argostranslate
  WARNING: The script tqdm is installed in '/home/me/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
  WARNING: The scripts ct2-fairseq-converter, ct2-marian-converter, ct2-openai-gpt2-converter, ct2-opennmt-py-converter, ct2-opennmt-tf-converter, ct2-opus-mt-converter and ct2-transformers-converter are installed in '/home/me/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
ERROR: Could not install packages due to an OSError: [Errno 28] No space left on device

Following what said last line, I moved some files to make room and I put again pip install argostranslate:

me@me-pc:~$ pip install argostranslate
Defaulting to user installation because normal site-packages is not writeable
Collecting argostranslate
  Using cached argostranslate-1.7.5-py3-none-any.whl (27 kB)
Requirement already satisfied: ctranslate2==2.24.0 in ./.local/lib/python3.10/site-packages (from argostranslate) (2.24.0)
Collecting stanza==1.1.1
  Using cached stanza-1.1.1-py3-none-any.whl (227 kB)
Requirement already satisfied: sentencepiece==0.1.96 in ./.local/lib/python3.10/site-packages (from argostranslate) (0.1.96)
Requirement already satisfied: numpy in /usr/lib/python3/dist-packages (from ctranslate2==2.24.0->argostranslate) (1.21.5)
Requirement already satisfied: pyyaml<7,>=5.3 in /usr/lib/python3/dist-packages (from ctranslate2==2.24.0->argostranslate) (5.4.1)
Requirement already satisfied: protobuf in /usr/lib/python3/dist-packages (from stanza==1.1.1->argostranslate) (3.12.4)
Requirement already satisfied: requests in /usr/lib/python3/dist-packages (from stanza==1.1.1->argostranslate) (2.25.1)
Requirement already satisfied: torch>=1.3.0 in ./.local/lib/python3.10/site-packages (from stanza==1.1.1->argostranslate) (1.13.1)
Requirement already satisfied: tqdm in ./.local/lib/python3.10/site-packages (from stanza==1.1.1->argostranslate) (4.64.1)
Requirement already satisfied: nvidia-cudnn-cu11==8.5.0.96 in ./.local/lib/python3.10/site-packages (from torch>=1.3.0->stanza==1.1.1->argostranslate) (8.5.0.96)
Requirement already satisfied: nvidia-cuda-nvrtc-cu11==11.7.99 in ./.local/lib/python3.10/site-packages (from torch>=1.3.0->stanza==1.1.1->argostranslate) (11.7.99)
Requirement already satisfied: nvidia-cublas-cu11==11.10.3.66 in ./.local/lib/python3.10/site-packages (from torch>=1.3.0->stanza==1.1.1->argostranslate) (11.10.3.66)
Requirement already satisfied: typing-extensions in ./.local/lib/python3.10/site-packages (from torch>=1.3.0->stanza==1.1.1->argostranslate) (4.4.0)
Requirement already satisfied: nvidia-cuda-runtime-cu11==11.7.99 in ./.local/lib/python3.10/site-packages (from torch>=1.3.0->stanza==1.1.1->argostranslate) (11.7.99)
Requirement already satisfied: setuptools in /usr/lib/python3/dist-packages (from nvidia-cublas-cu11==11.10.3.66->torch>=1.3.0->stanza==1.1.1->argostranslate) (59.6.0)
Requirement already satisfied: wheel in /usr/lib/python3/dist-packages (from nvidia-cublas-cu11==11.10.3.66->torch>=1.3.0->stanza==1.1.1->argostranslate) (0.37.1)
Installing collected packages: stanza, argostranslate
Successfully installed argostranslate-1.7.5 stanza-1.1.1
me@me-pc:~$

And then the GUI:

me@me-pc:~$ pip install argostranslategui
Defaulting to user installation because normal site-packages is not writeable
Collecting argostranslategui
  Downloading argostranslategui-1.6.3-py3-none-any.whl (6.7 kB)
Requirement already satisfied: argostranslate in ./.local/lib/python3.10/site-packages (from argostranslategui) (1.7.5)
Collecting PyQt5==5.15.4
  Downloading PyQt5-5.15.4-cp36.cp37.cp38.cp39-abi3-manylinux2014_x86_64.whl (8.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8.3/8.3 MB 1.1 MB/s eta 0:00:00
Requirement already satisfied: PyQt5-sip<13,>=12.8 in /usr/lib/python3/dist-packages (from PyQt5==5.15.4->argostranslategui) (12.9.1)
Collecting PyQt5-Qt5>=5.15
  Downloading PyQt5_Qt5-5.15.2-py3-none-manylinux2014_x86_64.whl (59.9 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 59.9/59.9 MB 1.0 MB/s eta 0:00:00
Requirement already satisfied: ctranslate2==2.24.0 in ./.local/lib/python3.10/site-packages (from argostranslate->argostranslategui) (2.24.0)
Requirement already satisfied: stanza==1.1.1 in ./.local/lib/python3.10/site-packages (from argostranslate->argostranslategui) (1.1.1)
Requirement already satisfied: sentencepiece==0.1.96 in ./.local/lib/python3.10/site-packages (from argostranslate->argostranslategui) (0.1.96)
Requirement already satisfied: pyyaml<7,>=5.3 in /usr/lib/python3/dist-packages (from ctranslate2==2.24.0->argostranslate->argostranslategui) (5.4.1)
Requirement already satisfied: numpy in /usr/lib/python3/dist-packages (from ctranslate2==2.24.0->argostranslate->argostranslategui) (1.21.5)
Requirement already satisfied: requests in /usr/lib/python3/dist-packages (from stanza==1.1.1->argostranslate->argostranslategui) (2.25.1)
Requirement already satisfied: protobuf in /usr/lib/python3/dist-packages (from stanza==1.1.1->argostranslate->argostranslategui) (3.12.4)
Requirement already satisfied: tqdm in ./.local/lib/python3.10/site-packages (from stanza==1.1.1->argostranslate->argostranslategui) (4.64.1)
Requirement already satisfied: torch>=1.3.0 in ./.local/lib/python3.10/site-packages (from stanza==1.1.1->argostranslate->argostranslategui) (1.13.1)
Requirement already satisfied: nvidia-cublas-cu11==11.10.3.66 in ./.local/lib/python3.10/site-packages (from torch>=1.3.0->stanza==1.1.1->argostranslate->argostranslategui) (11.10.3.66)
Requirement already satisfied: nvidia-cuda-runtime-cu11==11.7.99 in ./.local/lib/python3.10/site-packages (from torch>=1.3.0->stanza==1.1.1->argostranslate->argostranslategui) (11.7.99)
Requirement already satisfied: typing-extensions in ./.local/lib/python3.10/site-packages (from torch>=1.3.0->stanza==1.1.1->argostranslate->argostranslategui) (4.4.0)
Requirement already satisfied: nvidia-cuda-nvrtc-cu11==11.7.99 in ./.local/lib/python3.10/site-packages (from torch>=1.3.0->stanza==1.1.1->argostranslate->argostranslategui) (11.7.99)
Requirement already satisfied: nvidia-cudnn-cu11==8.5.0.96 in ./.local/lib/python3.10/site-packages (from torch>=1.3.0->stanza==1.1.1->argostranslate->argostranslategui) (8.5.0.96)
Requirement already satisfied: setuptools in /usr/lib/python3/dist-packages (from nvidia-cublas-cu11==11.10.3.66->torch>=1.3.0->stanza==1.1.1->argostranslate->argostranslategui) (59.6.0)
Requirement already satisfied: wheel in /usr/lib/python3/dist-packages (from nvidia-cublas-cu11==11.10.3.66->torch>=1.3.0->stanza==1.1.1->argostranslate->argostranslategui) (0.37.1)
Installing collected packages: PyQt5-Qt5, PyQt5, argostranslategui
  WARNING: The scripts pylupdate5, pyrcc5 and pyuic5 are installed in '/home/me/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed PyQt5-5.15.4 PyQt5-Qt5-5.15.2 argostranslategui-1.6.3
me@me-pc:~$

Last lines of the 2 outputs say "successfully installed". But well I couldn't find Argos in main Menu, and running argos-translate in terminal said something like "command unknown" if I remember.

There is a maybe problematic line in the beginning Defaulting to user installation because normal site-packages is not writeable But I don't know what to do with it so I left it.

But there are also several lines

WARNING: The scripts (blahblah...) are installed in '/home/me/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.

Had never heard of this "PATH" thing, but I found here for example that I seemed to have to add a line in /home/me/.bashrc file, which I did: export PATH="$PATH:/home/me/.local/bin"

After that, I made a small chek by re-running install command:

me@me-pc:~$ pip install argostranslate
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: argostranslate in ./.local/lib/python3.10/site-packages (1.7.5)
Requirement already satisfied: ctranslate2==2.24.0 in ./.local/lib/python3.10/site-packages (from argostranslate) (2.24.0)
Requirement already satisfied: stanza==1.1.1 in ./.local/lib/python3.10/site-packages (from argostranslate) (1.1.1)
Requirement already satisfied: sentencepiece==0.1.96 in ./.local/lib/python3.10/site-packages (from argostranslate) (0.1.96)
Requirement already satisfied: numpy in /usr/lib/python3/dist-packages (from ctranslate2==2.24.0->argostranslate) (1.21.5)
Requirement already satisfied: pyyaml<7,>=5.3 in /usr/lib/python3/dist-packages (from ctranslate2==2.24.0->argostranslate) (5.4.1)
Requirement already satisfied: tqdm in ./.local/lib/python3.10/site-packages (from stanza==1.1.1->argostranslate) (4.64.1)
Requirement already satisfied: requests in /usr/lib/python3/dist-packages (from stanza==1.1.1->argostranslate) (2.25.1)
Requirement already satisfied: torch>=1.3.0 in ./.local/lib/python3.10/site-packages (from stanza==1.1.1->argostranslate) (1.13.1)
Requirement already satisfied: protobuf in /usr/lib/python3/dist-packages (from stanza==1.1.1->argostranslate) (3.12.4)
Requirement already satisfied: nvidia-cuda-nvrtc-cu11==11.7.99 in ./.local/lib/python3.10/site-packages (from torch>=1.3.0->stanza==1.1.1->argostranslate) (11.7.99)
Requirement already satisfied: nvidia-cublas-cu11==11.10.3.66 in ./.local/lib/python3.10/site-packages (from torch>=1.3.0->stanza==1.1.1->argostranslate) (11.10.3.66)
Requirement already satisfied: typing-extensions in ./.local/lib/python3.10/site-packages (from torch>=1.3.0->stanza==1.1.1->argostranslate) (4.4.0)
Requirement already satisfied: nvidia-cudnn-cu11==8.5.0.96 in ./.local/lib/python3.10/site-packages (from torch>=1.3.0->stanza==1.1.1->argostranslate) (8.5.0.96)
Requirement already satisfied: nvidia-cuda-runtime-cu11==11.7.99 in ./.local/lib/python3.10/site-packages (from torch>=1.3.0->stanza==1.1.1->argostranslate) (11.7.99)
Requirement already satisfied: setuptools in /usr/lib/python3/dist-packages (from nvidia-cublas-cu11==11.10.3.66->torch>=1.3.0->stanza==1.1.1->argostranslate) (59.6.0)
Requirement already satisfied: wheel in /usr/lib/python3/dist-packages (from nvidia-cublas-cu11==11.10.3.66->torch>=1.3.0->stanza==1.1.1->argostranslate) (0.37.1)
me@me-pc:~$

Warnings had disappeared, which I thought was a good sign. But Argos still not in Menu, and when I ran argos-translate or argos-translate-gui in Terminal, answer was: Illegal instruction (core dumped) What does it mean??

So I tried to follow the found web pages, and then I added the indicated folder for the PATH in 2 other files: PATH="$PATH:/home/me/.local/bin" in /home/me/.profile and in /etc/environment and I restarted the computer. But then surprise, at session opening, I coudn't enter, cause after each confirmation of my password, the login page displayed again and asked me again the password! So from another OS, I entered my partition and I commented with # the 2 lines I had added. And then I could re-open my session normally.

Then I tried once more to add the folder in PATH, in etc/environment there is a line, in which I added it to the end: PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/me/.local/bin" And then I restarted, but still the same (I keep this modification for the moment).

I see yet 10 things to try, but it might take hours or days and might finally work or break my partition, so if someone has the right answer it would be great.

milahu commented 1 year ago

when I ran argos-translate or argos-translate-gui in Terminal, answer was: Illegal instruction (core dumped) What does it mean??

the binary builds of the nvidia cuda packages fail to run on your hardware, either because some CPU extensions are missing, for example AVX2 (grep -m1 ^flags /proc/cpuinfo)

you could build these package from source, to make them run on your hardware. this may take 1 or 2 days

see also https://github.com/pytorch/pytorch/issues/2714#issuecomment-329196030

Last lines of the 2 outputs say "successfully installed". But well I couldn't find Argos in main Menu

because argos-translate is a command line app, usually called with python3 -m argostranslate, or by adding ~/.local/bin to PATH

for argos-translate-gui, you could add a ~/.local/share/applications/argos-translate-gui.desktop file

user0022 commented 1 year ago

Thanks for answer!

In fact I had had an hypothesis that the problem comes from my too old CPU, which could coincide with what you said: "either because some CPU extensions are missing, for example AVX2 (grep -m1 ^flags /proc/cpuinfo)". What made me think about it is that:

Then I verified in CPU-X software. CPU tab (default tab) > Instructions line > Show: MMX, SSE(1, 2, 3, 3S), x86-64 So SSE4 is missing, probably because my computer is a bit too old. So this extension won't work, and as the goal of this extension and of LibreTranslate and ArgosTranslate are the same, they work maybe the same way at this level and all need this "SSE4" instruction.

Your command shows effectively that for example AVX2 is missing:

grep -m1 ^flags /proc/cpuinfo
flags        : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good nopl cpuid aperfmperf pni dtes64 monitor ds_cpl est tm2 ssse3 cx16 xtpr pdcm lahf_lm pti dtherm

* Yes why not try to build missing extensions (even if for now I don't know how to build a package but it doesn't seem too hard). But yet a question, when you say "1 or 2 days", it means mainly the time the computer will work alone, or the time I will need to work on building? For if it is the second case, oops as I'm not technician at all, I fear it would rather mean "1 or 2 weeks" for me.

* For the moment I didn't try to run python3 -m argostranslate or add ~/.local/bin to PATH because trying to install LibreTranslate not only didn't work but also put the mess in my OS: CPUs always at 100% when opening my session (which fortunately I found how to stop at each opening), and even a 2nd account created named "LibreTranslate"! So I'm a bit afraid to try new things.

* Before closing I checked the link you gave (just fast, not real reading), and surprise it talks about the "SSE4" thing. Though I had made a search in the Github search bar and there was no result. But now I see it seems to search only in the code. I should have rather tried site:github.com/argosopentech/argos-translate/ "sse4" in "a search engine", but it gives only 1 result, and being about a different issue. Maybe better searches would have given me solutions.

* In fact unfortunately I miss time to try further by now. I'll try again when more possible!

* Another hypothesis: maybe Argos-translate (and other software who have the same problem, if it is the case) should explain this in their presentation, like the Firefox addon does? For maybe I will not be the only one in the world with a 2007 computer, 16 years old but well 64b! :).

milahu commented 1 year ago

"A CPU that supports SSE4.1 extensions is required for this addon to function properly."

yeah, the required extension seems to be sse4, and maybe avx

it works on my cpu:

$ cat /proc/cpuinfo | grep -m1 ^flag | grep -o -E "(sse|avx)[^ ]*" 
sse
sse2
sse3
sse4_1
sse4_2
avx

when you say "1 or 2 days", it means mainly the time the computer will work alone, or the time I will need to work on building?

the compile time will be in that range. you only copy-paste some commands and wait, and hope that your cpu cooler works ; )

argos-translate depends on pytorch, so...

user0022 commented 12 months ago

Thanks again for explanations! So I'll try all this as soon as I can and tell result. Well 1 or 2 days of big CPU usage, I'm ready to say goodbye to my grandpa laptop, but at least I will be able to cook some eggs :).

And another hypothesis, maybe also the Firefox addon should have then added in its warning "If your CPU doesn't, follow those instructions", or with a 327000 users addons, they missed maybe tens or hundreds of other users. But now the addon is said stopped https://github.com/mozilla/firefox-translations and included to FF itself since FF108, I don't see it in mine (ESR, 115) but maybe juste because it doesn't work for me. Proposing users to follow compiling instructions "in Firefox itself" being still a possibility. Maybe I will ask them about that one day.