docker-compose installation fails

[guillaume-be]

Hello,

I am trying to install Opus-MT via the docker-compose solution, but I am encountering an error during the compilation step:

Marian compilation error

``` In file included from /usr/src/app/marian/src/3rd_party/sentencepiece/src/unigram_model.h:24, from /usr/src/app/marian/src/3rd_party/sentencepiece/src/unigram_model.cc:15: /usr/src/app/marian/src/3rd_party/sentencepiece/src/freelist.h: In instantiation of 'T* sentencepiece::model::FreeList::Allocate() [with T = sentencepiece::unigram::Lattice::Node]': /usr/src/app/marian/src/3rd_party/sentencepiece/src/unigram_model.cc:83:41: required from here /usr/src/app/marian/src/3rd_party/sentencepiece/src/freelist.h:62:13: warning: 'void* memset(void*, int, size_t)' clearing an object of non-trivial type 'struct sentencepiece::unigram::Lattice::Node'; use assignment or value-initialization instead [-Wclass-memaccess] memset(chunk, 0, sizeof(*chunk) * chunk_size_); ~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ In file included from /usr/src/app/marian/src/3rd_party/sentencepiece/src/unigram_model.cc:15: /usr/src/app/marian/src/3rd_party/sentencepiece/src/unigram_model.h:38:10: note: 'struct sentencepiece::unigram::Lattice::Node' declared here struct Node { ^~~~ In file included from /usr/src/app/marian/src/3rd_party/sentencepiece/src/unigram_model.h:24, from /usr/src/app/marian/src/3rd_party/sentencepiece/src/unigram_model.cc:15: /usr/src/app/marian/src/3rd_party/sentencepiece/src/freelist.h: In instantiation of 'void sentencepiece::model::FreeList::Free() [with T = sentencepiece::unigram::Lattice::Node]': /usr/src/app/marian/src/3rd_party/sentencepiece/src/unigram_model.cc:93:24: required from here /usr/src/app/marian/src/3rd_party/sentencepiece/src/freelist.h:39:13: warning: 'void* memset(void*, int, size_t)' clearing an object of non-trivial type 'struct sentencepiece::unigram::Lattice::Node'; use assignment or value-initialization instead [-Wclass-memaccess] memset(chunk, 0, sizeof(*chunk) * chunk_size_); ~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ In file included from /usr/src/app/marian/src/3rd_party/sentencepiece/src/unigram_model.cc:15: /usr/src/app/marian/src/3rd_party/sentencepiece/src/unigram_model.h:38:10: note: 'struct sentencepiece::unigram::Lattice::Node' declared here struct Node { ^~~~ ```

Python package installation error

``` ---------------------------------------- Failed building wheel for tornado Running setup.py clean for tornado Running setup.py bdist_wheel for mosestokenizer: started Running setup.py bdist_wheel for mosestokenizer: finished with status 'error' Complete output from command /usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-lughb366/mosestokenizer/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-thswx0b2 --python-tag cp37: usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...] or: -c --help [cmd1 cmd2 ...] or: -c --help-commands or: -c cmd --help error: invalid command 'bdist_wheel' ---------------------------------------- Failed building wheel for mosestokenizer Running setup.py clean for mosestokenizer Running setup.py bdist_wheel for pycld2: started Running setup.py bdist_wheel for pycld2: finished with status 'error' Complete output from command /usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-lughb366/pycld2/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-v5m0lqhx --python-tag cp37: usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...] or: -c --help [cmd1 cmd2 ...] or: -c --help-commands or: -c cmd --help error: invalid command 'bdist_wheel' ---------------------------------------- Failed building wheel for pycld2 Running setup.py clean for pycld2 Running setup.py bdist_wheel for sqlitedict: started Running setup.py bdist_wheel for sqlitedict: finished with status 'error' Complete output from command /usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-lughb366/sqlitedict/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-8em57psk --python-tag cp37: usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...] or: -c --help [cmd1 cmd2 ...] or: -c --help-commands or: -c cmd --help error: invalid command 'bdist_wheel' ---------------------------------------- Failed building wheel for sqlitedict Running setup.py clean for sqlitedict Running setup.py bdist_wheel for docopt: started Running setup.py bdist_wheel for docopt: finished with status 'error' Complete output from command /usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-lughb366/docopt/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-1536sh2e --python-tag cp37: usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...] or: -c --help [cmd1 cmd2 ...] or: -c --help-commands or: -c cmd --help error: invalid command 'bdist_wheel' ---------------------------------------- Failed building wheel for docopt Running setup.py clean for docopt Running setup.py bdist_wheel for toolwrapper: started Running setup.py bdist_wheel for toolwrapper: finished with status 'error' Complete output from command /usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-lughb366/toolwrapper/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-svf3gk7k --python-tag cp37: usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...] or: -c --help [cmd1 cmd2 ...] or: -c --help-commands or: -c cmd --help error: invalid command 'bdist_wheel' ---------------------------------------- Failed building wheel for toolwrapper Running setup.py clean for toolwrapper Running setup.py bdist_wheel for uctools: started Running setup.py bdist_wheel for uctools: finished with status 'error' Complete output from command /usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-lughb366/uctools/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-i2p1v3j0 --python-tag cp37: usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...] or: -c --help [cmd1 cmd2 ...] or: -c --help-commands or: -c cmd --help error: invalid command 'bdist_wheel' ---------------------------------------- Failed building wheel for uctools Running setup.py clean for uctools Failed to build tornado mosestokenizer pycld2 sqlitedict docopt toolwrapper uctools Installing collected packages: tornado, docopt, openfile, toolwrapper, uctools, mosestokenizer, pycld2, sqlitedict, sentencepiece Running setup.py install for tornado: started Running setup.py install for tornado: finished with status 'done' Running setup.py install for docopt: started Running setup.py install for docopt: finished with status 'done' Running setup.py install for toolwrapper: started Running setup.py install for toolwrapper: finished with status 'done' Running setup.py install for uctools: started Running setup.py install for uctools: finished with status 'done' Running setup.py install for mosestokenizer: started Running setup.py install for mosestokenizer: finished with status 'done' Running setup.py install for pycld2: started Running setup.py install for pycld2: finished with status 'done' Running setup.py install for sqlitedict: started Running setup.py install for sqlitedict: finished with status 'done' ```

Runtime error

``` λ docker-compose up Creating network "opus-mt_default" with the default driver Starting opus-mt_opus-mt_1 ... done Attaching to opus-mt_opus-mt_1 opus-mt_1 | marian-server: error while loading shared libraries: libsentencepiece_train.so.0: cannot open shared object file: No such file or directory opus-mt_1 | ERROR:asyncio:Future exception was never retrieved opus-mt_1 | future: opus-mt_1 | Traceback (most recent call last): opus-mt_1 | File "/usr/local/lib/python3.7/dist-packages/tornado/gen.py", line 742, in run opus-mt_1 | yielded = self.gen.throw(*exc_info) # type: ignore opus-mt_1 | File "server.py", line 49, in run opus-mt_1 | ret = yield self.p.wait_for_exit() opus-mt_1 | File "/usr/local/lib/python3.7/dist-packages/tornado/gen.py", line 735, in run opus-mt_1 | value = future.result() opus-mt_1 | subprocess.CalledProcessError: Command 'unknown' returned non-zero exit status 127. ```

I am running Windows 10 with Docker on a WSL2 backend. Is there any pre-requisite step in order to build the container? Thank you,

[thecarmo]

Hey,

I'm also trying to run the dockerized version of the service and just had the same issues. I believe there are some problems in the Dockerfile which I tried to solve as follows:

• It seems that the python-slim base image removes gcc in order to maintain its size https://github.com/docker-library/python/issues/60 -> I just reversed to normal base image, that solved the "unable to execute 'gcc': No such file or directory error: command 'gcc' failed with exit status 1" error. You'll get a bigger image but it will work

• The python packages in the requirements.txt files were correctly installed in the virtual environment venv but the python interpreter used to run the server was the default one, so using https://pythonspeed.com/articles/activate-virtualenv-dockerfile/ I've reordered and rewritten some of the commands. Also wheel needed to be installed prior to requirements.txt so that "Using legacy 'setup.py install' for mosestokenizer, since package 'wheel' is not installed." and similar issues are solved

• Finally, after downloading the models and adding these modifications I got the service running. Remember to acces the container IP which might not be localhost https://stackoverflow.com/questions/17157721/how-to-get-a-docker-containers-ip-address-from-the-host

Code a little less organized than what's on the elg branch but it got the service running. Dockerfile.txt

Thanks @mhepaixao and @Aguiar16 for the discussion on this issue. Rafael