search

amake / moses-smt

Dock You a Moses: Moses Statistical MT in a container
MIT License
12 stars 4 forks source link

Installation problem #2

Closed 4ka0 closed 6 years ago

4ka0 commented 6 years ago

Hi Aaron,

I'd like to set up and train a local instance of moses-smt and then use it through OmegaT to see if it would help with my translation work (patents Ja>En), but I'm receiving the following error when executing "make SOURCE_LANG=JA TARGET_LANG=EN".

4ka0-laptop:~ Jage$ git clone https://github.com/amake/moses-smt.git
Cloning into 'moses-smt'...
remote: Counting objects: 322, done.
remote: Compressing objects: 100% (6/6), done.
remote: Total 322 (delta 4), reused 9 (delta 4), pack-reused 312
Receiving objects: 100% (322/322), 43.60 KiB | 579.00 KiB/s, done.
Resolving deltas: 100% (178/178), done.
...
4ka0-laptop:moses-smt Jage$ ls
Dockerfile          makefile
Dockerrun.aws.image.json    mosesxmlrpcrepl.py
Dockerrun.aws.noimage.json  test.sh
LICENSE             tmx-train
README.md           tmx-tune
base                work
4ka0-laptop:moses-smt Jage$ make SOURCE_LANG=JA TARGET_LANG=EN
LC_ALL=C virtualenv env
New python executable in /Volumes/Untitled/mac local jobs/archive/moses-smt/env/bin/python
Installing setuptools, pip, wheel...done.
env/bin/pip install git+https://github.com/amake/tmx2corpus.git
make: env/bin/pip: No such file or directory
make: *** [env/bin/tmx2corpus] Error 1

I've tried manually downloading tmx2corpus from your repository on GitHub and placing the whole folder in env/bin but I'm guessing that's not what its looking for.

I have Homebrew 1.6.4, Docker 18.03.1-ce-mac65, Python 2.7.10, pip 10.0.1, and virtualenv 15.10.1 installed on High Sierra 10.13.4.

Any advice you could provide here would be a big help.

Thanks, Jon

amake commented 6 years ago

Sorry for the late reply. Apparently I wasn't "watching" my own repo :(

It says it couldn't find env/bin/pip, which is hard to believe since the virtualenv invocation seemed to work OK. Can you run ls env/bin and report the results?

4ka0 commented 6 years ago

No worries at all. Many thanks for the reply. I get the following when running ls env/bin

4ka0-laptop:moses-smt Jage$ ls env/bin
activate        easy_install-2.7    python-config
activate.csh        pip         python2
activate.fish       pip2            python2.7
activate_this.py    pip2.7          wheel
easy_install        python
amake commented 6 years ago

I honestly don't know how it's possible for make to report env/bin/pip: No such file or directory while ls shows that it is clearly there.

Some random shots in the dark:

4ka0 commented 6 years ago

Many thanks for the ideas. I replaced Homebrew with MacPorts and removed all spaces from the cwd and this seems to have fixed the previous issue. I ran make SOURCE_LANG=JA TARGET_LANG=EN and everything seemed to work fine (lots of heavy training etc.) until ...

Segmentation fault
Exit code: 139
The decoder died. CONFIG WAS -weight-overwrite 'TranslationModel0= 0.043478 0.043478 0.043478 0.043478 PhrasePenalty0= 0.043478 Distortion0= 0.065217 WordPenalty0= -0.217391 LexicalReordering0= 0.065217 0.065217 0.065217 0.065217 0.065217 0.065217 LM0= 0.108696' 
make: *** [work/myinstance-EN-JA/tune/mert-work/moses.ini] Error 2

I read on a Moses forum that this might be a memory issue so I set Docker to use all 8 CPUs and 16.0 GiB but I still get the same result. I initially used 100 TMX files to train and 20 for tuning and then I halved the amount but I still get the same result. I have the full output in addition to the above so I can post that too if you would like (not posted here as it is large). Do you have ideas why this may be happening? No worries if not.

amake commented 6 years ago

I'm glad you managed to solve your initial problem. Unfortunately I really have no idea about training issues, sorry.

4ka0 commented 6 years ago

No worries. Thanks a lot for the previous help, and of course also for making this all available. Hugely appreciated.