about train

[renhongkai]

when i run ./train.sh,I encountered this problem: 1、``` (python3.6_env) [renhongkai@xxkx-gpu1 training]# ./train.sh

• set -e
• source ../paths.sh ++++ dirname ../paths.sh +++ cd .. +++ pwd ++ BASE_DIR=/home/renhongkai/project/mlconvgec2018 ++ DATA_DIR=/home/renhongkai/project/mlconvgec2018/data ++ MODEL_DIR=/home/renhongkai/project/mlconvgec2018/models ++ SCRIPTS_DIR=/home/renhongkai/project/mlconvgec2018/scripts ++ SOFTWARE_DIR=/home/renhongkai/project/mlconvgec2018/software
• FAIRSEQPY=/home/renhongkai/project/mlconvgec2018/software/fairseq-py
• SEED=1000
• DATA_BIN_DIR=processed/bin
• OUT_DIR=models/mlconv/model1000/
• mkdir -p models/mlconv/model1000/
• PYTHONPATH=/home/renhongkai/project/mlconvgec2018/software/fairseq-py:
• CUDA_VISIBLE_DEVICES=0,1,2
• python3.5 /home/renhongkai/project/mlconvgec2018/software/fairseq-py/train.py --save-dir models/mlconv/model1000/ --encoder-embed-dim 500 --decoder-embed-dim 500 --decoder-out-embed-dim 500 --dropout 0.2 --clip-norm 0.1 --lr 0.25 --min-lr 1e-4 --encoder-layers '[(1024,3)] 7' --decoder-layers '[(1024,3)] 7' --momentum 0.99 --max-epoch 100 --batch-size 32 --no-progress-bar --seed 1000 processed/bin Traceback (most recent call last): File "/home/renhongkai/project/mlconvgec2018/software/fairseq-py/train.py", line 15, in from fairseq import data, options, utils File "/home/renhongkai/project/mlconvgec2018/software/fairseq-py/fairseq/options.py", line 11, in from fairseq import models File "/home/renhongkai/project/mlconvgec2018/software/fairseq-py/fairseq/models/init.py", line 14, in from . import fconv, lstm File "/home/renhongkai/project/mlconvgec2018/software/fairseq-py/fairseq/models/fconv.py", line 16, in from fairseq.modules import BeamableMM, GradMultiply, LearnedPositionalEmbedding, LinearizedConvolution File "/home/renhongkai/project/mlconvgec2018/software/fairseq-py/fairseq/modules/init.py", line 10, in from .conv_tbc import ConvTBC File "/home/renhongkai/project/mlconvgec2018/software/fairseq-py/fairseq/modules/conv_tbc.py", line 13, in from fairseq import utils File "/home/renhongkai/project/mlconvgec2018/software/fairseq-py/fairseq/utils.py", line 19, in from fairseq import criterions, progress_bar, tokenizer File "/home/renhongkai/project/mlconvgec2018/software/fairseq-py/fairseq/progress_bar.py", line 17, in from tqdm import tqdm ImportError: No module named 'tqdm'

  
  #  **but I have pip install tqdm ,and i can import tqdm in `python:**`

(python3.6_env) [renhongkai@xxkx-gpu1 training]# python
Python 3.6.3 (default, Oct  6 2017, 08:44:35) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tqdm
>>> 

2、when I testing with pre-trained models,I encountered this problem:

(python3.6_env) [renhongkai@xxkx-gpu1 mlconvgec2018]# ./run.sh  ./data/test/conll14st-test/conll14st-test.m2  ./log/log1.txt  0,1 ./models/mlconv_embed/model1.pt  eolm
++ source paths.sh
+++++ dirname paths.sh
++++ cd .
++++ pwd
+++ BASE_DIR=/home/renhongkai/project/mlconvgec2018
+++ DATA_DIR=/home/renhongkai/project/mlconvgec2018/data
+++ MODEL_DIR=/home/renhongkai/project/mlconvgec2018/models
+++ SCRIPTS_DIR=/home/renhongkai/project/mlconvgec2018/scripts
+++ SOFTWARE_DIR=/home/renhongkai/project/mlconvgec2018/software
++ '[' 5 -ge 4 ']'
++ input_file=./data/test/conll14st-test/conll14st-test.m2
++ output_dir=./log/log1.txt
++ device=0,1
++ model_path=./models/mlconv_embed/model1.pt
++ '[' 5 -eq 6 ']'
++ '[' -d ./models/mlconv_embed/model1.pt ']'
++ '[' -f ./models/mlconv_embed/model1.pt ']'
++ model=./models/mlconv_embed/model1.pt
++ FAIRSEQPY=/home/renhongkai/project/mlconvgec2018/software/fairseq-py
++ NBEST_RERANKER=/home/renhongkai/project/mlconvgec2018/software/nbest-reranker
++ beam=12
++ nbest=12
++ threads=12
++ mkdir -p ./log/log1.txt
++ /home/renhongkai/project/mlconvgec2018/scripts/apply_bpe.py -c /home/renhongkai/project/mlconvgec2018/models/bpe_model/train.bpe.model
++ CUDA_VISIBLE_DEVICES=0,1
++ python3.5 /home/renhongkai/project/mlconvgec2018/software/fairseq-py/generate.py --no-progress-bar --path --beam 12 --nbest 12 --interactive --workers 12 /home/renhongkai/project/mlconvgec2018/models/data_bin
ERROR: missing libbleu.so. run `python setup.py install`
Traceback (most recent call last):
  File "/home/renhongkai/project/mlconvgec2018/software/fairseq-py/generate.py", line 12, in <module>
    from fairseq import bleu, data, options, tokenizer, utils
  File "/home/renhongkai/project/mlconvgec2018/software/fairseq-py/fairseq/bleu.py", line 18, in <module>
    raise e
  File "/home/renhongkai/project/mlconvgec2018/software/fairseq-py/fairseq/bleu.py", line 14, in <module>
    from fairseq import libbleu
ImportError: cannot import name 'libbleu'

but I can use libbleu in python:

(python3.6_env) [renhongkai@xxkx-gpu1 mlconvgec2018]# python
Python 3.6.3 (default, Oct  6 2017, 08:44:35) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from fairseq import libbleu
>>> 

who can help me resolve it. thank you.

[shamilcm]

We used Python3.5 and have specified python3.5 within train.sh and run.sh. Try replacing that to your version of python (i.e. Python 3.6) and run.

[renhongkai]

@shamilcm Thank you very much. I have a confusion: MLConv (4 ens.) Refers to which four models are assembled

[shamilcm]

MLConv (4 ens.) refers to the ensemble of these four models: models/mlconv/model{1,2,3,4}.pt MLConv4_embed (4 ens.) refers to the ensemble of these four models: models/mlconv_embed/model{1,2,3,4}.pt

[NikhilCherian]

@renhongkai did you get the results of testing with m2 score? Can you share the results or guide me? I have not tried trained with reranker because of Moses problem. did the above script of run.sh with conll .m2 work?