Running mlconvgec2018 for inference ?

[yashkumaratri]

OS : Ubuntu 16.04 Python Version : 3.5 CUDA : 8

Whenever I setup farseq-py by running python3 setup.py install (The one which gets downloaded by download.sh script) I get

  cmd_obj.run()
  File "setup.py", line 49, in run
    conv_tbc.build()
  File "/home/ubuntu/.local/lib/python3.5/site-packages/torch/utils/ffi/__init__.py", line 184, in build
    _build_extension(ffi, cffi_wrapper_name, target_dir, verbose)
  File "/home/ubuntu/.local/lib/python3.5/site-packages/torch/utils/ffi/__init__.py", line 108, in _build_extension
    outfile = ffi.compile(tmpdir=tmpdir, verbose=verbose, target=libname)
  File "/home/ubuntu/.local/lib/python3.5/site-packages/cffi/api.py", line 697, in compile
    compiler_verbose=verbose, debug=debug, **kwds)
  File "/home/ubuntu/.local/lib/python3.5/site-packages/cffi/recompiler.py", line 1520, in recompile
    compiler_verbose, debug)
  File "/home/ubuntu/.local/lib/python3.5/site-packages/cffi/ffiplatform.py", line 22, in compile
    outputfilename = _build(tmpdir, ext, compiler_verbose, debug)
  File "/home/ubuntu/.local/lib/python3.5/site-packages/cffi/ffiplatform.py", line 58, in _build
    raise VerificationError('%s: %s' % (e.__class__.__name__, e))
cffi.error.VerificationError: CompileError: command 'x86_64-linux-gnu-gcc' failed with exit status 1

But as per issue#2 I used latest fairseq but now I am running in lot of conflict like

bash run.sh data/test/conll14st-test/conll14st-test.tok.src ~/mlconvgec2018/ 0 models/
+ source paths.sh
++++ dirname paths.sh
+++ cd .
+++ pwd
++ BASE_DIR=/home/ubuntu/mlconvgec2018
++ DATA_DIR=/home/ubuntu/mlconvgec2018/data
++ MODEL_DIR=/home/ubuntu/mlconvgec2018/models
++ SCRIPTS_DIR=/home/ubuntu/mlconvgec2018/scripts
++ SOFTWARE_DIR=/home/ubuntu/mlconvgec2018/software
+ '[' 4 -ge 4 ']'
+ input_file=data/test/conll14st-test/conll14st-test.tok.src
+ output_dir=/home/ubuntu/mlconvgec2018/
+ device=0
+ model_path=models/
+ '[' 4 -eq 6 ']'
+ [[ -d models/ ]]
++ ls 'models//*pt'
++ tr '\n' ' '
++ sed 's| \([^$]\)| --path \1|g'
ls: cannot access 'models//*pt': No such file or directory
+ models=
+ echo

+ FAIRSEQPY=/home/ubuntu/mlconvgec2018/software/fairseq-py
+ NBEST_RERANKER=/home/ubuntu/mlconvgec2018/software/nbest-reranker
+ beam=12
+ nbest=12
+ threads=12
+ mkdir -p /home/ubuntu/mlconvgec2018/
+ /home/ubuntu/mlconvgec2018/scripts/apply_bpe.py -c /home/ubuntu/mlconvgec2018/models/bpe_model/train.bpe.model
+ CUDA_VISIBLE_DEVICES=0
+ python3.5 /home/ubuntu/mlconvgec2018/software/fairseq-py/generate.py --no-progress-bar --path --beam 12 --nbest 12 --interactive --workers 12 /home/ubuntu/mlconvgec2018/models/data_bin
usage: generate.py [-h] [--no-progress-bar] [--log-interval N]
                   [--log-format {json,none,simple,tqdm}] [--seed N] [--fp16]
                   [--task TASK] [--skip-invalid-size-inputs-valid-test]
                   [--max-tokens N] [--max-sentences N] [--gen-subset SPLIT]
                   [--num-shards N] [--shard-id ID] [--path FILE]
                   [--remove-bpe [REMOVE_BPE]] [--cpu] [--quiet] [--beam N]
                   [--nbest N] [--max-len-a N] [--max-len-b N] [--min-len N]
                   [--no-early-stop] [--unnormalized] [--no-beamable-mm]
                   [--lenpen LENPEN] [--unkpen UNKPEN]
                   [--replace-unk [REPLACE_UNK]] [--score-reference]
                   [--prefix-size PS] [--sampling] [--sampling-topk PS]
                   [--sampling-temperature N] [--diverse-beam-groups N]
                   [--diverse-beam-strength N] [--print-alignment]
                   [--model-overrides DICT]
generate.py: error: argument --path: expected one argument

Now the question is how to use that ./run.sh -------------------------------------------------- to make it inference.

[yashkumaratri]

Tried the solution proposed by @renhongkai in https://github.com/nusnlp/mlconvgec2018/issues/2#issuecomment-364710330

But a new error popped up

.............
.....................
+ python3.5 /home/ubuntu/mlconvgec2018/software/fairseq-py/generate.py --no-progress-bar --path models/mlconv_embed//model1.pt --path models/mlconv_embed//model2.pt --path models/mlconv_embed//model3.pt --path models/mlconv_embed//model4.pt --beam 12 --nbest 12 /home/ubuntu/mlconvgec2018/models/data_bin
Traceback (most recent call last):
  File "/home/ubuntu/mlconvgec2018/software/fairseq-py/generate.py", line 171, in <module>
    main(args)
  File "/home/ubuntu/mlconvgec2018/software/fairseq-py/generate.py", line 34, in main
    task = tasks.setup_task(args)
  File "/home/ubuntu/mlconvgec2018/software/fairseq-py/fairseq/tasks/__init__.py", line 19, in setup_task
    return TASK_REGISTRY[args.task].setup_task(args)
  File "/home/ubuntu/mlconvgec2018/software/fairseq-py/fairseq/tasks/translation.py", line 88, in setup_task
    src_dict = Dictionary.load(os.path.join(args.data, 'dict.{}.txt'.format(args.source_lang)))
  File "/home/ubuntu/mlconvgec2018/software/fairseq-py/fairseq/data/dictionary.py", line 93, in load
    with open(f, 'r') as fd:
FileNotFoundError: [Errno 2] No such file or directory: '/home/ubuntu/mlconvgec2018/models/data_bin/dict.en.txt'

and there are 2 dictionary files in data_bin.

So How to make your code to work ?

[shamilcm]

To make the scripts work and replicate exact results, you have to use the specified Fairseq commit in download.sh (which requires PyTorch 0.2.0post3 compiled from source).

The latest version of Fairseq has numerous changes.

For example, in the latest version of Fairseq, generate.py --interactive is replaced by a separate script interactive.py. Also, when you have to decode with multiple models, instead of giving them as

--path models/mlconv_embed//model1.pt --path models/mlconv_embed//model2.pt --path models/mlconv_embed//model3.pt --path models/mlconv_embed//model4.pt

you have to give:

--path models/mlconv_embed//model1.pt:models/mlconv_embed//model2.pt:models/mlconv_embed//model3.pt:models/mlconv_embed//model4.pt

If you are using the latest Fairseq, do not rely on the released shell scripts as they were written for an older version that is used in this work. You will have to follow the Fairseq documentation and use the released models alone. It may perhaps be better to retrain models using the latest version of Fairseq and test using the same.