Problems with the provided recipes

[pmichel31415]

I'm trying to run the iwslt en-vi recipe in https://github.com/neulab/xnmt/tree/master/recipes/stanford-iwslt

But I get several errors:

1. I had to replace the command

   python -m xnmt.xnmt_run_experiments --dynet-gpu ./config.en-vi.yaml

   with

   python -m xnmt.xnmt_run_experiments --dynet-gpu --settings=config.en-vi.yaml

   because it would fall back to settings.standard

2. Even with this I get the following error message

   Traceback (most recent call last):
   File "/home/pmichel1/anaconda3/lib/python3.6/runpy.py", line 183, in _run_module_as_main
   mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
   File "/home/pmichel1/anaconda3/lib/python3.6/runpy.py", line 109, in _get_module_details
   __import__(pkg_name)
   File "/home/pmichel1/.local/lib/python3.6/site-packages/xnmt-0.0.1-py3.6.egg/xnmt/__init__.py", line 25, in <module>
   import xnmt.attender
   File "/home/pmichel1/.local/lib/python3.6/site-packages/xnmt-0.0.1-py3.6.egg/xnmt/attender.py", line 6, in <module>
   from xnmt.param_init import GlorotInitializer, ZeroInitializer
   File "/home/pmichel1/.local/lib/python3.6/site-packages/xnmt-0.0.1-py3.6.egg/xnmt/param_init.py", line 6, in <module>
   from xnmt.persistence import serializable_init, Serializable
   File "/home/pmichel1/.local/lib/python3.6/site-packages/xnmt-0.0.1-py3.6.egg/xnmt/persistence.py", line 18, in <module>
   from xnmt.tee import get_git_revision
   File "/home/pmichel1/.local/lib/python3.6/site-packages/xnmt-0.0.1-py3.6.egg/xnmt/tee.py", line 39, in <module>
   logger.setLevel(min(logging._checkLevel(settings.LOG_LEVEL_CONSOLE), logging._checkLevel(settings.LOG_LEVEL_FILE)))
   File "/home/pmichel1/.local/lib/python3.6/site-packages/simple_settings/core.py", line 84, in __getattr__
   self.setup()
   File "/home/pmichel1/.local/lib/python3.6/site-packages/simple_settings/core.py", line 65, in setup
   self._load_settings_pipeline()
   File "/home/pmichel1/.local/lib/python3.6/site-packages/simple_settings/core.py", line 73, in _load_settings_pipeline
   settings = strategy.load_settings_file(settings_file)
   File "/home/pmichel1/.local/lib/python3.6/site-packages/simple_settings/strategies/yaml_file.py", line 20, in load_settings_file
   return yaml.safe_load(f)
   File "/home/pmichel1/anaconda3/lib/python3.6/site-packages/yaml/__init__.py", line 94, in safe_load
   return load(stream, SafeLoader)
   File "/home/pmichel1/anaconda3/lib/python3.6/site-packages/yaml/__init__.py", line 72, in load
   return loader.get_single_data()
   File "/home/pmichel1/anaconda3/lib/python3.6/site-packages/yaml/constructor.py", line 37, in get_single_data
   return self.construct_document(node)
   File "/home/pmichel1/anaconda3/lib/python3.6/site-packages/yaml/constructor.py", line 46, in construct_document
   for dummy in generator:
   File "/home/pmichel1/anaconda3/lib/python3.6/site-packages/yaml/constructor.py", line 398, in construct_yaml_map
   value = self.construct_mapping(node)
   File "/home/pmichel1/anaconda3/lib/python3.6/site-packages/yaml/constructor.py", line 204, in construct_mapping
   return super().construct_mapping(node, deep=deep)
   File "/home/pmichel1/anaconda3/lib/python3.6/site-packages/yaml/constructor.py", line 129, in construct_mapping
   value = self.construct_object(value_node, deep=deep)
   File "/home/pmichel1/anaconda3/lib/python3.6/site-packages/yaml/constructor.py", line 86, in construct_object
   data = constructor(self, node)
   File "/home/pmichel1/anaconda3/lib/python3.6/site-packages/yaml/constructor.py", line 414, in construct_undefined
   node.start_mark)
   yaml.constructor.ConstructorError: could not determine a constructor for the tag '!Experiment'
   in "config.en-vi.yaml", line 2, column 19

Any idea where the 2nd error message comes from?

EDIT: I'm using the latest master

[msperber]

• This error would mean that some of the imports in xnmt.init.py were not carried out, in other words that the xnmt package was not loaded properly for some reason.
• Also, "settings" must point to a python module and is something different than the YAML config file. You shouldn't need to change the standard settings unless you need more verbosity or something. I'm not sure if this caused the problem from the first bullet point, but if not you could double-check if you ran setup.py as described in the documentation.

[pmichel31415]

OK so for the 1st point here's the error message I got

Traceback (most recent call last):
  File "/home/pmichel1/anaconda3/lib/python3.6/runpy.py", line 183, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/home/pmichel1/anaconda3/lib/python3.6/runpy.py", line 109, in _get_module_details
    __import__(pkg_name)
  File "/home/pmichel1/.local/lib/python3.6/site-packages/xnmt-0.0.1-py3.6.egg/xnmt/__init__.py", line 25, in <module>
    import xnmt.attender
  File "/home/pmichel1/.local/lib/python3.6/site-packages/xnmt-0.0.1-py3.6.egg/xnmt/attender.py", line 6, in <module>
    from xnmt.param_init import GlorotInitializer, ZeroInitializer
  File "/home/pmichel1/.local/lib/python3.6/site-packages/xnmt-0.0.1-py3.6.egg/xnmt/param_init.py", line 6, in <module>
    from xnmt.persistence import serializable_init, Serializable
  File "/home/pmichel1/.local/lib/python3.6/site-packages/xnmt-0.0.1-py3.6.egg/xnmt/persistence.py", line 18, in <module>
    from xnmt.tee import get_git_revision
  File "/home/pmichel1/.local/lib/python3.6/site-packages/xnmt-0.0.1-py3.6.egg/xnmt/tee.py", line 39, in <module>
    logger.setLevel(min(logging._checkLevel(settings.LOG_LEVEL_CONSOLE), logging._checkLevel(settings.LOG_LEVEL_FILE)))
  File "/home/pmichel1/.local/lib/python3.6/site-packages/simple_settings/core.py", line 84, in __getattr__
    self.setup()
  File "/home/pmichel1/.local/lib/python3.6/site-packages/simple_settings/core.py", line 65, in setup
    self._load_settings_pipeline()
  File "/home/pmichel1/.local/lib/python3.6/site-packages/simple_settings/core.py", line 72, in _load_settings_pipeline
    strategy = self._get_strategy_by_file(settings_file)
  File "/home/pmichel1/.local/lib/python3.6/site-packages/simple_settings/core.py", line 81, in _get_strategy_by_file
    raise RuntimeError('Invalid settings file [{}]'.format(settings_file))
RuntimeError: Invalid settings file [settings.standard]

I installed xnmt with python setup.py install --user and there was no error message. I can import xnmt from a python interpreter without problem.

[msperber]

--user suggests that you might not be using virtualenv or anaconda? I've never tried that, but maybe you can try a set up like this (I double-checked that this works with the current master branch):

python3.6 -m venv venv.dy.debug
source venv.dy.debug/bin/activate
git clone https://github.com/neulab/xnmt.git xnmt-debug
cd xnmt-debug
pip install -r requirements.txt
python setup.py install
python -m xnmt.xnmt_run_experiments examples/01_standard.yaml

[pmichel31415]

OK I checked and with my installation

python -m xnmt.xnmt_run_experiments examples/01_standard.yaml

already works

I tried the virtualenv installation and

python -m xnmt.xnmt_run_experiments examples/01_standard.yaml

still works but

python -m xnmt.xnmt_run_experiments ./config.en-vi.yaml

still gives the same error.

So I suspect there might be a problem with the recipe and not the example?

[pmichel31415]

OK the problem is that I was not running the recipe from the xnmt root directory.

Doing

export PYTHONPATH="/projects/tir1/users/pmichel1/xnmt:$PYTHONPATH"

solved it. Not sure if this is a problem with my particular setting or if this should be documented

[msperber]

Ok good! I'm thinking of re-doing / simplifying the settings mechanism because it seems to always produce trouble when things are not run in a very standard way.

[pmichel31415]

Yes something else I noticed that's not very intuitive is that if I run

python -m xnmt.xnmt_run_experiments --dynet-gpu config.en-vi.yaml

instead of

python -m xnmt.xnmt_run_experiments --dynet-gpu ./config.en-vi.yaml

(ie config.en-vi.yaml instead of ./config.en-vi.yaml)it tries to create the log directory at the root / and fails because of permissions. I would have expected it to automatically pick the current directory.