search

tryolabs / luminoth

Deep Learning toolkit for Computer Vision.
https://tryolabs.com
BSD 3-Clause "New" or "Revised" License
2.4k stars 400 forks source link

"AttributeError: _replace" error when parsing a config.yml file during training #223

Closed ogienma closed 5 years ago

ogienma commented 5 years ago

Hello,

When running lumi train -c config.yml, the program exits with an "AttributeError: _replace" message as the config file is being parsed (before the dataset is accessed and the actual training process starts).

Here's what I've tried testing so far:

The full message on 3.6.6 is as follows:

/Users/ogienma/miniconda3/envs/lumi/lib/python3.6/site-packages/tensorflow/python/util/tf_inspect.py:75: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
  return _inspect.getargspec(target)
/Users/ogienma/miniconda3/envs/lumi/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
  return f(*args, **kwds)
/Users/ogienma/miniconda3/envs/lumi/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
  return f(*args, **kwds)
Traceback (most recent call last):
  File "/Users/ogienma/miniconda3/envs/lumi/bin/lumi", line 11, in <module>
    sys.exit(cli())
  File "/Users/ogienma/miniconda3/envs/lumi/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/Users/ogienma/miniconda3/envs/lumi/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/Users/ogienma/miniconda3/envs/lumi/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/ogienma/miniconda3/envs/lumi/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/ogienma/miniconda3/envs/lumi/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/Users/ogienma/miniconda3/envs/lumi/lib/python3.6/site-packages/luminoth/train.py", line 296, in train
    config = get_config(config_files, override_params=override_params)
  File "/Users/ogienma/miniconda3/envs/lumi/lib/python3.6/site-packages/luminoth/utils/config.py", line 19, in get_config
    model_base_config, custom_config, override_params
  File "/Users/ogienma/miniconda3/envs/lumi/lib/python3.6/site-packages/luminoth/utils/config.py", line 225, in get_model_config
    return cleanup_config(config)
  File "/Users/ogienma/miniconda3/envs/lumi/lib/python3.6/site-packages/luminoth/utils/config.py", line 204, in cleanup_config
    config.pop(cleanup_key, None)
  File "/Users/ogienma/miniconda3/envs/lumi/lib/python3.6/site-packages/easydict/__init__.py", line 142, in pop
    delattr(self, k)
AttributeError: _replace

Python 3.5.6 gets past the configuration parse, but fails on another line related to _replace:

Traceback (most recent call last):
  File "/Users/ogienma/miniconda3/envs/lumi35/bin/lumi", line 11, in <module>
    load_entry_point('luminoth', 'console_scripts', 'lumi')()
  File "/Users/ogienma/miniconda3/envs/lumi35/lib/python3.5/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/Users/ogienma/miniconda3/envs/lumi35/lib/python3.5/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/Users/ogienma/miniconda3/envs/lumi35/lib/python3.5/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/ogienma/miniconda3/envs/lumi35/lib/python3.5/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/ogienma/miniconda3/envs/lumi35/lib/python3.5/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/Users/ogienma/projects/lumi/luminoth/luminoth/train.py", line 307, in train
    config, environment=environment
  File "/Users/ogienma/projects/lumi/luminoth/luminoth/train.py", line 66, in run
    prediction_dict = model(train_image, train_bboxes, is_training=True)
  File "/Users/ogienma/miniconda3/envs/lumi35/lib/python3.5/site-packages/sonnet/python/modules/base.py", line 367, in __call__
    return self._call(*args, **kwargs)
  File "/Users/ogienma/miniconda3/envs/lumi35/lib/python3.5/site-packages/sonnet/python/modules/base.py", line 389, in _call
    outputs, subgraph_name_scope = self._template(*args, **kwargs)
  File "/Users/ogienma/miniconda3/envs/lumi35/lib/python3.5/site-packages/tensorflow/python/ops/template.py", line 359, in __call__
    return self._call_func(args, kwargs)
  File "/Users/ogienma/miniconda3/envs/lumi35/lib/python3.5/site-packages/tensorflow/python/ops/template.py", line 310, in _call_func
    result = self._func(*args, **kwargs)
  File "/Users/ogienma/miniconda3/envs/lumi35/lib/python3.5/site-packages/sonnet/python/modules/base.py", line 202, in _build_wrapper
    output = self._build(*args, **kwargs)
  File "/Users/ogienma/projects/lumi/luminoth/luminoth/models/fasterrcnn/fasterrcnn.py", line 108, in _build
    debug=self._debug, seed=self._seed
  File "/Users/ogienma/projects/lumi/luminoth/luminoth/models/fasterrcnn/rpn.py", line 44, in __init__
    config.rpn_initializer, seed=seed
  File "/Users/ogienma/projects/lumi/luminoth/luminoth/utils/vars.py", line 77, in get_initializer
    return initializer(**config)
TypeError: __init__() got an unexpected keyword argument '_replace'
ogienma commented 5 years ago

I solved this by deleting the config keys in a different way. I replaced https://github.com/tryolabs/luminoth/blob/master/luminoth/utils/config.py#L204 with:

try:
    del config[cleanup_key]
except KeyError:
    pass

No idea why that works but pop doesn't - but there you go.

dekked commented 5 years ago

Thanks for the report.

This is caused by a bug in a recent version of a dependency (easydict). I have submitted this PR, which hopefully will be merged and released soon. If not, we'll work around this code.

dekked commented 5 years ago

I fixed this in 2d984c4d by pinning the version of easydict, while my PR gets merged upstream.