IGNF / myria3d

Myria3D: Aerial Lidar HD Semantic Segmentation with Deep Learning
https://ignf.github.io/myria3d/
BSD 3-Clause "New" or "Revised" License
169 stars 23 forks source link

"RecursionError: maximum recursion depth exceeded in comparison" while training new model #59

Closed jistiak closed 1 year ago

jistiak commented 1 year ago

I am trying to train a new model with the latest release v3.3.0 of Myria3D. But I am always getting this recursion error generating from Hydra. It looks like this-

File "/opt/anaconda3/envs/myria3d/lib/python3.9/site-packages/hydra/_internal/defaults_list.py", line 575, in _create_defaults_tree_impl
    add_child(children, new_root)
File "/opt/anaconda3/envs/myria3d/lib/python3.9/site-packages/hydra/_internal/defaults_list.py", line 518, in add_child
  subtree_ = _create_defaults_tree_impl(
File "/opt/anaconda3/envs/myria3d/lib/python3.9/site-packages/hydra/_internal/defaults_list.py", line 575, in _create_defaults_tree_impl
  add_child(children, new_root)
File "/opt/anaconda3/envs/myria3d/lib/python3.9/site-packages/hydra/_internal/defaults_list.py", line 518, in add_child
  subtree_ = _create_defaults_tree_impl(
File "/opt/anaconda3/envs/myria3d/lib/python3.9/site-packages/hydra/_internal/defaults_list.py", line 575, in _create_defaults_tree_impl
  add_child(children, new_root)
File "/opt/anaconda3/envs/myria3d/lib/python3.9/site-packages/hydra/_internal/defaults_list.py", line 518, in add_child
  subtree_ = _create_defaults_tree_impl(
File "/opt/anaconda3/envs/myria3d/lib/python3.9/site-packages/hydra/_internal/defaults_list.py", line 575, in _create_defaults_tree_impl
  add_child(children, new_root)
File "/opt/anaconda3/envs/myria3d/lib/python3.9/site-packages/hydra/_internal/defaults_list.py", line 518, in add_child
  subtree_ = _create_defaults_tree_impl(
File "/opt/anaconda3/envs/myria3d/lib/python3.9/site-packages/hydra/_internal/defaults_list.py", line 501, in _create_defaults_tree_impl
  ) and _has_config_content(loaded.config)
File "/opt/anaconda3/envs/myria3d/lib/python3.9/site-packages/hydra/_internal/defaults_list.py", line 437, in _has_config_content
  for key in cfg.keys():
File "/opt/anaconda3/envs/myria3d/lib/python3.9/site-packages/omegaconf/dictconfig.py", line 511, in keys
  assert isinstance(ret, AbstractSet)
File "/opt/anaconda3/envs/myria3d/lib/python3.9/typing.py", line 720, in __instancecheck__
  return self.__subclasscheck__(type(obj))
File "/opt/anaconda3/envs/myria3d/lib/python3.9/typing.py", line 852, in __subclasscheck__
  return issubclass(cls, self.__origin__)
File "/opt/anaconda3/envs/myria3d/lib/python3.9/abc.py", line 123, in __subclasscheck__
  return _abc_subclasscheck(cls, subclass)
RecursionError: maximum recursion depth exceeded in comparison

I followed the instructions from dataset preparation and train new models sections and still having this issue. Previously I have tried the toy_dataset as well as trained new model with previous versions of Myria3D without any issues.

I found out from here that this might happen if the config somehow loads itself. But I could not figure out which config file might have caused this.

Any suggestions on how to fix it? Thanks :)

System: Ubuntu 22.04 Python: 3.9.16 hydra version: 1.1.2

CharlesGaydon commented 1 year ago

Hi @jistiak, thank you for pointing to the origin of this error: I was able to find the culprit in the base training configuration.

I just merged a fix (see https://github.com/IGNF/myria3d/pull/61) and you may use V3.3.1 :)

CharlesGaydon commented 1 year ago

Out of curiosity: are you using in-house data or working on public aerial Lidar datasets? I would love to hear more about your use case (Discussions might be a great place to Show and Tell!) πŸ˜ƒ

jistiak commented 1 year ago

I just merged a fix (see #61) and you may use V3.3.1 :)

@CharlesGaydon Thanks a lot for addressing the issue so quickly. It solved the problem πŸ˜ƒ

are you using in-house data or working on public aerial Lidar datasets?

As of now, I am using public lidar datasets. If I manage to do anything worthwhile, I would definitely love to discuss it in Show and Tell πŸ˜ƒ

CharlesGaydon commented 1 year ago

You are very welcome. Lovely! Closing this issue :)