Dependencies broken for production release ?

[phardy-egis]

Thank you for your work and sharing it as Open-Source.

I noticed that the file proto151_V2.0_epoch_100_Myria3DV3.1.0_predict_config_V3.1.0 has the following content for predict command:

predict:
        CopyFullPos:
          _target_: myria3d.pctl.transforms.transforms.CopyFullPos
        GridSampling:
          _target_: torch_geometric.transforms.GridSampling
          _args_:
          - 0.25
        MinimumNumNodes:
          _target_: myria3d.pctl.transforms.transforms.MinimumNumNodes
          _args_:
          - 300
        MaximumNumNodes:
          _target_: myria3d.pctl.transforms.transforms.MaximumNumNodes
          _args_:
          - 40000
        CopySampledPos:
          _target_: myria3d.pctl.transforms.transforms.CopySampledPos
        Center:
          _target_: torch_geometric.transforms.Center

It thus makes reference to myria3d.pctl.transforms.transforms.MinimumNumNodes class, which is called when we run myria3d.predict command.

This class is not provided inside the file the production release: URL Prod Release

The class MinimumNumNodes does not appear in the file part of release source (the file referenced in by the release commit does not have this class either: URL pointing to transform.py used in production release )

Would it be possible to update the sources attached to production release to get an actually working release ?

A first help could be to provide the commit SHA pointing to the correct dependencies to be used for production.

Thank you in advance for your help. Feel free to email me in french.

[phardy-egis]

I tried build an environment from the "main" branch sources. It looks like running production config and ckpt is not working when using the "main" branch sources to build a docker image.

python -m myria3d.predict \
--config-path /model/ \
--config-name proto151_V2.0_epoch_100_Myria3DV3.1.0_predict_config_V3.1.0.yaml \
predict.src_las=/lidar/test.las \
predict.output_dir=/lidar/output \
predict.ckpt_path=/model/proto151_V2.0_epoch_100_Myria3DV3.1.0.ckpt \
predict.gpus=0 \
datamodule.batch_size=N \
hydra.run.dir=/log/hydra

I hit the following error:

Traceback (most recent call last):
  File "/miniconda/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/miniconda/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/app/myria3d/predict.py", line 104, in <module>
    main()
  File "/app/myria3d/utils/utils.py", line 159, in timed
    result = method(*args, **kwargs)
  File "/miniconda/lib/python3.9/site-packages/hydra/main.py", line 48, in decorated_main
    _run_hydra(
  File "/miniconda/lib/python3.9/site-packages/hydra/_internal/utils.py", line 377, in _run_hydra
    run_and_report(
  File "/miniconda/lib/python3.9/site-packages/hydra/_internal/utils.py", line 214, in run_and_report
    raise ex
  File "/miniconda/lib/python3.9/site-packages/hydra/_internal/utils.py", line 211, in run_and_report
    return func()
  File "/miniconda/lib/python3.9/site-packages/hydra/_internal/utils.py", line 378, in <lambda>
    lambda: hydra.run(
  File "/miniconda/lib/python3.9/site-packages/hydra/_internal/hydra.py", line 111, in run
    _ = ret.return_value
  File "/miniconda/lib/python3.9/site-packages/hydra/core/utils.py", line 233, in return_value
    raise self._return_value
  File "/miniconda/lib/python3.9/site-packages/hydra/core/utils.py", line 160, in run_job
    ret.return_value = task_function(task_cfg)
  File "/app/myria3d/predict.py", line 100, in main
    predict(config)
  File "/app/myria3d/utils/utils.py", line 159, in timed
    result = method(*args, **kwargs)
  File "/app/myria3d/predict.py", line 55, in predict
    itp = hydra.utils.instantiate(config.predict.interpolator)
  File "/miniconda/lib/python3.9/site-packages/omegaconf/dictconfig.py", line 357, in __getattr__
    self._format_and_raise(key=key, value=None, cause=e)
  File "/miniconda/lib/python3.9/site-packages/omegaconf/base.py", line 190, in _format_and_raise
    format_and_raise(
  File "/miniconda/lib/python3.9/site-packages/omegaconf/_utils.py", line 738, in format_and_raise
    _raise(ex, cause)
  File "/miniconda/lib/python3.9/site-packages/omegaconf/_utils.py", line 716, in _raise
    raise ex.with_traceback(sys.exc_info()[2])  # set end OC_CAUSE=1 for full backtrace
  File "/miniconda/lib/python3.9/site-packages/omegaconf/dictconfig.py", line 351, in __getattr__
    return self._get_impl(key=key, default_value=_DEFAULT_MARKER_)
  File "/miniconda/lib/python3.9/site-packages/omegaconf/dictconfig.py", line 438, in _get_impl
    node = self._get_node(key=key, throw_on_missing_key=True)
  File "/miniconda/lib/python3.9/site-packages/omegaconf/dictconfig.py", line 465, in _get_node
    self._validate_get(key)
  File "/miniconda/lib/python3.9/site-packages/omegaconf/dictconfig.py", line 166, in _validate_get
    self._format_and_raise(
  File "/miniconda/lib/python3.9/site-packages/omegaconf/base.py", line 190, in _format_and_raise
    format_and_raise(
  File "/miniconda/lib/python3.9/site-packages/omegaconf/_utils.py", line 818, in format_and_raise
    _raise(ex, cause)
  File "/miniconda/lib/python3.9/site-packages/omegaconf/_utils.py", line 716, in _raise
    raise ex.with_traceback(sys.exc_info()[2])  # set end OC_CAUSE=1 for full backtrace
omegaconf.errors.ConfigAttributeError: Key 'interpolator' is not in struct
    full_key: predict.interpolator
    object_type=dict

Running version 1.0 and 0.0 of the model seems to raise errors too.

[jistiak]

Hi @phardy-egis , it looks like you used the ...config_V3.1.0.yaml to run the prediction. Could you please use the version 3.2.0 and see if the error persists?

[phardy-egis]

Thank you @jistiak, you were right. The error does not persist when I use the github "main" branch sources and the ...config_V3.1.0.yaml hydra config file.

However, I still let this issue open because the sources of the production release are not easily reusable, we need to clone the master branch instead of using the production release to get it work. Maybe @CharlesGaydon could help us at fixing the source files of the production release ?

[CharlesGaydon]

Hi, and thank you both for sharing these issues.

I am sorry for the confusion: what was called a Production Release is in fact not a release, but was rather a way for me to publish/host some models and their model cards. The description of the Production Release mentionned that the tracked commit was arbitrary, but it was still misleading.

I renamed the release to Production Assets [arbitrary commit] so that (hopefully) this confusion is alleviated.

Additionnaly, thank you @jistiak for pointing out that version ...config_V3.2.0.yaml of the config should be used with the current main branch.

We have an incoming PR that will embed a defaut model + config directly in the configuration so that using the trained model is facilitated. :)

[phardy-egis]

Thank you @CharlesGaydon for clarification. We are looking forward to the arrival of the new PR !