Open ihoholko opened 1 year ago
Okay, fixed with W&B and Tensor-board by changing the structure of config. But have some troubles with MLflowVisBackend
yet. Loggers are not custom_hooks
as I got:
├── visualizer
│ └── type: mmdet.DetLocalVisualizer
│ name: visualizer
│ vis_backends:
│ - type: LocalVisBackend
│ - type: TensorboardVisBackend
│ - type: MLflowVisBackend
│ save_dir: null
│ exp_name: Mosaic-roi
│ run_name: mask-rcnn_r50-caffe_fpn_ms-poly-1x_coco
│ tags:
│ - ai-mosaic
│ - mask-rcnn
│ params: null
│ tracking_uri: http://mlflow:
│ artifact_suffix:
│ - .json
│ - .log
│ - .py
│ - yaml
│
-->
env_cfg = dict(
cudnn_benchmark=False,
mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0),
dist_cfg=dict(backend='nccl'))
vis_backends = [dict(type='LocalVisBackend')]
visualizer = dict(
type='DetLocalVisualizer',
vis_backends=[
dict(type='LocalVisBackend'),
dict(type='TensorboardVisBackend'),
dict(
type='MLflowVisBackend',
save_dir=None,
exp_name='Mosaic-roi',
run_name='mask-rcnn_r50-caffe_fpn_ms-poly-1x_coco',
tags=['ai-mosaic', 'mask-rcnn'],
params=None,
tracking_uri='http://',
artifact_suffix=['.json', '.log', '.py', 'yaml'])
],
name='visualizer')
But still have a problem with Mlflow in this case:
Error executing job with overrides: ['experiment=mmdet/mask_rcnn/mask-rcnn_r50-caffe_fpn_ms-poly-1x_coco.yaml', 'name=Mosaic-roi']
Traceback (most recent call last):
File "/home/oem/Projects/aimosaic-roi/venv2/lib/python3.9/site-packages/mmengine/registry/build_functions.py", line 119, in build_from_cfg
obj = obj_cls.get_instance(**args) # type: ignore
File "/home/oem/Projects/aimosaic-roi/venv2/lib/python3.9/site-packages/mmengine/visualization/visualizer.py", line 1165, in get_instance
instance = super().get_instance(name, **kwargs)
File "/home/oem/Projects/aimosaic-roi/venv2/lib/python3.9/site-packages/mmengine/utils/manager.py", line 110, in get_instance
instance = cls(name=name, **kwargs) # type: ignore
File "/home/oem/Projects/aimosaic-roi/venv2/lib/python3.9/site-packages/mmdet/visualization/local_visualizer.py", line 88, in __init__
super().__init__(
File "/home/oem/Projects/aimosaic-roi/venv2/lib/python3.9/site-packages/mmengine/visualization/visualizer.py", line 202, in __init__
self._vis_backends[name] = VISBACKENDS.build(vis_backend)
File "/home/oem/Projects/aimosaic-roi/venv2/lib/python3.9/site-packages/mmengine/registry/registry.py", line 545, in build
return self.build_func(cfg, *args, **kwargs, registry=self)
File "/home/oem/Projects/aimosaic-roi/venv2/lib/python3.9/site-packages/mmengine/registry/build_functions.py", line 100, in build_from_cfg
raise KeyError(
KeyError: 'MLflowVisBackend is not in the vis_backend registry. Please check whether the value of `MLflowVisBackend` is correct or it was registered as expected. More details can be found at https://mmengine.readthedocs.io/en/latest/advanced_tutorials/config.html#import-the-custom-module'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/oem/Projects/aimosaic-roi/training/run.py", line 53, in main
metric_dict = train(cfg)
File "/home/oem/Projects/aimosaic-roi/training/mmdet/train_mmdet.py", line 46, in train_mmdet
runner = Runner.from_cfg(mm_conf)
File "/home/oem/Projects/aimosaic-roi/venv2/lib/python3.9/site-packages/mmengine/runner/runner.py", line 439, in from_cfg
runner = cls(
File "/home/oem/Projects/aimosaic-roi/venv2/lib/python3.9/site-packages/mmengine/runner/runner.py", line 393, in __init__
self.visualizer = self.build_visualizer(visualizer)
File "/home/oem/Projects/aimosaic-roi/venv2/lib/python3.9/site-packages/mmengine/runner/runner.py", line 780, in build_visualizer
return VISUALIZERS.build(visualizer)
File "/home/oem/Projects/aimosaic-roi/venv2/lib/python3.9/site-packages/mmengine/registry/registry.py", line 545, in build
return self.build_func(cfg, *args, **kwargs, registry=self)
File "/home/oem/Projects/aimosaic-roi/venv2/lib/python3.9/site-packages/mmengine/registry/build_functions.py", line 135, in build_from_cfg
raise type(e)(
KeyError: "class `DetLocalVisualizer` in mmdet/visualization/local_visualizer.py: 'MLflowVisBackend is not in the vis_backend registry. Please check whether the value of `MLflowVisBackend` is correct or it was registered as expected. More details can be found at https://mmengine.readthedocs.io/en/latest/advanced_tutorials/config.html#import-the-custom-module'"
Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
Describe the bug
I am developing project based on MMdetection. I am trying to use customised configuration. And it works well, the training process is good. But when I am trying to add
custom_hooks
I am getting some errors.The project idea (config & train loop):
Training code:
I installed mmdet as submodule and as a package. In both scenarios I can't use hooks. Getting following errors:
Reproduction
As described before. I rewrote everything to support yaml configuration. Info above.
Custom.
Environment