Add option to validate at training start of deepedit - Githubissues

Project-MONAI / model-zoo

MONAI Model Zoo that hosts models in the MONAI Bundle format.

Apache License 2.0

174 stars 65 forks source link

Add option to validate at training start of deepedit #515

Closed mingxin-zheng closed 9 months ago

mingxin-zheng commented 9 months ago

Description

Modify bundle as the new feature enabled in https://github.com/Project-MONAI/MONAI/commit/fc1350aae1de702404189f334ac17fd6f51321d0

Status

Ready/Work in progress/Hold

Please ensure all the checkboxes:

[x] Codeformat tests passed locally by running ./runtests.sh --codeformat.
[ ] In-line docstrings updated.
[ ] Update version and changelog in metadata.json if changing an existing bundle.
[ ] Please ensure the naming rules in config files meet our requirements (please refer to: CONTRIBUTING.md).
[ ] Ensure versions of packages such as monai, pytorch and numpy are correct in metadata.json.
[ ] Descriptions should be consistent with the content, such as eval_metrics of the provided weights and TorchScript modules.
[ ] Files larger than 25MB are excluded and replaced by providing download links in large_file.yml.
[ ] Avoid using path that contains personal information within config files (such as use /home/your_name/ for "bundle_root").

yiheng-wang-nv commented 9 months ago

/build

yiheng-wang-nv commented 9 months ago

need to use new feature in MONAI dev branch, thus it requires a new monai version (currently 1.2.0 shown in https://github.com/Project-MONAI/model-zoo/pull/515/files#diff-90cacb33aaa1f750be54f039f8d7b09347e90155d8293c52c4fd7c3f79420575L30). We may have to use a new tag cc @wyli

mingxin-zheng commented 9 months ago

I hacked the ci install dev branch of monai for testing purposes 8484dbd. That may not be something interesting to keep, but just to get an idea if the bundle can pass test.

mingxin-zheng commented 9 months ago

After discussion with @yiheng-wang-nv , I reverted and will wait for a new rc tagging,

wyli commented 9 months ago

sure, I'll tag another rc today to test the latest feature

wyli commented 9 months ago

/build

wyli commented 9 months ago

/build

wyli commented 9 months ago

some errors testing with rc4, probably due to https://github.com/Project-MONAI/model-zoo/issues/517:

[2023-10-11T08:13:25.441Z] 2023-10-11 08:13:25,380 - ignite.engine.engine.SupervisedEvaluator - INFO - Got new best metric of val_mean_dice: 0.5179497599601746
[2023-10-11T08:13:25.441Z] 2023-10-11 08:13:25,380 - INFO - Epoch[1] Metrics -- val_mean_dice: 0.5179 
[2023-10-11T08:13:25.441Z] 2023-10-11 08:13:25,380 - INFO - Key metric: val_mean_dice best value: 0.5179497599601746 at epoch: 1
[2023-10-11T08:13:26.004Z] 2023-10-11 08:13:25,783 - ignite.engine.engine.SupervisedEvaluator - INFO - Epoch[1] Complete. Time taken: 00:00:01
[2023-10-11T08:13:26.004Z] 2023-10-11 08:13:25,784 - ignite.engine.engine.SupervisedEvaluator - INFO - Engine run complete. Time taken: 00:00:01
[2023-10-11T08:13:26.004Z] 2023-10-11 08:13:25,784 - INFO - Epoch[1] Metrics -- train_dice: 0.5166 
[2023-10-11T08:13:26.004Z] 2023-10-11 08:13:25,784 - INFO - Key metric: train_dice best value: 0.5165802836418152 at epoch: 1
[2023-10-11T08:13:26.004Z] 2023-10-11 08:13:25,785 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch[1] Complete. Time taken: 00:00:02
[2023-10-11T08:13:26.004Z] 2023-10-11 08:13:25,785 - ignite.engine.engine.SupervisedTrainer - INFO - Engine run complete. Time taken: 00:00:02
[2023-10-11T08:13:26.004Z] there are more than 1 component have name `Interaction`: ['monai.apps.deepedit.interaction', 'monai.apps.deepgrow.interaction'], use the first one `monai.apps.deepedit.interaction. if want to use others, please set its full module path in `_target_` directly.
[2023-10-11T08:13:27.370Z] .2023-10-11 08:13:27,138 - INFO - Setting logging properties based on config: models/spleen_deepedit_annotation/configs/logging.conf.
[2023-10-11T08:13:28.005Z] 2023-10-11 08:13:27,791 - ignite.engine.engine.SupervisedEvaluator - INFO - Engine run resuming from iteration 0, epoch 0 until 1 epochs
[2023-10-11T08:13:28.005Z] 2023-10-11 08:13:27,853 - ignite.engine.engine.SupervisedEvaluator - INFO - Restored all variables from models/spleen_deepedit_annotation/models/model.pt
[2023-10-11T08:13:28.005Z] 2023-10-11 08:13:27,926 - INFO - Number of simulated clicks: 2
[2023-10-11T08:13:28.005Z] 2023-10-11 08:13:27,951 - ignite.engine.engine.SupervisedEvaluator - ERROR - Current run is terminating due to exception: 'image_meta_dict'
[2023-10-11T08:13:28.005Z] 2023-10-11 08:13:27,951 - ERROR - Exception: 'image_meta_dict'
[2023-10-11T08:13:28.005Z] Traceback (most recent call last):
[2023-10-11T08:13:28.005Z]   File "/root/.local/share/virtualenvs/MONAI-zoo-premerge-CHR6GQKS/lib/python3.8/site-packages/ignite/engine/engine.py", line 841, in _run_once_on_dataset
[2023-10-11T08:13:28.005Z]     self._fire_event(Events.ITERATION_COMPLETED)
[2023-10-11T08:13:28.005Z]   File "/root/.local/share/virtualenvs/MONAI-zoo-premerge-CHR6GQKS/lib/python3.8/site-packages/ignite/engine/engine.py", line 421, in _fire_event
[2023-10-11T08:13:28.005Z]     func(*first, *(event_args + others), **kwargs)
[2023-10-11T08:13:28.005Z]   File "/root/.local/share/virtualenvs/MONAI-zoo-premerge-CHR6GQKS/lib/python3.8/site-packages/monai/handlers/metrics_saver.py", line 124, in _get_filenames
[2023-10-11T08:13:28.005Z]     meta_data = self.batch_transform(engine.state.batch)
[2023-10-11T08:13:28.005Z]   File "/root/.local/share/virtualenvs/MONAI-zoo-premerge-CHR6GQKS/lib/python3.8/site-packages/monai/handlers/utils.py", line 199, in _wrapper
[2023-10-11T08:13:28.005Z]     ret = [data[0][k] if first else [i[k] for i in data] for k in _keys]
[2023-10-11T08:13:28.005Z]   File "/root/.local/share/virtualenvs/MONAI-zoo-premerge-CHR6GQKS/lib/python3.8/site-packages/monai/handlers/utils.py", line 199, in <listcomp>
[2023-10-11T08:13:28.005Z]     ret = [data[0][k] if first else [i[k] for i in data] for k in _keys]
[2023-10-11T08:13:28.005Z]   File "/root/.local/share/virtualenvs/MONAI-zoo-premerge-CHR6GQKS/lib/python3.8/site-packages/monai/handlers/utils.py", line 199, in <listcomp>
[2023-10-11T08:13:28.005Z]     ret = [data[0][k] if first else [i[k] for i in data] for k in _keys]
[2023-10-11T08:13:28.005Z] KeyError: 'image_meta_dict'
[2023-10-11T08:13:28.005Z] 2023-10-11 08:13:27,951 - ignite.engine.engine.SupervisedEvaluator - ERROR - Engine run is terminating due to exception: 'image_meta_dict'
[2023-10-11T08:13:28.005Z] 2023-10-11 08:13:27,951 - ERROR - Exception: 'image_meta_dict'
[2023-10-11T08:13:28.005Z] Traceback (most recent call last):
[2023-10-11T08:13:28.005Z]   File "/root/.local/share/virtualenvs/MONAI-zoo-premerge-CHR6GQKS/lib/python3.8/site-packages/ignite/engine/engine.py", line 753, in _internal_run
[2023-10-11T08:13:28.005Z]     time_taken = self._run_once_on_dataset()
[2023-10-11T08:13:28.005Z]   File "/root/.local/share/virtualenvs/MONAI-zoo-premerge-CHR6GQKS/lib/python3.8/site-packages/ignite/engine/engine.py", line 854, in _run_once_on_dataset
[2023-10-11T08:13:28.005Z]     self._handle_exception(e)
[2023-10-11T08:13:28.005Z]   File "/root/.local/share/virtualenvs/MONAI-zoo-premerge-CHR6GQKS/lib/python3.8/site-packages/ignite/engine/engine.py", line 464, in _handle_exception
[2023-10-11T08:13:28.005Z]     self._fire_event(Events.EXCEPTION_RAISED, e)
[2023-10-11T08:13:28.005Z]   File "/root/.local/share/virtualenvs/MONAI-zoo-premerge-CHR6GQKS/lib/python3.8/site-packages/ignite/engine/engine.py", line 421, in _fire_event
[2023-10-11T08:13:28.005Z]     func(*first, *(event_args + others), **kwargs)
[2023-10-11T08:13:28.005Z]   File "/root/.local/share/virtualenvs/MONAI-zoo-premerge-CHR6GQKS/lib/python3.8/site-packages/monai/handlers/stats_handler.py", line 202, in exception_raised
[2023-10-11T08:13:28.005Z]     raise e
[2023-10-11T08:13:28.005Z]   File "/root/.local/share/virtualenvs/MONAI-zoo-premerge-CHR6GQKS/lib/python3.8/site-packages/ignite/engine/engine.py", line 841, in _run_once_on_dataset
[2023-10-11T08:13:28.005Z]     self._fire_event(Events.ITERATION_COMPLETED)
[2023-10-11T08:13:28.005Z]   File "/root/.local/share/virtualenvs/MONAI-zoo-premerge-CHR6GQKS/lib/python3.8/site-packages/ignite/engine/engine.py", line 421, in _fire_event
[2023-10-11T08:13:28.005Z]     func(*first, *(event_args + others), **kwargs)
[2023-10-11T08:13:28.005Z]   File "/root/.local/share/virtualenvs/MONAI-zoo-premerge-CHR6GQKS/lib/python3.8/site-packages/monai/handlers/metrics_saver.py", line 124, in _get_filenames
[2023-10-11T08:13:28.005Z]     meta_data = self.batch_transform(engine.state.batch)
[2023-10-11T08:13:28.005Z]   File "/root/.local/share/virtualenvs/MONAI-zoo-premerge-CHR6GQKS/lib/python3.8/site-packages/monai/handlers/utils.py", line 199, in _wrapper
[2023-10-11T08:13:28.005Z]     ret = [data[0][k] if first else [i[k] for i in data] for k in _keys]
[2023-10-11T08:13:28.005Z]   File "/root/.local/share/virtualenvs/MONAI-zoo-premerge-CHR6GQKS/lib/python3.8/site-packages/monai/handlers/utils.py", line 199, in <listcomp>
[2023-10-11T08:13:28.005Z]     ret = [data[0][k] if first else [i[k] for i in data] for k in _keys]
[2023-10-11T08:13:28.005Z]   File "/root/.local/share/virtualenvs/MONAI-zoo-premerge-CHR6GQKS/lib/python3.8/site-packages/monai/handlers/utils.py", line 199, in <listcomp>
[2023-10-11T08:13:28.005Z]     ret = [data[0][k] if first else [i[k] for i in data] for k in _keys]
[2023-10-11T08:13:28.005Z] KeyError: 'image_meta_dict'

wyli commented 9 months ago

possible fix Screenshot 2023-10-11 at 09 43 14

wyli commented 9 months ago

/build

mingxin-zheng commented 9 months ago

/build

yiheng-wang-nv commented 9 months ago

/build

wyli commented 9 months ago

/build