Open cmcamdy opened 2 years ago
From your hook level, the priority of evalhook is NORMAL. Please double-check your code for the training launch.
have you solved it?
have you solved it?
I do not solved this problem, because I do not sure where the priority set I should modify. But I try to modify the schedule config file to avoid this situation, eg.
checkpoint_config = dict(by_epoch=False, interval=4000)
evaluation = dict(interval=4000, metric='mIoU')
checkpoint_config = dict(by_epoch=False, interval=4000) evaluation = dict(interval=4001, metric='mIoU')
The level of eval hook is low, like
(VERY_HIGH ) PolyLrUpdaterHook
(NORMAL ) CheckpointHook
(LOW ) DistEvalHook
(VERY_LOW ) TextLoggerHook
--------------------
before_train_epoch:
(VERY_HIGH ) PolyLrUpdaterHook
(LOW ) IterTimerHook
(LOW ) DistEvalHook
(VERY_LOW ) TextLoggerHook
--------------------
before_train_iter:
(VERY_HIGH ) PolyLrUpdaterHook
(LOW ) IterTimerHook
(LOW ) DistEvalHook
--------------------
after_train_iter:
(ABOVE_NORMAL) OptimizerHook
(NORMAL ) CheckpointHook
(LOW ) IterTimerHook
(LOW ) DistEvalHook
(VERY_LOW ) TextLoggerHook
--------------------
after_train_epoch:
(NORMAL ) CheckpointHook
(LOW ) DistEvalHook
(VERY_LOW ) TextLoggerHook
--------------------
before_val_epoch:
(LOW ) IterTimerHook
(VERY_LOW ) TextLoggerHook
--------------------
before_val_iter:
(LOW ) IterTimerHook
Hi, I want to share the solution of this issue.
In mmseg/apis/train.py
, plus priority='NORMAL'
so that
runner.register_hook(eval_hook(val_dataloader, save_best='mIoU', **eval_cfg), priority='NORMAL')
When I choose IterBasedRunner in my schedule_cfg, this bug regular presence. And I have tried to follow https://github.com/open-mmlab/mmcv/pull/1252 to fix this, but it does not work. And I have check the file in mmseg/apis/train.py, EvalHook is already set LOW. Here is the trackback
And my Hooks level is
Is there another way that might work?