JonasSchult / Mask3D

Mask3D predicts accurate 3D semantic instances achieving state-of-the-art on ScanNet, ScanNet200, S3DIS and STPLS3D.
MIT License
525 stars 103 forks source link

`ModelCheckpoint(monitor='val_mean_ap_50')` could not find the monitored key in the returned metrics: ['train_loss_ce', 'train_loss_mask', 'train_loss_dice', ................ #144

Open asadjan1801 opened 10 months ago

asadjan1801 commented 10 months ago

Traceback (most recent call last): File "/home/aurahman/aurahman/Mask3D/main_instance_segmentation.py", line 114, in main() File "/home/aurahman/anaconda3/envs/mask3d_cuda113/lib/python3.10/site-packages/hydra/main.py", line 32, in decorated_main _run_hydra( File "/home/aurahman/anaconda3/envs/mask3d_cuda113/lib/python3.10/site-packages/hydra/_internal/utils.py", line 346, in _run_hydra run_and_report( File "/home/aurahman/anaconda3/envs/mask3d_cuda113/lib/python3.10/site-packages/hydra/_internal/utils.py", line 201, in run_and_report raise ex File "/home/aurahman/anaconda3/envs/mask3d_cuda113/lib/python3.10/site-packages/hydra/_internal/utils.py", line 198, in run_and_report return func() File "/home/aurahman/anaconda3/envs/mask3d_cuda113/lib/python3.10/site-packages/hydra/_internal/utils.py", line 347, in lambda: hydra.run( File "/home/aurahman/anaconda3/envs/mask3d_cuda113/lib/python3.10/site-packages/hydra/_internal/hydra.py", line 107, in run return run_job( File "/home/aurahman/anaconda3/envs/mask3d_cuda113/lib/python3.10/site-packages/hydra/core/utils.py", line 128, in run_job ret.return_value = task_function(task_cfg) File "/home/aurahman/aurahman/Mask3D/main_instance_segmentation.py", line 108, in main train(cfg) File "/home/aurahman/anaconda3/envs/mask3d_cuda113/lib/python3.10/site-packages/hydra/main.py", line 27, in decorated_main return task_function(cfg_passthrough) File "/home/aurahman/aurahman/Mask3D/main_instance_segmentation.py", line 83, in train runner.fit(model) File "/home/aurahman/anaconda3/envs/mask3d_cuda113/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 696, in fit self._call_and_handle_interrupt( File "/home/aurahman/anaconda3/envs/mask3d_cuda113/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 650, in _call_and_handle_interrupt return trainer_fn(*args, *kwargs) File "/home/aurahman/anaconda3/envs/mask3d_cuda113/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 735, in _fit_impl results = self._run(model, ckpt_path=self.ckpt_path) File "/home/aurahman/anaconda3/envs/mask3d_cuda113/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1166, in _run results = self._run_stage() File "/home/aurahman/anaconda3/envs/mask3d_cuda113/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1252, in _run_stage return self._run_train() File "/home/aurahman/anaconda3/envs/mask3d_cuda113/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1283, in _run_train self.fit_loop.run() File "/home/aurahman/anaconda3/envs/mask3d_cuda113/lib/python3.10/site-packages/pytorch_lightning/loops/loop.py", line 201, in run self.on_advance_end() File "/home/aurahman/anaconda3/envs/mask3d_cuda113/lib/python3.10/site-packages/pytorch_lightning/loops/fit_loop.py", line 299, in on_advance_end self.trainer._call_callback_hooks("on_train_epoch_end") File "/home/aurahman/anaconda3/envs/mask3d_cuda113/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1597, in _call_callback_hooks fn(self, self.lightning_module, args, **kwargs) File "/home/aurahman/anaconda3/envs/mask3d_cuda113/lib/python3.10/site-packages/pytorch_lightning/callbacks/model_checkpoint.py", line 311, in on_train_epoch_end self._save_topk_checkpoint(trainer, monitor_candidates) File "/home/aurahman/anaconda3/envs/mask3d_cuda113/lib/python3.10/site-packages/pytorch_lightning/callbacks/model_checkpoint.py", line 380, in _save_topk_checkpoint raise MisconfigurationException(m) pytorch_lightning.utilities.exceptions.MisconfigurationException: ModelCheckpoint(monitor='val_mean_ap_50') could not find the monitored key in the returned metrics: ['train_loss_ce', 'train_loss_mask', 'train_loss_dice', 'train_loss_ce_0', 'train_loss_mask_0', 'train_loss_dice_0', 'train_loss_ce_1', 'train_loss_mask_1', 'train_loss_dice_1', 'train_loss_ce_2', 'train_loss_mask_2', 'train_loss_dice_2', 'train_loss_ce_3', 'train_loss_mask_3', 'train_loss_dice_3', 'train_loss_ce_4', 'train_loss_mask_4', 'train_loss_dice_4', 'train_loss_ce_5', 'train_loss_mask_5', 'train_loss_dice_5', 'train_loss_ce_6', 'train_loss_mask_6', 'train_loss_dice_6', 'train_loss_ce_7', 'train_loss_mask_7', 'train_loss_dice_7', 'train_loss_ce_8', 'train_loss_mask_8', 'train_loss_dice_8', 'train_loss_ce_9', 'train_loss_mask_9', 'train_loss_dice_9', 'train_loss_ce_10', 'train_loss_mask_10', 'train_loss_dice_10', 'train_loss_ce_11', 'train_loss_mask_11', 'train_loss_dice_11', 'train_mean_loss_ce', 'train_mean_loss_mask', 'train_mean_loss_dice', 'train_loss_mean', 'epoch', 'step']. HINT: Did you call log('val_mean_ap_50', value) in the LightningModule?