You may add addition that may be helpful for locating the problem, such as
How you installed PyTorch [e.g., pip, conda, source]
Other environment variables that may be related (such as $PATH, $LD_LIBRARY_PATH, $PYTHONPATH, etc.)
Error traceback
If applicable, paste the error trackback here.
06/14 08:08:07 - mmengine - INFO - paramwise_options -- decode_head.query_feat.weight:lr_mult=1.0
06/14 08:08:07 - mmengine - INFO - paramwise_options -- decode_head.query_feat.weight:decay_mult=0.0
06/14 08:08:07 - mmengine - INFO - paramwise_options -- decode_head.level_embed.weight:lr=0.0001
06/14 08:08:07 - mmengine - INFO - paramwise_options -- decode_head.level_embed.weight:weight_decay=0.0
06/14 08:08:07 - mmengine - INFO - paramwise_options -- decode_head.level_embed.weight:lr_mult=1.0
06/14 08:08:07 - mmengine - INFO - paramwise_options -- decode_head.level_embed.weight:decay_mult=0.0
06/14 08:08:07 - mmengine - WARNING - The prefix is not set in metric class IoUMetric.
06/14 08:08:07 - mmengine - INFO - load model from: torchvision://resnet50
06/14 08:08:07 - mmengine - INFO - Loads checkpoint by torchvision backend from path: torchvision://resnet50
06/14 08:08:07 - mmengine - WARNING - The model and loaded state dict do not match exactly
unexpected key in source state_dict: fc.weight, fc.bias
06/14 08:08:07 - mmengine - WARNING - "FileClient" will be deprecated in future. Please use io functions in https://mmengine.readthedocs.io/en/lates
t/api/fileio.html#file-io
06/14 08:08:07 - mmengine - WARNING - "HardDiskBackend" is the alias of "LocalBackend" and the former will be deprecated in future.
06/14 08:08:07 - mmengine - INFO - Checkpoints will be saved to /home/incar/tms/source/rsdemo/mmsegmentation/work_dirs/mask2former_r50_8xb2-160k_ade
20k-512x512.
/home/incar/miniconda3/envs/openmmlab/lib/python3.8/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it w
ill be required to pass the indexing argument. (Triggered internally at /opt/conda/conda-bld/pytorch_1695392036766/work/aten/src/ATen/native/TensorS
hape.cpp:3526.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
/home/incar/miniconda3/envs/openmmlab/lib/python3.8/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it w
ill be required to pass the indexing argument. (Triggered internally at /opt/conda/conda-bld/pytorch_1695392036766/work/aten/src/ATen/native/TensorS
hape.cpp:3526.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
/home/incar/miniconda3/envs/openmmlab/lib/python3.8/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it w
ill be required to pass the indexing argument. (Triggered internally at /opt/conda/conda-bld/pytorch_1695392036766/work/aten/src/ATen/native/TensorS
hape.cpp:3526.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
/home/incar/miniconda3/envs/openmmlab/lib/python3.8/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it w
ill be required to pass the indexing argument. (Triggered internally at /opt/conda/conda-bld/pytorch_1695392036766/work/aten/src/ATen/native/TensorS
hape.cpp:3526.)
return forward_call(*args, **kwargs) [84/1973]
File "/home/incar/miniconda3/envs/openmmlab/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 1519, in forward
else self._run_ddp_forward(*inputs, **kwargs)
File "/home/incar/miniconda3/envs/openmmlab/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 1355, in _run_ddp_forward
return self.module(*inputs, **kwargs) # type: ignore[index]
File "/home/incar/miniconda3/envs/openmmlab/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/incar/miniconda3/envs/openmmlab/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/home/incar/tms/source/rsdemo/mmsegmentation/mmseg/models/segmentors/base.py", line 94, in forward
return self.loss(inputs, data_samples)
File "/home/incar/tms/source/rsdemo/mmsegmentation/mmseg/models/segmentors/encoder_decoder.py", line 178, in loss
loss_decode = self._decode_head_forward_train(x, data_samples)
File "/home/incar/tms/source/rsdemo/mmsegmentation/mmseg/models/segmentors/encoder_decoder.py", line 139, in _decode_head_forward_train
loss_decode = self.decode_head.loss(inputs, data_samples,
File "/home/incar/tms/source/rsdemo/mmsegmentation/mmseg/models/decode_heads/mask2former_head.py", line 126, in loss
losses = self.loss_by_feat(all_cls_scores, all_mask_preds,
File "/home/incar/tms/source/rsdemo/mmdetection/mmdet/models/dense_heads/maskformer_head.py", line 348, in loss_by_feat
losses_cls, losses_mask, losses_dice = multi_apply(
File "/home/incar/tms/source/rsdemo/mmdetection/mmdet/models/utils/misc.py", line 219, in multi_apply
return tuple(map(list, zip(*map_results)))
File "/home/incar/tms/source/rsdemo/mmdetection/mmdet/models/dense_heads/mask2former_head.py", line 273, in _loss_by_feat_single
avg_factor) = self.get_targets(cls_scores_list, mask_preds_list,
File "/home/incar/tms/source/rsdemo/mmdetection/mmdet/models/dense_heads/maskformer_head.py", line 237, in get_targets
results = multi_apply(self._get_targets_single, cls_scores_list,
File "/home/incar/tms/source/rsdemo/mmdetection/mmdet/models/utils/misc.py", line 219, in multi_apply
return tuple(map(list, zip(*map_results)))
File "/home/incar/tms/source/rsdemo/mmdetection/mmdet/models/dense_heads/mask2former_head.py", line 222, in _get_targets_single
assign_result = self.assigner.assign(
File "/home/incar/tms/source/rsdemo/mmdetection/mmdet/models/task_modules/assigners/hungarian_assigner.py", line 131, in assign
matched_row_inds, matched_col_inds = linear_sum_assignment(cost)
ValueError: cost matrix is infeasible
Bug fix
If you have already identified the reason, you can provide the information here. If you are willing to create a PR to fix it, please also leave a comment here and that would be much appreciated!
Thanks for your error report and we appreciate it a lot.
Checklist
Describe the bug A clear and concise description of what the bug is.
Reproduction
What command or script did you run?
Did you make any modifications on the code or config? Did you understand what you have modified?
no
Environment
python mmseg/utils/collect_env.py
to collect necessary environment information and paste it here.$PATH
,$LD_LIBRARY_PATH
,$PYTHONPATH
, etc.)Error traceback
If applicable, paste the error trackback here.
Bug fix
If you have already identified the reason, you can provide the information here. If you are willing to create a PR to fix it, please also leave a comment here and that would be much appreciated!