Closed leo-hao closed 2 years ago
lhoyer
commented
2 years ago Hi @leo-hao,
Thanks for your interest in our work. This error is produced by the logger of the mmcv library. It seems to be related to this bug report: https://github.com/open-mmlab/mmcv/issues/1261. I would assume that you might have installed different versions of mmcv or mmsegmentation than recommended for this repository. Please refer to the readme on how to set up the environment with the correct library versions.
leo-hao
commented
2 years ago Hi @lhoyer , Thank you for your reply. I am very sorry for not being able to reply to you immediately. Thank you so much for your guidance.
zhou-rui1
commented
2 years ago Hi, I am wodering how to fix this problem... should I edit the code for this repo or the installed mmcv source code? Since that I was unable to change to the same versions and I installed mmcv 1.4.4
lhoyer
commented
2 years ago I would recommend setting up a new virtual environment or conda environment and installing mmcv version 1.3.7 in this environment as described in the README.md. If that is no option, please have a look at the mentioned mmcv issue and see if you can implement a fix.
leo-hao
commented
2 years ago 
Hi @zhou-rui1 , I'm sorry reply you late. I just follow the code in the picture, or you can fix it by following the the author suggestion.
kaigelee
commented
2 years ago Thanks for your detailed codes. I am (just) able to fit DAFormer in 10GiB of GPU memory by disabling FD and reducing the crop size from 512x512 to 480x480. However, when it trained [4000/40000], a key error has occurred. Looking forward to your reply!
Best.
2022-01-09 23:01:50,074 - mmseg - INFO - Iter [3950/40000] lr: 5.408e-05, eta: 11:05:38, time: 1.110, data_time: 0.014, memory: 8163, decode.loss_seg: 0.2515, decode.acc_seg: 86.1694, mix.decode.loss_seg: 0.2489, mix.decode.acc_seg: 86.0454 2022-01-09 23:02:46,390 - mmseg - INFO - Exp name: 220109_2148_gta2cs_uda_warm_fdthings_rcs_croppl_a999_daformer_mitb5_s0_33f34 2022-01-09 23:02:46,390 - mmseg - INFO - Iter [4000/40000] lr: 5.400e-05, eta: 11:04:51, time: 1.127, data_time: 0.015, memory: 8163, decode.loss_seg: 0.2198, decode.acc_seg: 85.9952, mix.decode.loss_seg: 0.2331, mix.decode.acc_seg: 86.3791 [>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 500/500, 7.2 task/s, elapsed: 70s, ETA: 0s2022-01-09 23:04:41,465 - mmseg - INFO - per class results: 2022-01-09 23:04:41,468 - mmseg - INFO - +---------------+-------+-------+ | Class | IoU | Acc | +---------------+-------+-------+ | road | 88.63 | 92.44 | | sidewalk | 46.11 | 71.19 | | building | 83.71 | 94.21 | | wall | 27.07 | 33.25 | | fence | 7.09 | 7.36 | | pole | 29.68 | 32.26 | | traffic light | 37.11 | 50.64 | | traffic sign | 26.5 | 27.16 | | vegetation | 87.65 | 94.58 | | terrain | 44.26 | 52.82 | | sky | 85.91 | 97.84 | | person | 62.97 | 84.29 | | rider | 34.35 | 54.13 | | car | 85.22 | 93.03 | | truck | 48.87 | 65.72 | | bus | 47.98 | 77.11 | | train | 16.64 | 17.55 | | motorcycle | 40.64 | 61.24 | | bicycle | 47.22 | 51.02 | +---------------+-------+-------+ 2022-01-09 23:04:41,468 - mmseg - INFO - Summary: 2022-01-09 23:04:41,468 - mmseg - INFO - +-------+-------+-------+ | aAcc | mIoU | mAcc | +-------+-------+-------+ | 88.67 | 49.87 | 60.94 | +-------+-------+-------+ 2022-01-09 23:04:41,562 - mmseg - INFO - Exp name: 220109_2148_gta2cs_uda_warm_fdthings_rcs_croppl_a999_daformer_mitb5_s0_33f34 Traceback (most recent call last): File "run_experiments.py", line 104, in <module> train.main([config_files[i]]) File "/home/data/liuhao/experiments/DAFormer-master/tools/train.py", line 173, in main meta=meta) File "/home/data/liuhao/experiments/DAFormer-master/mmseg/apis/train.py", line 131, in train_segmentor runner.run(data_loaders, cfg.workflow) File "/home/cv428/anaconda3/envs/liuhaommlab/lib/python3.6/site-packages/mmcv/runner/iter_based_runner.py", line 133, in run iter_runner(iter_loaders[i], **kwargs) File "/home/cv428/anaconda3/envs/liuhaommlab/lib/python3.6/site-packages/mmcv/runner/iter_based_runner.py", line 66, in train self.call_hook('after_train_iter') File "/home/cv428/anaconda3/envs/liuhaommlab/lib/python3.6/site-packages/mmcv/runner/base_runner.py", line 307, in call_hook getattr(hook, fn_name)(self) File "/home/cv428/anaconda3/envs/liuhaommlab/lib/python3.6/site-packages/mmcv/runner/hooks/logger/base.py", line 152, in after_train_iter self.log(runner) File "/home/cv428/anaconda3/envs/liuhaommlab/lib/python3.6/site-packages/mmcv/runner/hooks/logger/text.py", line 234, in log self._log_info(log_dict, runner) File "/home/cv428/anaconda3/envs/liuhaommlab/lib/python3.6/site-packages/mmcv/runner/hooks/logger/text.py", line 153, in _log_info log_str += f'time: {log_dict["time"]:.3f}, ' \ KeyError: 'data_time'
Have you obtained the result like the original paper, and How to get?
EudicL
commented
1 year ago Thanks for your detailed codes. I am (just) able to fit DAFormer in 10GiB of GPU memory by disabling FD and reducing the crop size from 512x512 to 480x480. However, when it trained [4000/40000], a key error has occurred. Looking forward to your reply!
Best.
2022-01-09 23:01:50,074 - mmseg - INFO - Iter [3950/40000] lr: 5.408e-05, eta: 11:05:38, time: 1.110, data_time: 0.014, memory: 8163, decode.loss_seg: 0.2515, decode.acc_seg: 86.1694, mix.decode.loss_seg: 0.2489, mix.decode.acc_seg: 86.0454 2022-01-09 23:02:46,390 - mmseg - INFO - Exp name: 220109_2148_gta2cs_uda_warm_fdthings_rcs_croppl_a999_daformer_mitb5_s0_33f34 2022-01-09 23:02:46,390 - mmseg - INFO - Iter [4000/40000] lr: 5.400e-05, eta: 11:04:51, time: 1.127, data_time: 0.015, memory: 8163, decode.loss_seg: 0.2198, decode.acc_seg: 85.9952, mix.decode.loss_seg: 0.2331, mix.decode.acc_seg: 86.3791 [>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 500/500, 7.2 task/s, elapsed: 70s, ETA: 0s2022-01-09 23:04:41,465 - mmseg - INFO - per class results: 2022-01-09 23:04:41,468 - mmseg - INFO - +---------------+-------+-------+ | Class | IoU | Acc | +---------------+-------+-------+ | road | 88.63 | 92.44 | | sidewalk | 46.11 | 71.19 | | building | 83.71 | 94.21 | | wall | 27.07 | 33.25 | | fence | 7.09 | 7.36 | | pole | 29.68 | 32.26 | | traffic light | 37.11 | 50.64 | | traffic sign | 26.5 | 27.16 | | vegetation | 87.65 | 94.58 | | terrain | 44.26 | 52.82 | | sky | 85.91 | 97.84 | | person | 62.97 | 84.29 | | rider | 34.35 | 54.13 | | car | 85.22 | 93.03 | | truck | 48.87 | 65.72 | | bus | 47.98 | 77.11 | | train | 16.64 | 17.55 | | motorcycle | 40.64 | 61.24 | | bicycle | 47.22 | 51.02 | +---------------+-------+-------+ 2022-01-09 23:04:41,468 - mmseg - INFO - Summary: 2022-01-09 23:04:41,468 - mmseg - INFO - +-------+-------+-------+ | aAcc | mIoU | mAcc | +-------+-------+-------+ | 88.67 | 49.87 | 60.94 | +-------+-------+-------+ 2022-01-09 23:04:41,562 - mmseg - INFO - Exp name: 220109_2148_gta2cs_uda_warm_fdthings_rcs_croppl_a999_daformer_mitb5_s0_33f34 Traceback (most recent call last): File "run_experiments.py", line 104, in <module> train.main([config_files[i]]) File "/home/data/liuhao/experiments/DAFormer-master/tools/train.py", line 173, in main meta=meta) File "/home/data/liuhao/experiments/DAFormer-master/mmseg/apis/train.py", line 131, in train_segmentor runner.run(data_loaders, cfg.workflow) File "/home/cv428/anaconda3/envs/liuhaommlab/lib/python3.6/site-packages/mmcv/runner/iter_based_runner.py", line 133, in run iter_runner(iter_loaders[i], **kwargs) File "/home/cv428/anaconda3/envs/liuhaommlab/lib/python3.6/site-packages/mmcv/runner/iter_based_runner.py", line 66, in train self.call_hook('after_train_iter') File "/home/cv428/anaconda3/envs/liuhaommlab/lib/python3.6/site-packages/mmcv/runner/base_runner.py", line 307, in call_hook getattr(hook, fn_name)(self) File "/home/cv428/anaconda3/envs/liuhaommlab/lib/python3.6/site-packages/mmcv/runner/hooks/logger/base.py", line 152, in after_train_iter self.log(runner) File "/home/cv428/anaconda3/envs/liuhaommlab/lib/python3.6/site-packages/mmcv/runner/hooks/logger/text.py", line 234, in log self._log_info(log_dict, runner) File "/home/cv428/anaconda3/envs/liuhaommlab/lib/python3.6/site-packages/mmcv/runner/hooks/logger/text.py", line 153, in _log_info log_str += f'time: {log_dict["time"]:.3f}, ' \ KeyError: 'data_time'
Hello, may i ask how you fixed the problem?
Thanks for your detailed codes. I am (just) able to fit DAFormer in 10GiB of GPU memory by disabling FD and reducing the crop size from 512x512 to 480x480. However, when it trained [4000/40000], a key error has occurred. Looking forward to your reply!
Best.