Question about MMSegmentation version

[letme-hj]

Hi, Thank you for the work and sharing code! I find it very helpful :)

I have one question though, I am trying to reproduce your result on ResNet50 + Potsdam dataset (Semantic segmentation), so I followed the instructions you mentioned in SemanticSegmentation. (I used your pretrained weights that's trained on MillionAID for the backbone.)

That is,

• I cloned mmsegmentation from https://github.com/open-mmlab/mmsegmentation,
• and switched the mmseg version to v0.27.0 (I referred to the environment file in Pretrain/environment.yml).
• and replaced the files that correspond to the files you provided.

However I completely fail to train the model...

I wonder if I used the right version of mmsegmentation. (I'm confused since the environment.yml is only in the Pretrain folder) Or is there any other lines that I have to be aware of?

Thank you in advance for your help!

[pUmpKin-Co]

Hi~Thanks for your interest in our work. It seems that the network doesn't learn anything (the loss curve doesn't make any changes). I reconfirmed that the version of MMsegmentation is 0.27.0. Please try run for scratch to confirm the correctenss of the whole workflow (lie the path of data, model, etc.). And then make sure that the pre-trained checkpoint is loaded correctly. If this issue still exits, I hope you can share the log and configuration files. I would try to see if there are any problems with them.

Thanks again :)

[letme-hj]

Hi, thank you for the quick response!

Actually, I already tried to train from scratch and it still does not work. Here is the config file and log file:

config

```python log_config = dict( interval=50, hooks=[ dict(type='TextLoggerHook', by_epoch=False), dict(type='TensorboardLoggerHook'), dict(type='MMSegWandbHook', init_kwargs=dict(project='segmentation_potsdam', name='mmseg0.27.0')) ]) dist_params = dict(backend='nccl') log_level = 'INFO' load_from = None resume_from = None workflow = [('train', 1)] cudnn_benchmark = True optimizer = dict(type='SGD', lr=0.01, momentum=0.9, weight_decay=0.0001) optimizer_config = None lr_config = dict( policy='CosineAnnealing', warmup='linear', warmup_iters=100, min_lr=0.000001, by_epoch=False) runner = dict(type='EpochBasedRunner', max_epochs=50) checkpoint_config = dict(by_epoch=True, interval=10) evaluation = dict(interval=5, metric=['mIoU', "mFscore"], pre_eval=True, by_epoch=True) norm_cfg = dict(type='BN', requires_grad=True) checkpoint_path = None # change this to path of cmid_pretrained_resnet50 model = dict( type='EncoderDecoder', pretrained=checkpoint_path, backbone=dict( type='ResNet', depth=50, num_stages=4, out_indices=(0, 1, 2, 3), dilations=(1, 1, 1, 1), strides=(1, 2, 2, 2), norm_cfg=dict(type='BN', requires_grad=True), norm_eval=False, style='pytorch', contract_dilation=True), decode_head=dict( type='UPerHead', in_channels=[256, 512, 1024, 2048], in_index=[0, 1, 2, 3], pool_scales=(1, 2, 3, 6), channels=512, ignore_index=255, dropout_ratio=0.1, num_classes=6, norm_cfg=dict(type='BN', requires_grad=True), align_corners=False, loss_decode=dict( type='CrossEntropyLoss', use_sigmoid=False, loss_weight=1.0)), auxiliary_head=dict( type='FCNHead', in_channels=1024, in_index=2, channels=256, ignore_index=255, num_convs=1, concat_input=False, dropout_ratio=0.1, num_classes=6, norm_cfg=dict(type='BN', requires_grad=True), align_corners=False, loss_decode=dict( type='CrossEntropyLoss', use_sigmoid=False, loss_weight=0.4)), train_cfg=dict(), test_cfg=dict(mode='whole')) # dataset_type = 'PotsdamAllDataset' dataset_type="PotsdamDataset" data_root = '/workspace/SSL_RS/data/segmentation/potsdam' # data_root = 'F:\VisionData\Potsdam\mmseg' img_norm_cfg = dict( mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_rgb=True) crop_size = (512, 512) train_pipeline = [ dict(type='LoadImageFromFile'), # dict(type='LoadAnnotations', reduce_zero_label=True), dict(type='LoadAnnotationsReduceIgnoreIndex', reduce_zero_label=True, ignore_index=6), dict(type='Resize', img_scale=(512, 512), ratio_range=(0.5, 2.0)), dict(type='RandomCrop', crop_size=(512, 512), cat_max_ratio=0.75), dict(type='RandomFlip', prob=0.5), dict(type='PhotoMetricDistortion'), dict( type='Normalize', mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_rgb=True), dict(type='Pad', size=(512, 512), pad_val=0, seg_pad_val=255), dict(type='DefaultFormatBundle'), dict(type='Collect', keys=['img', 'gt_semantic_seg']) ] test_pipeline = [ dict(type='LoadImageFromFile'), dict( type='MultiScaleFlipAug', img_scale=(512, 512), flip=False, transforms=[ dict(type='Resize', keep_ratio=True), dict(type='RandomFlip'), dict( type='Normalize', mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_rgb=True), dict(type='ImageToTensor', keys=['img']), dict(type='Collect', keys=['img']) ]) ] data = dict( samples_per_gpu=8, workers_per_gpu=4, train=dict( type=dataset_type, data_root=data_root, img_dir='img_dir/train', ann_dir='ann_dir/train', pipeline=train_pipeline), val=dict( type=dataset_type, data_root=data_root, img_dir='img_dir/val', ann_dir='ann_dir/val', pipeline=test_pipeline), test=dict( type=dataset_type, data_root=data_root, img_dir='img_dir/val', ann_dir='ann_dir/val', pipeline=test_pipeline)) ```

log file (when trained from scratch)

``` 2023-06-21 05:36:01,309 - mmseg - INFO - workflow: [('train', 1)], max: 50 epochs 2023-06-21 05:36:01,309 - mmseg - INFO - Checkpoints will be saved to /workspace/CMID/SemanticSegmentation/mmsegmentation/logs/potsdam-pretrained_cmid-resnet-v0.27.0 by HardDiskBackend. 2023-06-21 05:36:45,305 - mmseg - INFO - Iter [50/21600] lr: 5.410e-03, eta: 1:52:20, time: 0.313, data_time: 0.060, memory: 22159, decode.loss_ce: 1.3476, decode.acc_seg: 11.0600, aux.loss_ce: 0.5207, aux.acc_seg: 17.1900, loss: 1.8683 2023-06-21 05:36:52,574 - mmseg - INFO - Iter [100/21600] lr: 9.909e-03, eta: 1:22:05, time: 0.145, data_time: 0.009, memory: 22159, decode.loss_ce: 1.3627, decode.acc_seg: 10.9382, aux.loss_ce: 0.5264, aux.acc_seg: 16.3224, loss: 1.8891 2023-06-21 05:36:59,490 - mmseg - INFO - Iter [150/21600] lr: 9.999e-03, eta: 1:11:04, time: 0.138, data_time: 0.007, memory: 22159, decode.loss_ce: 1.3416, decode.acc_seg: 11.9290, aux.loss_ce: 0.5199, aux.acc_seg: 16.4487, loss: 1.8615 2023-06-21 05:37:06,540 - mmseg - INFO - Iter [200/21600] lr: 9.998e-03, eta: 1:05:45, time: 0.141, data_time: 0.008, memory: 22159, decode.loss_ce: 1.2669, decode.acc_seg: 11.8965, aux.loss_ce: 0.4904, aux.acc_seg: 16.9658, loss: 1.7573 2023-06-21 05:37:13,184 - mmseg - INFO - Iter [250/21600] lr: 9.997e-03, eta: 1:01:56, time: 0.133, data_time: 0.006, memory: 22159, decode.loss_ce: 1.3371, decode.acc_seg: 11.8680, aux.loss_ce: 0.5175, aux.acc_seg: 16.7662, loss: 1.8545 2023-06-21 05:37:19,979 - mmseg - INFO - Iter [300/21600] lr: 9.995e-03, eta: 0:59:32, time: 0.136, data_time: 0.007, memory: 22159, decode.loss_ce: 1.3190, decode.acc_seg: 11.2785, aux.loss_ce: 0.5102, aux.acc_seg: 16.3988, loss: 1.8293 2023-06-21 05:37:26,521 - mmseg - INFO - Iter [350/21600] lr: 9.994e-03, eta: 0:57:31, time: 0.131, data_time: 0.005, memory: 22159, decode.loss_ce: 1.3679, decode.acc_seg: 11.2241, aux.loss_ce: 0.5279, aux.acc_seg: 16.3586, loss: 1.8958 2023-06-21 05:37:33,547 - mmseg - INFO - Iter [400/21600] lr: 9.992e-03, eta: 0:56:25, time: 0.141, data_time: 0.008, memory: 22159, decode.loss_ce: 1.3378, decode.acc_seg: 11.8004, aux.loss_ce: 0.5183, aux.acc_seg: 16.2972, loss: 1.8561 2023-06-21 05:37:42,321 - mmseg - INFO - Iter [450/21600] lr: 9.989e-03, eta: 1:00:01, time: 0.255, data_time: 0.132, memory: 22159, decode.loss_ce: 1.3461, decode.acc_seg: 11.4840, aux.loss_ce: 0.5234, aux.acc_seg: 14.9932, loss: 1.8695 2023-06-21 05:37:49,228 - mmseg - INFO - Iter [500/21600] lr: 9.987e-03, eta: 0:58:45, time: 0.138, data_time: 0.007, memory: 22159, decode.loss_ce: 1.3548, decode.acc_seg: 12.0710, aux.loss_ce: 0.5252, aux.acc_seg: 16.5237, loss: 1.8799 2023-06-21 05:37:56,548 - mmseg - INFO - Iter [550/21600] lr: 9.984e-03, eta: 0:57:57, time: 0.146, data_time: 0.008, memory: 22159, decode.loss_ce: 1.3107, decode.acc_seg: 11.4908, aux.loss_ce: 0.5069, aux.acc_seg: 16.8754, loss: 1.8176 2023-06-21 05:38:03,868 - mmseg - INFO - Iter [600/21600] lr: 9.981e-03, eta: 0:57:15, time: 0.146, data_time: 0.009, memory: 22159, decode.loss_ce: 1.3658, decode.acc_seg: 11.6690, aux.loss_ce: 0.5292, aux.acc_seg: 16.1067, loss: 1.8951 2023-06-21 05:38:10,948 - mmseg - INFO - Iter [650/21600] lr: 9.978e-03, eta: 0:56:32, time: 0.142, data_time: 0.009, memory: 22159, decode.loss_ce: 1.3132, decode.acc_seg: 11.6799, aux.loss_ce: 0.5072, aux.acc_seg: 16.9193, loss: 1.8204 2023-06-21 05:38:17,813 - mmseg - INFO - Iter [700/21600] lr: 9.974e-03, eta: 0:55:47, time: 0.137, data_time: 0.007, memory: 22159, decode.loss_ce: 1.3584, decode.acc_seg: 11.2004, aux.loss_ce: 0.5263, aux.acc_seg: 16.3965, loss: 1.8847 2023-06-21 05:38:24,708 - mmseg - INFO - Iter [750/21600] lr: 9.970e-03, eta: 0:55:08, time: 0.138, data_time: 0.008, memory: 22159, decode.loss_ce: 1.4056, decode.acc_seg: 10.9687, aux.loss_ce: 0.5436, aux.acc_seg: 16.6404, loss: 1.9492 2023-06-21 05:38:31,226 - mmseg - INFO - Iter [800/21600] lr: 9.966e-03, eta: 0:54:23, time: 0.130, data_time: 0.006, memory: 22159, decode.loss_ce: 1.3546, decode.acc_seg: 10.2988, aux.loss_ce: 0.5202, aux.acc_seg: 17.9410, loss: 1.8747 2023-06-21 05:38:37,610 - mmseg - INFO - Iter [850/21600] lr: 9.962e-03, eta: 0:53:40, time: 0.128, data_time: 0.005, memory: 22159, decode.loss_ce: 1.3372, decode.acc_seg: 11.6438, aux.loss_ce: 0.5180, aux.acc_seg: 16.3310, loss: 1.8552 2023-06-21 05:38:46,888 - mmseg - INFO - Iter [900/21600] lr: 9.957e-03, eta: 0:54:33, time: 0.208, data_time: 0.072, memory: 22159, decode.loss_ce: 1.3274, decode.acc_seg: 11.3113, aux.loss_ce: 0.5121, aux.acc_seg: 17.2288, loss: 1.8395 2023-06-21 05:38:53,983 - mmseg - INFO - Iter [950/21600] lr: 9.952e-03, eta: 0:54:07, time: 0.142, data_time: 0.008, memory: 22159, decode.loss_ce: 1.3557, decode.acc_seg: 11.2974, aux.loss_ce: 0.5256, aux.acc_seg: 16.4664, loss: 1.8813 2023-06-21 05:39:01,067 - mmseg - INFO - Exp name: CMID-ResNet50_config.py 2023-06-21 05:39:01,067 - mmseg - INFO - Iter [1000/21600] lr: 9.947e-03, eta: 0:53:43, time: 0.142, data_time: 0.008, memory: 22159, decode.loss_ce: 1.3036, decode.acc_seg: 11.1519, aux.loss_ce: 0.5031, aux.acc_seg: 16.5269, loss: 1.8067 2023-06-21 05:39:08,357 - mmseg - INFO - Iter [1050/21600] lr: 9.942e-03, eta: 0:53:25, time: 0.146, data_time: 0.009, memory: 22159, decode.loss_ce: 1.3282, decode.acc_seg: 11.1486, aux.loss_ce: 0.5150, aux.acc_seg: 16.3076, loss: 1.8432 2023-06-21 05:39:15,459 - mmseg - INFO - Iter [1100/21600] lr: 9.936e-03, eta: 0:53:04, time: 0.142, data_time: 0.010, memory: 22159, decode.loss_ce: 1.3828, decode.acc_seg: 10.7996, aux.loss_ce: 0.5328, aux.acc_seg: 17.9513, loss: 1.9155 2023-06-21 05:39:22,854 - mmseg - INFO - Iter [1150/21600] lr: 9.930e-03, eta: 0:52:50, time: 0.148, data_time: 0.010, memory: 22159, decode.loss_ce: 1.3137, decode.acc_seg: 11.0442, aux.loss_ce: 0.5077, aux.acc_seg: 15.8702, loss: 1.8214 2023-06-21 05:39:29,978 - mmseg - INFO - Iter [1200/21600] lr: 9.924e-03, eta: 0:52:31, time: 0.143, data_time: 0.008, memory: 22159, decode.loss_ce: 1.3316, decode.acc_seg: 11.2525, aux.loss_ce: 0.5159, aux.acc_seg: 15.9931, loss: 1.8476 2023-06-21 05:39:36,957 - mmseg - INFO - Iter [1250/21600] lr: 9.918e-03, eta: 0:52:12, time: 0.140, data_time: 0.007, memory: 22159, decode.loss_ce: 1.3672, decode.acc_seg: 11.3193, aux.loss_ce: 0.5288, aux.acc_seg: 16.3317, loss: 1.8961 2023-06-21 05:39:46,293 - mmseg - INFO - Iter [1300/21600] lr: 9.911e-03, eta: 0:59:40, time: 0.738, data_time: 0.597, memory: 22159, decode.loss_ce: 1.2988, decode.acc_seg: 9.6013, aux.loss_ce: 0.5018, aux.acc_seg: 15.9931, loss: 1.8006 2023-06-21 05:39:53,530 - mmseg - INFO - Iter [1350/21600] lr: 9.904e-03, eta: 0:59:07, time: 0.145, data_time: 0.009, memory: 22159, decode.loss_ce: 1.3088, decode.acc_seg: 11.5013, aux.loss_ce: 0.5069, aux.acc_seg: 15.9626, loss: 1.8157 2023-06-21 05:40:00,448 - mmseg - INFO - Iter [1400/21600] lr: 9.897e-03, eta: 0:58:32, time: 0.138, data_time: 0.006, memory: 22159, decode.loss_ce: 1.3525, decode.acc_seg: 10.5520, aux.loss_ce: 0.5225, aux.acc_seg: 17.1467, loss: 1.8750 2023-06-21 05:40:07,327 - mmseg - INFO - Iter [1450/21600] lr: 9.889e-03, eta: 0:57:58, time: 0.138, data_time: 0.006, memory: 22159, decode.loss_ce: 1.3537, decode.acc_seg: 11.3899, aux.loss_ce: 0.5247, aux.acc_seg: 16.9683, loss: 1.8784 2023-06-21 05:40:14,228 - mmseg - INFO - Iter [1500/21600] lr: 9.882e-03, eta: 0:57:26, time: 0.138, data_time: 0.006, memory: 22159, decode.loss_ce: 1.3231, decode.acc_seg: 11.8089, aux.loss_ce: 0.5129, aux.acc_seg: 16.2825, loss: 1.8360 2023-06-21 05:40:21,134 - mmseg - INFO - Iter [1550/21600] lr: 9.874e-03, eta: 0:56:56, time: 0.138, data_time: 0.006, memory: 22159, decode.loss_ce: 1.3694, decode.acc_seg: 11.7338, aux.loss_ce: 0.5308, aux.acc_seg: 16.2170, loss: 1.9002 2023-06-21 05:40:28,041 - mmseg - INFO - Iter [1600/21600] lr: 9.865e-03, eta: 0:56:28, time: 0.138, data_time: 0.008, memory: 22159, decode.loss_ce: 1.3525, decode.acc_seg: 11.5205, aux.loss_ce: 0.5221, aux.acc_seg: 17.4177, loss: 1.8746 2023-06-21 05:40:34,935 - mmseg - INFO - Iter [1650/21600] lr: 9.857e-03, eta: 0:56:00, time: 0.138, data_time: 0.008, memory: 22159, decode.loss_ce: 1.3152, decode.acc_seg: 11.7278, aux.loss_ce: 0.5101, aux.acc_seg: 15.2592, loss: 1.8253 2023-06-21 05:40:41,639 - mmseg - INFO - Iter [1700/21600] lr: 9.848e-03, eta: 0:55:31, time: 0.134, data_time: 0.006, memory: 22159, decode.loss_ce: 1.3417, decode.acc_seg: 11.3749, aux.loss_ce: 0.5181, aux.acc_seg: 17.3311, loss: 1.8597 2023-06-21 05:40:50,713 - mmseg - INFO - Iter [1750/21600] lr: 9.839e-03, eta: 0:56:08, time: 0.246, data_time: 0.113, memory: 22159, decode.loss_ce: 1.3198, decode.acc_seg: 12.4731, aux.loss_ce: 0.5108, aux.acc_seg: 18.2419, loss: 1.8306 2023-06-21 05:40:57,622 - mmseg - INFO - Iter [1800/21600] lr: 9.830e-03, eta: 0:55:42, time: 0.138, data_time: 0.008, memory: 22159, decode.loss_ce: 1.3614, decode.acc_seg: 11.2585, aux.loss_ce: 0.5254, aux.acc_seg: 17.3565, loss: 1.8868 2023-06-21 05:41:04,372 - mmseg - INFO - Iter [1850/21600] lr: 9.820e-03, eta: 0:55:15, time: 0.135, data_time: 0.007, memory: 22159, decode.loss_ce: 1.3611, decode.acc_seg: 10.8874, aux.loss_ce: 0.5245, aux.acc_seg: 16.8103, loss: 1.8857 2023-06-21 05:41:11,166 - mmseg - INFO - Iter [1900/21600] lr: 9.811e-03, eta: 0:54:50, time: 0.136, data_time: 0.007, memory: 22159, decode.loss_ce: 1.3927, decode.acc_seg: 11.8865, aux.loss_ce: 0.5392, aux.acc_seg: 16.7155, loss: 1.9320 2023-06-21 05:41:19,634 - mmseg - INFO - Iter [1950/21600] lr: 9.800e-03, eta: 0:54:43, time: 0.169, data_time: 0.005, memory: 22159, decode.loss_ce: 1.3527, decode.acc_seg: 11.6378, aux.loss_ce: 0.5239, aux.acc_seg: 16.9217, loss: 1.8767 2023-06-21 05:41:29,520 - mmseg - INFO - Exp name: CMID-ResNet50_config.py 2023-06-21 05:41:29,521 - mmseg - INFO - Iter [2000/21600] lr: 9.790e-03, eta: 0:54:50, time: 0.198, data_time: 0.005, memory: 22159, decode.loss_ce: 1.3302, decode.acc_seg: 11.6891, aux.loss_ce: 0.5157, aux.acc_seg: 16.3458, loss: 1.8459 2023-06-21 05:41:40,536 - mmseg - INFO - Iter [2050/21600] lr: 9.780e-03, eta: 0:55:06, time: 0.220, data_time: 0.007, memory: 22159, decode.loss_ce: 1.3072, decode.acc_seg: 12.0301, aux.loss_ce: 0.5058, aux.acc_seg: 16.0055, loss: 1.8130 2023-06-21 05:41:51,571 - mmseg - INFO - Iter [2100/21600] lr: 9.769e-03, eta: 0:55:22, time: 0.221, data_time: 0.006, memory: 22159, decode.loss_ce: 1.3216, decode.acc_seg: 11.0571, aux.loss_ce: 0.5106, aux.acc_seg: 17.6822, loss: 1.8322 2023-06-21 05:42:02,159 - mmseg - INFO - Iter [2150/21600] lr: 9.758e-03, eta: 0:55:32, time: 0.212, data_time: 0.007, memory: 22159, decode.loss_ce: 1.2913, decode.acc_seg: 11.8627, aux.loss_ce: 0.5008, aux.acc_seg: 16.1086, loss: 1.7922 2023-06-21 05:43:31,665 - mmseg - INFO - per class results: 2023-06-21 05:43:31,667 - mmseg - INFO - +--------------------+-------+-------+--------+-----------+--------+ | Class | IoU | Acc | Fscore | Precision | Recall | +--------------------+-------+-------+--------+-----------+--------+ | impervious_surface | 3.95 | 4.18 | 7.6 | 42.42 | 4.18 | | building | 13.0 | 19.12 | 23.0 | 28.87 | 19.12 | | low_vegetation | 0.47 | 0.5 | 0.94 | 7.97 | 0.5 | | tree | 19.93 | 45.98 | 33.24 | 26.03 | 45.98 | | car | 2.67 | 20.25 | 5.2 | 2.98 | 20.25 | +--------------------+-------+-------+--------+-----------+--------+ 2023-06-21 05:43:31,667 - mmseg - INFO - Summary: 2023-06-21 05:43:31,667 - mmseg - INFO - +-------+------+------+---------+------------+---------+ | aAcc | mIoU | mAcc | mFscore | mPrecision | mRecall | +-------+------+------+---------+------------+---------+ | 15.05 | 8.0 | 18.0 | 14.0 | 21.65 | 18.0 | +-------+------+------+---------+------------+---------+ 2023-06-21 05:43:31,668 - mmseg - INFO - Iter(val) [2016] aAcc: 0.1505, mIoU: 0.0800, mAcc: 0.1800, mFscore: 0.1400, mPrecision: 0.2165, mRecall: 0.1800, IoU.impervious_surface: 0.0395, IoU.building: 0.1300, IoU.low_vegetation: 0.0047, IoU.tree: 0.1993, IoU.car: 0.0267, Acc.impervious_surface: 0.0418, Acc.building: 0.1912, Acc.low_vegetation: 0.0050, Acc.tree: 0.4598, Acc.car: 0.2025, Fscore.impervious_surface: 0.0760, Fscore.building: 0.2300, Fscore.low_vegetation: 0.0094, Fscore.tree: 0.3324, Fscore.car: 0.0520, Precision.impervious_surface: 0.4242, Precision.building: 0.2887, Precision.low_vegetation: 0.0797, Precision.tree: 0.2603, Precision.car: 0.0298, Recall.impervious_surface: 0.0418, Recall.building: 0.1912, Recall.low_vegetation: 0.0050, Recall.tree: 0.4598, Recall.car: 0.2025 2023-06-21 05:43:42,992 - mmseg - INFO - Iter [2200/21600] lr: 9.746e-03, eta: 0:56:13, time: 0.283, data_time: 0.066, memory: 22159, decode.loss_ce: 1.3297, decode.acc_seg: 11.8913, aux.loss_ce: 0.5152, aux.acc_seg: 16.4342, loss: 1.8450 2023-06-21 05:43:54,118 - mmseg - INFO - Iter [2250/21600] lr: 9.735e-03, eta: 0:56:25, time: 0.223, data_time: 0.007, memory: 22159, decode.loss_ce: 1.3375, decode.acc_seg: 12.0845, aux.loss_ce: 0.5179, aux.acc_seg: 17.1187, loss: 1.8554 2023-06-21 05:44:05,368 - mmseg - INFO - Iter [2300/21600] lr: 9.723e-03, eta: 0:56:37, time: 0.225, data_time: 0.008, memory: 22159, decode.loss_ce: 1.3430, decode.acc_seg: 11.1489, aux.loss_ce: 0.5182, aux.acc_seg: 16.6155, loss: 1.8612 2023-06-21 05:44:14,885 - mmseg - INFO - Iter [2350/21600] lr: 9.711e-03, eta: 0:56:34, time: 0.190, data_time: 0.007, memory: 22159, decode.loss_ce: 1.3658, decode.acc_seg: 10.8118, aux.loss_ce: 0.5285, aux.acc_seg: 16.8646, loss: 1.8943 2023-06-21 05:44:25,812 - mmseg - INFO - Iter [2400/21600] lr: 9.699e-03, eta: 0:56:42, time: 0.219, data_time: 0.007, memory: 22159, decode.loss_ce: 1.3473, decode.acc_seg: 11.4471, aux.loss_ce: 0.5204, aux.acc_seg: 17.5463, loss: 1.8677 2023-06-21 05:44:36,613 - mmseg - INFO - Iter [2450/21600] lr: 9.686e-03, eta: 0:56:48, time: 0.216, data_time: 0.006, memory: 22159, decode.loss_ce: 1.3393, decode.acc_seg: 11.5049, aux.loss_ce: 0.5184, aux.acc_seg: 17.7023, loss: 1.8577 2023-06-21 05:44:47,619 - mmseg - INFO - Iter [2500/21600] lr: 9.673e-03, eta: 0:56:56, time: 0.220, data_time: 0.007, memory: 22159, decode.loss_ce: 1.3191, decode.acc_seg: 11.0201, aux.loss_ce: 0.5093, aux.acc_seg: 17.5929, loss: 1.8284 2023-06-21 05:44:58,983 - mmseg - INFO - Iter [2550/21600] lr: 9.660e-03, eta: 0:57:05, time: 0.227, data_time: 0.008, memory: 22159, decode.loss_ce: 1.3554, decode.acc_seg: 11.8114, aux.loss_ce: 0.5269, aux.acc_seg: 14.7027, loss: 1.8823 2023-06-21 05:45:11,794 - mmseg - INFO - Iter [2600/21600] lr: 9.647e-03, eta: 0:58:46, time: 0.482, data_time: 0.298, memory: 22159, decode.loss_ce: 1.3804, decode.acc_seg: 10.4700, aux.loss_ce: 0.5303, aux.acc_seg: 16.0008, loss: 1.9107 2023-06-21 05:45:22,007 - mmseg - INFO - Iter [2650/21600] lr: 9.634e-03, eta: 0:58:44, time: 0.204, data_time: 0.006, memory: 22159, decode.loss_ce: 1.3235, decode.acc_seg: 11.3761, aux.loss_ce: 0.5119, aux.acc_seg: 16.7344, loss: 1.8354 2023-06-21 05:45:33,108 - mmseg - INFO - Iter [2700/21600] lr: 9.620e-03, eta: 0:58:47, time: 0.222, data_time: 0.007, memory: 22159, decode.loss_ce: 1.3361, decode.acc_seg: 10.9783, aux.loss_ce: 0.5167, aux.acc_seg: 16.1323, loss: 1.8528 2023-06-21 05:45:42,626 - mmseg - INFO - Iter [2750/21600] lr: 9.606e-03, eta: 0:58:39, time: 0.190, data_time: 0.006, memory: 22159, decode.loss_ce: 1.3719, decode.acc_seg: 11.5767, aux.loss_ce: 0.5302, aux.acc_seg: 17.0130, loss: 1.9021 2023-06-21 05:45:53,194 - mmseg - INFO - Iter [2800/21600] lr: 9.591e-03, eta: 0:58:38, time: 0.211, data_time: 0.008, memory: 22159, decode.loss_ce: 1.3578, decode.acc_seg: 10.7895, aux.loss_ce: 0.5250, aux.acc_seg: 16.9505, loss: 1.8828 2023-06-21 05:46:04,169 - mmseg - INFO - Iter [2850/21600] lr: 9.577e-03, eta: 0:58:39, time: 0.219, data_time: 0.008, memory: 22159, decode.loss_ce: 1.2900, decode.acc_seg: 11.2161, aux.loss_ce: 0.4984, aux.acc_seg: 16.8945, loss: 1.7884 2023-06-21 05:46:15,117 - mmseg - INFO - Iter [2900/21600] lr: 9.562e-03, eta: 0:58:40, time: 0.219, data_time: 0.007, memory: 22159, decode.loss_ce: 1.2977, decode.acc_seg: 10.7824, aux.loss_ce: 0.5011, aux.acc_seg: 16.9338, loss: 1.7988 2023-06-21 05:46:26,314 - mmseg - INFO - Iter [2950/21600] lr: 9.547e-03, eta: 0:58:42, time: 0.224, data_time: 0.008, memory: 22159, decode.loss_ce: 1.3899, decode.acc_seg: 11.0438, aux.loss_ce: 0.5375, aux.acc_seg: 17.0298, loss: 1.9274 2023-06-21 05:46:37,120 - mmseg - INFO - Exp name: CMID-ResNet50_config.py 2023-06-21 05:46:37,121 - mmseg - INFO - Iter [3000/21600] lr: 9.532e-03, eta: 0:58:41, time: 0.216, data_time: 0.008, memory: 22159, decode.loss_ce: 1.3374, decode.acc_seg: 11.9616, aux.loss_ce: 0.5192, aux.acc_seg: 16.4424, loss: 1.8566 2023-06-21 05:46:49,695 - mmseg - INFO - Iter [3050/21600] lr: 9.516e-03, eta: 0:58:59, time: 0.282, data_time: 0.096, memory: 22159, decode.loss_ce: 1.2816, decode.acc_seg: 12.8364, aux.loss_ce: 0.4978, aux.acc_seg: 17.1060, loss: 1.7794 2023-06-21 05:47:00,643 - mmseg - INFO - Iter [3100/21600] lr: 9.501e-03, eta: 0:58:58, time: 0.219, data_time: 0.007, memory: 22159, decode.loss_ce: 1.3575, decode.acc_seg: 11.3882, aux.loss_ce: 0.5254, aux.acc_seg: 16.3617, loss: 1.8828 2023-06-21 05:47:11,656 - mmseg - INFO - Iter [3150/21600] lr: 9.485e-03, eta: 0:58:57, time: 0.220, data_time: 0.007, memory: 22159, decode.loss_ce: 1.3612, decode.acc_seg: 11.1422, aux.loss_ce: 0.5270, aux.acc_seg: 15.5273, loss: 1.8882 2023-06-21 05:47:20,933 - mmseg - INFO - Iter [3200/21600] lr: 9.469e-03, eta: 0:58:46, time: 0.186, data_time: 0.008, memory: 22159, decode.loss_ce: 1.3541, decode.acc_seg: 11.2511, aux.loss_ce: 0.5226, aux.acc_seg: 16.7572, loss: 1.8767 2023-06-21 05:47:32,118 - mmseg - INFO - Iter [3250/21600] lr: 9.452e-03, eta: 0:58:45, time: 0.224, data_time: 0.008, memory: 22159, decode.loss_ce: 1.3688, decode.acc_seg: 11.0347, aux.loss_ce: 0.5294, aux.acc_seg: 15.7544, loss: 1.8983 2023-06-21 05:47:43,348 - mmseg - INFO - Iter [3300/21600] lr: 9.435e-03, eta: 0:58:45, time: 0.225, data_time: 0.007, memory: 22159, decode.loss_ce: 1.3409, decode.acc_seg: 11.5303, aux.loss_ce: 0.5171, aux.acc_seg: 17.1613, loss: 1.8581 2023-06-21 05:47:54,340 - mmseg - INFO - Iter [3350/21600] lr: 9.419e-03, eta: 0:58:42, time: 0.220, data_time: 0.006, memory: 22159, decode.loss_ce: 1.2871, decode.acc_seg: 11.4257, aux.loss_ce: 0.4981, aux.acc_seg: 16.7350, loss: 1.7852 2023-06-21 05:48:05,042 - mmseg - INFO - Iter [3400/21600] lr: 9.401e-03, eta: 0:58:38, time: 0.214, data_time: 0.007, memory: 22159, decode.loss_ce: 1.3282, decode.acc_seg: 12.3983, aux.loss_ce: 0.5147, aux.acc_seg: 16.1535, loss: 1.8430 2023-06-21 05:48:16,065 - mmseg - INFO - Iter [3450/21600] lr: 9.384e-03, eta: 0:58:36, time: 0.220, data_time: 0.007, memory: 22159, decode.loss_ce: 1.3044, decode.acc_seg: 11.4673, aux.loss_ce: 0.5053, aux.acc_seg: 16.5432, loss: 1.8096 2023-06-21 05:48:29,126 - mmseg - INFO - Iter [3500/21600] lr: 9.366e-03, eta: 0:58:45, time: 0.266, data_time: 0.059, memory: 22159, decode.loss_ce: 1.3465, decode.acc_seg: 11.4222, aux.loss_ce: 0.5194, aux.acc_seg: 18.0659, loss: 1.8659 2023-06-21 05:48:39,749 - mmseg - INFO - Iter [3550/21600] lr: 9.349e-03, eta: 0:58:40, time: 0.212, data_time: 0.007, memory: 22159, decode.loss_ce: 1.3341, decode.acc_seg: 11.4041, aux.loss_ce: 0.5170, aux.acc_seg: 15.7848, loss: 1.8511 2023-06-21 05:48:49,362 - mmseg - INFO - Iter [3600/21600] lr: 9.331e-03, eta: 0:58:29, time: 0.192, data_time: 0.006, memory: 22159, decode.loss_ce: 1.3522, decode.acc_seg: 11.5652, aux.loss_ce: 0.5252, aux.acc_seg: 15.1545, loss: 1.8774 2023-06-21 05:48:56,460 - mmseg - INFO - Iter [3650/21600] lr: 9.312e-03, eta: 0:58:06, time: 0.142, data_time: 0.008, memory: 22159, decode.loss_ce: 1.3137, decode.acc_seg: 11.1683, aux.loss_ce: 0.5074, aux.acc_seg: 17.3472, loss: 1.8210 2023-06-21 05:49:03,580 - mmseg - INFO - Iter [3700/21600] lr: 9.294e-03, eta: 0:57:44, time: 0.142, data_time: 0.008, memory: 22159, decode.loss_ce: 1.3288, decode.acc_seg: 11.5040, aux.loss_ce: 0.5143, aux.acc_seg: 16.5274, loss: 1.8430 2023-06-21 05:49:10,764 - mmseg - INFO - Iter [3750/21600] lr: 9.275e-03, eta: 0:57:23, time: 0.144, data_time: 0.009, memory: 22159, decode.loss_ce: 1.4042, decode.acc_seg: 11.6313, aux.loss_ce: 0.5426, aux.acc_seg: 16.3851, loss: 1.9468 2023-06-21 05:49:17,814 - mmseg - INFO - Iter [3800/21600] lr: 9.256e-03, eta: 0:57:01, time: 0.141, data_time: 0.008, memory: 22159, decode.loss_ce: 1.3246, decode.acc_seg: 12.2894, aux.loss_ce: 0.5141, aux.acc_seg: 16.7884, loss: 1.8387 2023-06-21 05:49:24,806 - mmseg - INFO - Iter [3850/21600] lr: 9.237e-03, eta: 0:56:39, time: 0.140, data_time: 0.008, memory: 22159, decode.loss_ce: 1.3497, decode.acc_seg: 11.4579, aux.loss_ce: 0.5239, aux.acc_seg: 16.8228, loss: 1.8736 2023-06-21 05:49:34,296 - mmseg - INFO - Iter [3900/21600] lr: 9.217e-03, eta: 0:57:03, time: 0.339, data_time: 0.203, memory: 22159, decode.loss_ce: 1.3530, decode.acc_seg: 9.6071, aux.loss_ce: 0.5221, aux.acc_seg: 14.9843, loss: 1.8751 2023-06-21 05:49:41,137 - mmseg - INFO - Iter [3950/21600] lr: 9.198e-03, eta: 0:56:41, time: 0.137, data_time: 0.008, memory: 22159, decode.loss_ce: 1.3373, decode.acc_seg: 11.0161, aux.loss_ce: 0.5160, aux.acc_seg: 16.9317, loss: 1.8532 2023-06-21 05:49:47,653 - mmseg - INFO - Exp name: CMID-ResNet50_config.py 2023-06-21 05:49:47,653 - mmseg - INFO - Iter [4000/21600] lr: 9.178e-03, eta: 0:56:17, time: 0.130, data_time: 0.007, memory: 22159, decode.loss_ce: 1.3647, decode.acc_seg: 11.8143, aux.loss_ce: 0.5286, aux.acc_seg: 16.4045, loss: 1.8932 2023-06-21 05:49:54,695 - mmseg - INFO - Iter [4050/21600] lr: 9.158e-03, eta: 0:55:57, time: 0.141, data_time: 0.008, memory: 22159, decode.loss_ce: 1.3371, decode.acc_seg: 11.9766, aux.loss_ce: 0.5183, aux.acc_seg: 16.3901, loss: 1.8554 2023-06-21 05:50:02,010 - mmseg - INFO - Iter [4100/21600] lr: 9.138e-03, eta: 0:55:37, time: 0.146, data_time: 0.010, memory: 22159, decode.loss_ce: 1.3515, decode.acc_seg: 12.2343, aux.loss_ce: 0.5228, aux.acc_seg: 16.7147, loss: 1.8743 2023-06-21 05:50:09,454 - mmseg - INFO - Iter [4150/21600] lr: 9.117e-03, eta: 0:55:19, time: 0.149, data_time: 0.010, memory: 22159, decode.loss_ce: 1.3540, decode.acc_seg: 10.7979, aux.loss_ce: 0.5205, aux.acc_seg: 18.0262, loss: 1.8745 2023-06-21 05:50:16,143 - mmseg - INFO - Iter [4200/21600] lr: 9.096e-03, eta: 0:54:58, time: 0.134, data_time: 0.007, memory: 22159, decode.loss_ce: 1.3379, decode.acc_seg: 11.5458, aux.loss_ce: 0.5178, aux.acc_seg: 17.5679, loss: 1.8556 2023-06-21 05:50:23,164 - mmseg - INFO - Iter [4250/21600] lr: 9.075e-03, eta: 0:54:38, time: 0.140, data_time: 0.007, memory: 22159, decode.loss_ce: 1.3575, decode.acc_seg: 11.4251, aux.loss_ce: 0.5259, aux.acc_seg: 16.3577, loss: 1.8834 2023-06-21 05:50:30,154 - mmseg - INFO - Iter [4300/21600] lr: 9.054e-03, eta: 0:54:19, time: 0.140, data_time: 0.008, memory: 22159, decode.loss_ce: 1.2921, decode.acc_seg: 11.3702, aux.loss_ce: 0.4979, aux.acc_seg: 17.4307, loss: 1.7901 2023-06-21 05:50:33,131 - mmseg - INFO - Saving checkpoint at 10 epochs 2023-06-21 05:51:50,918 - mmseg - INFO - per class results: 2023-06-21 05:51:50,920 - mmseg - INFO - +--------------------+------+-------+--------+-----------+--------+ | Class | IoU | Acc | Fscore | Precision | Recall | +--------------------+------+-------+--------+-----------+--------+ | impervious_surface | 4.03 | 4.25 | 7.75 | 43.62 | 4.25 | | building | 13.5 | 20.12 | 23.8 | 29.11 | 20.12 | | low_vegetation | 0.21 | 0.21 | 0.41 | 6.15 | 0.21 | | tree | 19.3 | 45.81 | 32.35 | 25.01 | 45.81 | | car | 3.11 | 20.93 | 6.02 | 3.52 | 20.93 | +--------------------+------+-------+--------+-----------+--------+ 2023-06-21 05:51:50,920 - mmseg - INFO - Summary: 2023-06-21 05:51:50,921 - mmseg - INFO - +-------+------+-------+---------+------------+---------+ | aAcc | mIoU | mAcc | mFscore | mPrecision | mRecall | +-------+------+-------+---------+------------+---------+ | 15.26 | 8.03 | 18.27 | 14.07 | 21.48 | 18.27 | +-------+------+-------+---------+------------+---------+ 2023-06-21 05:51:50,921 - mmseg - INFO - Iter(val) [2016] aAcc: 0.1526, mIoU: 0.0803, mAcc: 0.1827, mFscore: 0.1407, mPrecision: 0.2148, mRecall: 0.1827, IoU.impervious_surface: 0.0403, IoU.building: 0.1350, IoU.low_vegetation: 0.0021, IoU.tree: 0.1930, IoU.car: 0.0311, Acc.impervious_surface: 0.0425, Acc.building: 0.2012, Acc.low_vegetation: 0.0021, Acc.tree: 0.4581, Acc.car: 0.2093, Fscore.impervious_surface: 0.0775, Fscore.building: 0.2380, Fscore.low_vegetation: 0.0041, Fscore.tree: 0.3235, Fscore.car: 0.0602, Precision.impervious_surface: 0.4362, Precision.building: 0.2911, Precision.low_vegetation: 0.0615, Precision.tree: 0.2501, Precision.car: 0.0352, Recall.impervious_surface: 0.0425, Recall.building: 0.2012, Recall.low_vegetation: 0.0021, Recall.tree: 0.4581, Recall.car: 0.2093 2023-06-21 05:51:57,565 - mmseg - INFO - Iter [4350/21600] lr: 9.033e-03, eta: 0:54:16, time: 0.221, data_time: 0.086, memory: 22159, decode.loss_ce: 1.2838, decode.acc_seg: 11.5846, aux.loss_ce: 0.4980, aux.acc_seg: 15.7293, loss: 1.7818 ```

log file (trained with pretrained weights)

``` 2023-06-21 05:40:37,817 - mmseg - INFO - workflow: [('train', 1)], max: 50 epochs 2023-06-21 05:40:37,817 - mmseg - INFO - Checkpoints will be saved to /workspace/CMID/SemanticSegmentation/mmsegmentation/logs/potsdam-pretrained_cmid-resnet-v0.27.0 by HardDiskBackend. 2023-06-21 05:41:27,610 - mmseg - INFO - Iter [50/21600] lr: 5.410e-03, eta: 2:26:05, time: 0.407, data_time: 0.055, memory: 15093, decode.loss_ce: 1.3236, decode.acc_seg: 14.9561, aux.loss_ce: 0.5251, aux.acc_seg: 13.9273, loss: 1.8487 2023-06-21 05:41:38,526 - mmseg - INFO - Iter [100/21600] lr: 9.909e-03, eta: 1:51:59, time: 0.218, data_time: 0.006, memory: 15093, decode.loss_ce: 1.3286, decode.acc_seg: 15.4890, aux.loss_ce: 0.5273, aux.acc_seg: 14.3646, loss: 1.8559 2023-06-21 05:41:49,574 - mmseg - INFO - Iter [150/21600] lr: 9.999e-03, eta: 1:40:49, time: 0.221, data_time: 0.006, memory: 15093, decode.loss_ce: 1.3418, decode.acc_seg: 15.6072, aux.loss_ce: 0.5330, aux.acc_seg: 13.7402, loss: 1.8748 2023-06-21 05:42:00,262 - mmseg - INFO - Iter [200/21600] lr: 9.998e-03, eta: 1:34:29, time: 0.214, data_time: 0.006, memory: 15093, decode.loss_ce: 1.3429, decode.acc_seg: 13.5135, aux.loss_ce: 0.5317, aux.acc_seg: 12.9696, loss: 1.8746 2023-06-21 05:42:09,782 - mmseg - INFO - Iter [250/21600] lr: 9.997e-03, eta: 1:28:58, time: 0.190, data_time: 0.007, memory: 15093, decode.loss_ce: 1.2839, decode.acc_seg: 15.0826, aux.loss_ce: 0.5089, aux.acc_seg: 14.7038, loss: 1.7928 2023-06-21 05:42:18,402 - mmseg - INFO - Iter [300/21600] lr: 9.995e-03, eta: 1:24:10, time: 0.172, data_time: 0.008, memory: 15093, decode.loss_ce: 1.3023, decode.acc_seg: 14.1125, aux.loss_ce: 0.5142, aux.acc_seg: 14.5471, loss: 1.8165 2023-06-21 05:42:27,120 - mmseg - INFO - Iter [350/21600] lr: 9.994e-03, eta: 1:20:47, time: 0.174, data_time: 0.008, memory: 15093, decode.loss_ce: 1.3426, decode.acc_seg: 14.0630, aux.loss_ce: 0.5314, aux.acc_seg: 14.7013, loss: 1.8740 2023-06-21 05:42:35,915 - mmseg - INFO - Iter [400/21600] lr: 9.992e-03, eta: 1:18:18, time: 0.176, data_time: 0.009, memory: 15093, decode.loss_ce: 1.2947, decode.acc_seg: 14.8305, aux.loss_ce: 0.5137, aux.acc_seg: 14.1968, loss: 1.8083 2023-06-21 05:42:46,893 - mmseg - INFO - Iter [450/21600] lr: 9.989e-03, eta: 1:21:29, time: 0.308, data_time: 0.140, memory: 15093, decode.loss_ce: 1.3745, decode.acc_seg: 13.9209, aux.loss_ce: 0.5435, aux.acc_seg: 12.7362, loss: 1.9181 2023-06-21 05:42:55,921 - mmseg - INFO - Iter [500/21600] lr: 9.987e-03, eta: 1:19:31, time: 0.181, data_time: 0.009, memory: 15093, decode.loss_ce: 1.3372, decode.acc_seg: 14.6259, aux.loss_ce: 0.5298, aux.acc_seg: 13.9689, loss: 1.8670 2023-06-21 05:43:04,844 - mmseg - INFO - Iter [550/21600] lr: 9.984e-03, eta: 1:17:48, time: 0.178, data_time: 0.009, memory: 15093, decode.loss_ce: 1.3209, decode.acc_seg: 14.6053, aux.loss_ce: 0.5241, aux.acc_seg: 14.4251, loss: 1.8449 2023-06-21 05:43:13,533 - mmseg - INFO - Iter [600/21600] lr: 9.981e-03, eta: 1:16:13, time: 0.174, data_time: 0.008, memory: 15093, decode.loss_ce: 1.3064, decode.acc_seg: 14.4156, aux.loss_ce: 0.5177, aux.acc_seg: 13.8158, loss: 1.8241 2023-06-21 05:43:22,289 - mmseg - INFO - Iter [650/21600] lr: 9.978e-03, eta: 1:14:53, time: 0.175, data_time: 0.007, memory: 15093, decode.loss_ce: 1.2879, decode.acc_seg: 15.0590, aux.loss_ce: 0.5114, aux.acc_seg: 13.4209, loss: 1.7993 2023-06-21 05:43:30,991 - mmseg - INFO - Iter [700/21600] lr: 9.974e-03, eta: 1:13:42, time: 0.174, data_time: 0.007, memory: 15093, decode.loss_ce: 1.2901, decode.acc_seg: 14.0119, aux.loss_ce: 0.5110, aux.acc_seg: 14.4216, loss: 1.8010 2023-06-21 05:43:40,805 - mmseg - INFO - Iter [750/21600] lr: 9.970e-03, eta: 1:13:10, time: 0.196, data_time: 0.009, memory: 15093, decode.loss_ce: 1.3050, decode.acc_seg: 13.7334, aux.loss_ce: 0.5168, aux.acc_seg: 13.8657, loss: 1.8218 2023-06-21 05:43:51,846 - mmseg - INFO - Iter [800/21600] lr: 9.966e-03, eta: 1:13:13, time: 0.221, data_time: 0.007, memory: 15093, decode.loss_ce: 1.3296, decode.acc_seg: 15.1871, aux.loss_ce: 0.5263, aux.acc_seg: 14.7255, loss: 1.8559 2023-06-21 05:44:03,170 - mmseg - INFO - Iter [850/21600] lr: 9.962e-03, eta: 1:13:21, time: 0.226, data_time: 0.008, memory: 15093, decode.loss_ce: 1.3111, decode.acc_seg: 14.4543, aux.loss_ce: 0.5203, aux.acc_seg: 13.6542, loss: 1.8313 2023-06-21 05:44:16,187 - mmseg - INFO - Iter [900/21600] lr: 9.957e-03, eta: 1:14:23, time: 0.275, data_time: 0.072, memory: 15093, decode.loss_ce: 1.3315, decode.acc_seg: 14.4941, aux.loss_ce: 0.5280, aux.acc_seg: 13.5160, loss: 1.8595 2023-06-21 05:44:27,097 - mmseg - INFO - Iter [950/21600] lr: 9.952e-03, eta: 1:14:15, time: 0.218, data_time: 0.006, memory: 15093, decode.loss_ce: 1.3450, decode.acc_seg: 15.3793, aux.loss_ce: 0.5342, aux.acc_seg: 13.7473, loss: 1.8792 2023-06-21 05:44:37,913 - mmseg - INFO - Exp name: CMID-ResNet50_config.py 2023-06-21 05:44:37,914 - mmseg - INFO - Iter [1000/21600] lr: 9.947e-03, eta: 1:14:05, time: 0.216, data_time: 0.007, memory: 15093, decode.loss_ce: 1.3424, decode.acc_seg: 14.1571, aux.loss_ce: 0.5317, aux.acc_seg: 14.2359, loss: 1.8741 2023-06-21 05:44:49,014 - mmseg - INFO - Iter [1050/21600] lr: 9.942e-03, eta: 1:14:00, time: 0.222, data_time: 0.007, memory: 15093, decode.loss_ce: 1.3190, decode.acc_seg: 14.8608, aux.loss_ce: 0.5225, aux.acc_seg: 14.2192, loss: 1.8415 2023-06-21 05:45:00,210 - mmseg - INFO - Iter [1100/21600] lr: 9.936e-03, eta: 1:13:56, time: 0.224, data_time: 0.008, memory: 15093, decode.loss_ce: 1.3295, decode.acc_seg: 14.4388, aux.loss_ce: 0.5261, aux.acc_seg: 14.0158, loss: 1.8556 2023-06-21 05:45:09,809 - mmseg - INFO - Iter [1150/21600] lr: 9.930e-03, eta: 1:13:24, time: 0.192, data_time: 0.007, memory: 15093, decode.loss_ce: 1.2839, decode.acc_seg: 14.4853, aux.loss_ce: 0.5082, aux.acc_seg: 14.5166, loss: 1.7921 2023-06-21 05:45:19,733 - mmseg - INFO - Iter [1200/21600] lr: 9.924e-03, eta: 1:12:59, time: 0.198, data_time: 0.008, memory: 15093, decode.loss_ce: 1.3692, decode.acc_seg: 14.5961, aux.loss_ce: 0.5424, aux.acc_seg: 13.9979, loss: 1.9116 2023-06-21 05:45:30,923 - mmseg - INFO - Iter [1250/21600] lr: 9.918e-03, eta: 1:12:55, time: 0.224, data_time: 0.008, memory: 15093, decode.loss_ce: 1.3147, decode.acc_seg: 14.3471, aux.loss_ce: 0.5201, aux.acc_seg: 14.2187, loss: 1.8348 2023-06-21 05:45:43,947 - mmseg - INFO - Iter [1300/21600] lr: 9.911e-03, eta: 1:20:21, time: 0.799, data_time: 0.596, memory: 15093, decode.loss_ce: 1.2780, decode.acc_seg: 15.2484, aux.loss_ce: 0.5084, aux.acc_seg: 10.8536, loss: 1.7864 2023-06-21 05:45:54,740 - mmseg - INFO - Iter [1350/21600] lr: 9.904e-03, eta: 1:19:53, time: 0.216, data_time: 0.007, memory: 15093, decode.loss_ce: 1.3522, decode.acc_seg: 14.4458, aux.loss_ce: 0.5350, aux.acc_seg: 13.6134, loss: 1.8873 2023-06-21 05:46:05,752 - mmseg - INFO - Iter [1400/21600] lr: 9.897e-03, eta: 1:19:29, time: 0.220, data_time: 0.007, memory: 15093, decode.loss_ce: 1.2761, decode.acc_seg: 14.4216, aux.loss_ce: 0.5058, aux.acc_seg: 13.7552, loss: 1.7818 2023-06-21 05:46:16,749 - mmseg - INFO - Iter [1450/21600] lr: 9.889e-03, eta: 1:19:06, time: 0.220, data_time: 0.008, memory: 15093, decode.loss_ce: 1.2918, decode.acc_seg: 13.6476, aux.loss_ce: 0.5113, aux.acc_seg: 13.4040, loss: 1.8031 2023-06-21 05:46:27,765 - mmseg - INFO - Iter [1500/21600] lr: 9.882e-03, eta: 1:18:44, time: 0.220, data_time: 0.007, memory: 15093, decode.loss_ce: 1.3083, decode.acc_seg: 15.1450, aux.loss_ce: 0.5185, aux.acc_seg: 14.5750, loss: 1.8269 2023-06-21 05:46:38,544 - mmseg - INFO - Iter [1550/21600] lr: 9.874e-03, eta: 1:18:20, time: 0.216, data_time: 0.006, memory: 15093, decode.loss_ce: 1.3073, decode.acc_seg: 14.5244, aux.loss_ce: 0.5183, aux.acc_seg: 14.0311, loss: 1.8256 2023-06-21 05:46:47,762 - mmseg - INFO - Iter [1600/21600] lr: 9.865e-03, eta: 1:17:37, time: 0.184, data_time: 0.007, memory: 15093, decode.loss_ce: 1.3479, decode.acc_seg: 14.9974, aux.loss_ce: 0.5346, aux.acc_seg: 15.1374, loss: 1.8825 2023-06-21 05:46:58,541 - mmseg - INFO - Iter [1650/21600] lr: 9.857e-03, eta: 1:17:14, time: 0.216, data_time: 0.006, memory: 15093, decode.loss_ce: 1.3514, decode.acc_seg: 14.4869, aux.loss_ce: 0.5348, aux.acc_seg: 14.2562, loss: 1.8862 2023-06-21 05:47:09,445 - mmseg - INFO - Iter [1700/21600] lr: 9.848e-03, eta: 1:16:55, time: 0.218, data_time: 0.006, memory: 15093, decode.loss_ce: 1.3272, decode.acc_seg: 13.7781, aux.loss_ce: 0.5252, aux.acc_seg: 13.4528, loss: 1.8524 2023-06-21 05:47:22,311 - mmseg - INFO - Iter [1750/21600] lr: 9.839e-03, eta: 1:17:28, time: 0.310, data_time: 0.115, memory: 15093, decode.loss_ce: 1.3088, decode.acc_seg: 15.2751, aux.loss_ce: 0.5192, aux.acc_seg: 13.1423, loss: 1.8280 2023-06-21 05:47:33,372 - mmseg - INFO - Iter [1800/21600] lr: 9.830e-03, eta: 1:17:09, time: 0.221, data_time: 0.007, memory: 15093, decode.loss_ce: 1.2991, decode.acc_seg: 14.7415, aux.loss_ce: 0.5153, aux.acc_seg: 14.0881, loss: 1.8144 2023-06-21 05:47:44,714 - mmseg - INFO - Iter [1850/21600] lr: 9.820e-03, eta: 1:16:53, time: 0.227, data_time: 0.007, memory: 15093, decode.loss_ce: 1.3094, decode.acc_seg: 13.0988, aux.loss_ce: 0.5171, aux.acc_seg: 13.6851, loss: 1.8265 2023-06-21 05:47:55,674 - mmseg - INFO - Iter [1900/21600] lr: 9.811e-03, eta: 1:16:34, time: 0.219, data_time: 0.007, memory: 15093, decode.loss_ce: 1.3433, decode.acc_seg: 14.9849, aux.loss_ce: 0.5314, aux.acc_seg: 15.1562, loss: 1.8747 2023-06-21 05:48:06,098 - mmseg - INFO - Iter [1950/21600] lr: 9.800e-03, eta: 1:16:10, time: 0.208, data_time: 0.006, memory: 15093, decode.loss_ce: 1.3402, decode.acc_seg: 14.8466, aux.loss_ce: 0.5313, aux.acc_seg: 13.5646, loss: 1.8716 2023-06-21 05:48:17,171 - mmseg - INFO - Exp name: CMID-ResNet50_config.py 2023-06-21 05:48:17,172 - mmseg - INFO - Iter [2000/21600] lr: 9.790e-03, eta: 1:15:53, time: 0.222, data_time: 0.006, memory: 15093, decode.loss_ce: 1.2927, decode.acc_seg: 14.1114, aux.loss_ce: 0.5121, aux.acc_seg: 13.4364, loss: 1.8048 2023-06-21 05:48:26,431 - mmseg - INFO - Iter [2050/21600] lr: 9.780e-03, eta: 1:15:19, time: 0.185, data_time: 0.006, memory: 15093, decode.loss_ce: 1.3002, decode.acc_seg: 14.5986, aux.loss_ce: 0.5146, aux.acc_seg: 14.7706, loss: 1.8148 2023-06-21 05:48:37,002 - mmseg - INFO - Iter [2100/21600] lr: 9.769e-03, eta: 1:14:58, time: 0.211, data_time: 0.006, memory: 15093, decode.loss_ce: 1.3220, decode.acc_seg: 14.6840, aux.loss_ce: 0.5243, aux.acc_seg: 13.2052, loss: 1.8462 2023-06-21 05:48:47,140 - mmseg - INFO - Iter [2150/21600] lr: 9.758e-03, eta: 1:14:34, time: 0.203, data_time: 0.007, memory: 15093, decode.loss_ce: 1.3362, decode.acc_seg: 14.5819, aux.loss_ce: 0.5306, aux.acc_seg: 13.4575, loss: 1.8668 ```

I believe the pretrained weights are loaded properly, because I get this message

backbone.layer2.2.bn2.weight - torch.Size([128]): 
PretrainedInit: load from /workspace/CMID/Pretrain/logs/pretrained_weights/resnet50_backbone.pth 

backbone.layer2.2.bn2.bias - torch.Size([128]): 
PretrainedInit: load from /workspace/CMID/Pretrain/logs/pretrained_weights/resnet50_backbone.pth 

backbone.layer2.2.conv3.weight - torch.Size([512, 128, 1, 1]): 
PretrainedInit: load from /workspace/CMID/Pretrain/logs/pretrained_weights/resnet50_backbone.pth 

backbone.layer2.2.bn3.weight - torch.Size([512]): 
PretrainedInit: load from /workspace/CMID/Pretrain/logs/pretrained_weights/resnet50_backbone.pth 

backbone.layer2.2.bn3.bias - torch.Size([512]): 
PretrainedInit: load from /workspace/CMID/Pretrain/logs/pretrained_weights/resnet50_backbone.pth 

...

for all backbone layers when I set checkpoint_path={path of pretrained backbone}, while I get this

backbone.layer2.2.bn2.weight - torch.Size([128]): 
The value is the same before and after calling `init_weights` of EncoderDecoder  

backbone.layer2.2.bn2.bias - torch.Size([128]): 
The value is the same before and after calling `init_weights` of EncoderDecoder  

backbone.layer2.2.conv3.weight - torch.Size([512, 128, 1, 1]): 
KaimingInit: a=0, mode=fan_out, nonlinearity=relu, distribution =normal, bias=0 

backbone.layer2.2.bn3.weight - torch.Size([512]): 
ConstantInit: val=0, bias=0 

backbone.layer2.2.bn3.bias - torch.Size([512]): 
The value is the same before and after calling `init_weights` of EncoderDecoder  

...

when I train from scratch(checkpoint_path=None).

Please let me know if there seems to be something wrong in my config. Thank you!

[pUmpKin-Co]

Hi~.

It seems that this error is from computation precision. I used the mixed precision during my whole experiment (e.g FP16OptimierHook). However, I mistakenly forgot to change the corresponding code.

I used your same configuration and meet the same problem. All of these process fail to converge whatever training from scratch, from CMID-Backbone, or from ImageNet backbone.

Finally, I have a look at my original configuration, and find the devil:

Change the line: optimizer_config = None to optimizer_config = dict(type='Fp16OptimizerHook', distributed=False)

Whole Config

```python log_config = dict( interval=50, hooks=[ dict(type='TextLoggerHook', by_epoch=False), dict(type='TensorboardLoggerHook') ]) dist_params = dict(backend='nccl') log_level = 'INFO' load_from = None resume_from = None workflow = [('train', 1)] cudnn_benchmark = True optimizer = dict(type='SGD', lr=0.01, momentum=0.9, weight_decay=0.0001) optimizer_config = dict(type='Fp16OptimizerHook', distributed=False) lr_config = dict( policy='CosineAnnealing', warmup='linear', warmup_iters=100, min_lr=1e-06, by_epoch=False) runner = dict(type='EpochBasedRunner', max_epochs=50) checkpoint_config = dict(by_epoch=True, interval=1) evaluation = dict( interval=5, metric=['mIoU', 'mFscore'], pre_eval=True, by_epoch=True) norm_cfg = dict(type='BN', requires_grad=True) checkpoint_path = 'F:\pumpkinCode\mmseg\pretrain_checkpoint\CMID_ResNet50_bk_200ep' model = dict( type='EncoderDecoder', pretrained= 'F:\pumpkinCode\mmseg\pretrain_checkpoint\CMID_ResNet50_bk_200ep', backbone=dict( type='ResNet', depth=50, num_stages=4, out_indices=(0, 1, 2, 3), dilations=(1, 1, 1, 1), strides=(1, 2, 2, 2), norm_cfg=dict(type='BN', requires_grad=True), norm_eval=False, style='pytorch', contract_dilation=True), decode_head=dict( type='UPerHead', in_channels=[256, 512, 1024, 2048], in_index=[0, 1, 2, 3], pool_scales=(1, 2, 3, 6), channels=512, ignore_index=255, dropout_ratio=0.1, num_classes=6, norm_cfg=dict(type='BN', requires_grad=True), align_corners=False, loss_decode=dict( type='CrossEntropyLoss', use_sigmoid=False, loss_weight=1.0)), auxiliary_head=dict( type='FCNHead', in_channels=1024, in_index=2, channels=256, ignore_index=255, num_convs=1, concat_input=False, dropout_ratio=0.1, num_classes=6, norm_cfg=dict(type='BN', requires_grad=True), align_corners=False, loss_decode=dict( type='CrossEntropyLoss', use_sigmoid=False, loss_weight=0.4)), train_cfg=dict(), test_cfg=dict(mode='whole')) dataset_type = 'PotsdamDataset' data_root = 'F:\VisionData\Potsdam\mmseg' img_norm_cfg = dict( mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_rgb=True) crop_size = (512, 512) train_pipeline = [ dict(type='LoadImageFromFile'), dict( type='LoadAnnotationsReduceIgnoreIndex', reduce_zero_label=True, ignore_index=6), dict(type='Resize', img_scale=(512, 512), ratio_range=(0.5, 2.0)), dict(type='RandomCrop', crop_size=(512, 512), cat_max_ratio=0.75), dict(type='RandomFlip', prob=0.5), dict(type='PhotoMetricDistortion'), dict( type='Normalize', mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_rgb=True), dict(type='Pad', size=(512, 512), pad_val=0, seg_pad_val=255), dict(type='DefaultFormatBundle'), dict(type='Collect', keys=['img', 'gt_semantic_seg']) ] test_pipeline = [ dict(type='LoadImageFromFile'), dict( type='MultiScaleFlipAug', img_scale=(512, 512), flip=False, transforms=[ dict(type='Resize', keep_ratio=True), dict(type='RandomFlip'), dict( type='Normalize', mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_rgb=True), dict(type='ImageToTensor', keys=['img']), dict(type='Collect', keys=['img']) ]) ] data = dict( samples_per_gpu=4, workers_per_gpu=4, train=dict( type='PotsdamDataset', data_root='F:\VisionData\Potsdam\mmseg', img_dir='img_dir/train', ann_dir='ann_dir/train', pipeline=[ dict(type='LoadImageFromFile'), dict( type='LoadAnnotationsReduceIgnoreIndex', reduce_zero_label=True, ignore_index=6), dict(type='Resize', img_scale=(512, 512), ratio_range=(0.5, 2.0)), dict(type='RandomCrop', crop_size=(512, 512), cat_max_ratio=0.75), dict(type='RandomFlip', prob=0.5), dict(type='PhotoMetricDistortion'), dict( type='Normalize', mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_rgb=True), dict(type='Pad', size=(512, 512), pad_val=0, seg_pad_val=255), dict(type='DefaultFormatBundle'), dict(type='Collect', keys=['img', 'gt_semantic_seg']) ]), val=dict( type='PotsdamDataset', data_root='F:\VisionData\Potsdam\mmseg', img_dir='img_dir/val', ann_dir='ann_dir/val', pipeline=[ dict(type='LoadImageFromFile'), dict( type='MultiScaleFlipAug', img_scale=(512, 512), flip=False, transforms=[ dict(type='Resize', keep_ratio=True), dict(type='RandomFlip'), dict( type='Normalize', mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_rgb=True), dict(type='ImageToTensor', keys=['img']), dict(type='Collect', keys=['img']) ]) ]), test=dict( type='PotsdamDataset', data_root='F:\VisionData\Potsdam\mmseg', img_dir='img_dir/val', ann_dir='ann_dir/val', pipeline=[ dict(type='LoadImageFromFile'), dict( type='MultiScaleFlipAug', img_scale=(512, 512), flip=False, transforms=[ dict(type='Resize', keep_ratio=True), dict(type='RandomFlip'), dict( type='Normalize', mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_rgb=True), dict(type='ImageToTensor', keys=['img']), dict(type='Collect', keys=['img']) ]) ])) work_dir = 'test_demo2' gpu_ids = [0] auto_resume = False ```

Training Log

```python 2023-06-21 15:25:59,041 - mmseg - INFO - Iter [50/43200] lr: 5.410e-03, eta: 7:20:02, time: 0.612, data_time: 0.276, memory: 6714, decode.loss_ce: 0.9882, decode.acc_seg: 38.1258, aux.loss_ce: 0.4508, aux.acc_seg: 30.2004, loss: 1.4390 2023-06-21 15:26:08,144 - mmseg - INFO - Iter [100/43200] lr: 9.910e-03, eta: 4:45:09, time: 0.182, data_time: 0.002, memory: 6714, decode.loss_ce: 0.7975, decode.acc_seg: 53.3647, aux.loss_ce: 0.3711, aux.acc_seg: 44.6582, loss: 1.1685 2023-06-21 15:26:17,265 - mmseg - INFO - Iter [150/43200] lr: 1.000e-02, eta: 3:53:30, time: 0.182, data_time: 0.002, memory: 6714, decode.loss_ce: 0.8577, decode.acc_seg: 54.0806, aux.loss_ce: 0.3297, aux.acc_seg: 53.6072, loss: 1.1874 2023-06-21 15:26:26,422 - mmseg - INFO - Iter [200/43200] lr: 9.999e-03, eta: 3:27:44, time: 0.183, data_time: 0.002, memory: 6714, decode.loss_ce: 0.8101, decode.acc_seg: 55.3112, aux.loss_ce: 0.3256, aux.acc_seg: 53.8390, loss: 1.1357 2023-06-21 15:26:35,577 - mmseg - INFO - Iter [250/43200] lr: 9.999e-03, eta: 3:12:12, time: 0.183, data_time: 0.002, memory: 6714, decode.loss_ce: 0.7792, decode.acc_seg: 58.9318, aux.loss_ce: 0.2886, aux.acc_seg: 59.8018, loss: 1.0679 2023-06-21 15:26:44,745 - mmseg - INFO - Iter [300/43200] lr: 9.999e-03, eta: 3:01:50, time: 0.183, data_time: 0.002, memory: 6714, decode.loss_ce: 0.8659, decode.acc_seg: 55.2330, aux.loss_ce: 0.2944, aux.acc_seg: 59.6692, loss: 1.1604 2023-06-21 15:26:53,933 - mmseg - INFO - Iter [350/43200] lr: 9.998e-03, eta: 2:54:25, time: 0.184, data_time: 0.002, memory: 6714, decode.loss_ce: 0.6874, decode.acc_seg: 63.2813, aux.loss_ce: 0.2708, aux.acc_seg: 62.7625, loss: 0.9582 2023-06-21 15:27:03,127 - mmseg - INFO - Iter [400/43200] lr: 9.998e-03, eta: 2:48:50, time: 0.184, data_time: 0.002, memory: 6714, decode.loss_ce: 0.7390, decode.acc_seg: 58.8482, aux.loss_ce: 0.2670, aux.acc_seg: 59.9143, loss: 1.0060 2023-06-21 15:27:12,324 - mmseg - INFO - Iter [450/43200] lr: 9.997e-03, eta: 2:44:28, time: 0.184, data_time: 0.002, memory: 6714, decode.loss_ce: 0.7364, decode.acc_seg: 60.0075, aux.loss_ce: 0.2764, aux.acc_seg: 61.6727, loss: 1.0128 .... ```

You can have a try to see if this will be helpful. It's really strange that this error occurs, maybe it's due to a problem in the optimization process. I look forward to your feedback.

[letme-hj]

Thank you so much!!

I tried and it seems to be working now! Thank you again for your kind response :) 👍

[pUmpKin-Co]

Thanks for your feedback. I will update the corresponding code.

[pUmpKin-Co]

Closed as long periods of inactivity, feel free to reopen if there is any problem.