search

MengyuWang826 / SegRefiner

SegRefiner: Towards Model-Agnostic Segmentation Refinement with Discrete Diffusion Process
Apache License 2.0
137 stars 8 forks source link

Training process abnormal. #4

Open lixiang007666 opened 6 months ago

lixiang007666 commented 6 months ago

I have tried many different settings, but during the training of segrefiner, the IoU becomes zero after only a few steps. Is this a known issue?

Thanks.

image

MengyuWang826 commented 6 months ago

I have tried many different settings, but during the training of segrefiner, the IoU becomes zero after only a few steps. Is this a known issue?

Thanks.

image

I haven't encountered this issue before. Please paste your complete settings and startup command here, and I'll find some time in the next few days to test it.

yusuke-ai commented 5 months ago

I got an similar issue and the training configuration and the log is below.

Training Configuration

checkpoint_config = dict(
    interval=5000, by_epoch=False, save_last=True, max_keep_ckpts=20)
log_config = dict(
    interval=50, hooks=[dict(type='TextLoggerHook', by_epoch=False)])
dist_params = dict(backend='nccl')
log_level = 'INFO'
load_from = None
resume_from = 'segrefiner_hr_latest.pth'
workflow = [('train', 5000)]
opencv_num_threads = 0
mp_start_method = 'fork'
auto_scale_lr = dict(enable=False, base_batch_size=16)
object_size = 256
task = 'instance'
model = dict(
    type='SegRefiner',
    task='instance',
    step=6,
    denoise_model=dict(
        type='DenoiseUNet',
        in_channels=4,
        out_channels=1,
        model_channels=128,
        num_res_blocks=2,
        num_heads=4,
        num_heads_upsample=-1,
        attention_strides=(16, 32),
        learn_time_embd=True,
        channel_mult=(1, 1, 2, 2, 4, 4),
        dropout=0.0),
    diffusion_cfg=dict(
        betas=dict(type='linear', start=0.8, stop=0, num_timesteps=6),
        diff_iter=False),
    test_cfg=dict())
img_norm_cfg = dict(
    mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_rgb=True)
train_pipeline = [
    dict(type='LoadImageFromFile'),
    dict(
        type='LoadAnnotations',
        with_bbox=False,
        with_label=False,
        with_mask=True),
    dict(type='LoadPatchData', object_size=256, patch_size=256),
    dict(type='Resize', img_scale=(256, 256), keep_ratio=False),
    dict(type='RandomFlip', flip_ratio=0.5),
    dict(
        type='Normalize',
        mean=[123.675, 116.28, 103.53],
        std=[58.395, 57.12, 57.375],
        to_rgb=True),
    dict(type='DefaultFormatBundle'),
    dict(
        type='Collect',
        keys=[
            'object_img', 'object_gt_masks', 'object_coarse_masks',
            'patch_img', 'patch_gt_masks', 'patch_coarse_masks'
        ])
]
dataset_type = 'HRCollectionDataset'
img_root = '/share/project/datasets/MSCOCO/coco2017/'
ann_root = '/share/project/datasets/LVIS/'
train_dataloader = dict(samples_per_gpu=1, workers_per_gpu=1)
data = dict(
    train=dict(
        type='HRCollectionDataset',
        pipeline=[
            dict(type='LoadImageFromFile'),
            dict(
                type='LoadAnnotations',
                with_bbox=False,
                with_label=False,
                with_mask=True),
            dict(type='LoadPatchData', object_size=256, patch_size=256),
            dict(type='Resize', img_scale=(256, 256), keep_ratio=False),
            dict(type='RandomFlip', flip_ratio=0.5),
            dict(
                type='Normalize',
                mean=[123.675, 116.28, 103.53],
                std=[58.395, 57.12, 57.375],
                to_rgb=True),
            dict(type='DefaultFormatBundle'),
            dict(
                type='Collect',
                keys=[
                    'object_img', 'object_gt_masks', 'object_coarse_masks',
                    'patch_img', 'patch_gt_masks', 'patch_coarse_masks'
                ])
        ],
        data_root='data/',
        collection_datasets=['thin', 'dis'],
        collection_json='data/collection_hr.json'),
    train_dataloader=dict(samples_per_gpu=1, workers_per_gpu=1),
    val=dict(),
    test=dict())
optimizer = dict(
    type='AdamW', lr=0.0004, weight_decay=0, eps=1e-08, betas=(0.9, 0.999))
optimizer_config = dict(grad_clip=None)
max_iters = 120000
runner = dict(type='IterBasedRunner', max_iters=120000)
lr_config = dict(
    policy='step',
    gamma=0.5,
    by_epoch=False,
    step=[80000, 100000],
    warmup='linear',
    warmup_by_epoch=False,
    warmup_ratio=1.0,
    warmup_iters=10)
interval = 5000
data_root = 'data/'
work_dir = './work_dirs/segrefiner_hr'
auto_resume = False
gpu_ids = [0]

The Log

2024-01-19 12:24:03,655 - mmdet - INFO - workflow: [('train', 5000)], max: 120000 iters
2024-01-19 12:24:03,655 - mmdet - INFO - Checkpoints will be saved to /home/yusuke/gitrepos/SegRefiner/work_dirs/segrefiner_hr by HardDiskBackend.
2024-01-19 12:24:15,210 - mmdet - INFO - Iter [50/120000]   lr: 4.000e-04, eta: 7:41:07, time: 0.231, data_time: 0.005, memory: 6267, loss_mask: 0.0749, loss_texture: 0.0805, iou: 0.9283, loss: 0.1554
2024-01-19 12:24:25,180 - mmdet - INFO - Iter [100/120000]  lr: 4.000e-04, eta: 7:09:42, time: 0.199, data_time: 0.003, memory: 6267, loss_mask: 0.0736, loss_texture: 0.0857, iou: 0.9084, loss: 0.1592
2024-01-19 12:24:35,107 - mmdet - INFO - Iter [150/120000]  lr: 4.000e-04, eta: 6:58:32, time: 0.199, data_time: 0.003, memory: 6267, loss_mask: 0.4370, loss_texture: 0.2199, iou: 0.5518, loss: 0.6569
2024-01-19 12:24:45,140 - mmdet - INFO - Iter [200/120000]  lr: 4.000e-04, eta: 6:53:55, time: 0.201, data_time: 0.003, memory: 6267, loss_mask: 0.2852, loss_texture: 0.2145, iou: 0.5794, loss: 0.4996
2024-01-19 12:24:55,056 - mmdet - INFO - Iter [250/120000]  lr: 4.000e-04, eta: 6:50:09, time: 0.198, data_time: 0.003, memory: 6267, loss_mask: 0.2346, loss_texture: 0.2139, iou: 0.6228, loss: 0.4484
2024-01-19 12:25:05,039 - mmdet - INFO - Iter [300/120000]  lr: 4.000e-04, eta: 6:48:02, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.2050, loss_texture: 0.1940, iou: 0.6370, loss: 0.3990
2024-01-19 12:25:14,964 - mmdet - INFO - Iter [350/120000]  lr: 4.000e-04, eta: 6:46:09, time: 0.198, data_time: 0.003, memory: 6267, loss_mask: 0.2655, loss_texture: 0.2266, iou: 0.6330, loss: 0.4921
2024-01-19 12:25:24,949 - mmdet - INFO - Iter [400/120000]  lr: 4.000e-04, eta: 6:44:59, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.2196, loss_texture: 0.2139, iou: 0.7239, loss: 0.4336
2024-01-19 12:25:34,935 - mmdet - INFO - Iter [450/120000]  lr: 4.000e-04, eta: 6:44:03, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1660, loss_texture: 0.1727, iou: 0.7408, loss: 0.3387
2024-01-19 12:25:44,947 - mmdet - INFO - Iter [500/120000]  lr: 4.000e-04, eta: 6:43:22, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.2036, loss_texture: 0.2130, iou: 0.7069, loss: 0.4167
2024-01-19 12:25:54,943 - mmdet - INFO - Iter [550/120000]  lr: 4.000e-04, eta: 6:42:44, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1658, loss_texture: 0.1909, iou: 0.7922, loss: 0.3566
2024-01-19 12:26:04,933 - mmdet - INFO - Iter [600/120000]  lr: 4.000e-04, eta: 6:42:09, time: 0.200, data_time: 0.004, memory: 6267, loss_mask: 0.1983, loss_texture: 0.2182, iou: 0.7171, loss: 0.4165
2024-01-19 12:26:14,901 - mmdet - INFO - Iter [650/120000]  lr: 4.000e-04, eta: 6:41:33, time: 0.199, data_time: 0.003, memory: 6267, loss_mask: 0.1579, loss_texture: 0.1868, iou: 0.7118, loss: 0.3447
2024-01-19 12:26:24,843 - mmdet - INFO - Iter [700/120000]  lr: 4.000e-04, eta: 6:40:57, time: 0.199, data_time: 0.003, memory: 6267, loss_mask: 0.2049, loss_texture: 0.2065, iou: 0.7138, loss: 0.4114
2024-01-19 12:26:34,789 - mmdet - INFO - Iter [750/120000]  lr: 4.000e-04, eta: 6:40:25, time: 0.199, data_time: 0.003, memory: 6267, loss_mask: 0.2391, loss_texture: 0.2162, iou: 0.6966, loss: 0.4553
2024-01-19 12:26:44,700 - mmdet - INFO - Iter [800/120000]  lr: 4.000e-04, eta: 6:39:51, time: 0.198, data_time: 0.003, memory: 6267, loss_mask: 0.1903, loss_texture: 0.2007, iou: 0.6805, loss: 0.3910
2024-01-19 12:26:54,823 - mmdet - INFO - Iter [850/120000]  lr: 4.000e-04, eta: 6:39:49, time: 0.202, data_time: 0.006, memory: 6267, loss_mask: 0.2275, loss_texture: 0.2216, iou: 0.6903, loss: 0.4491
2024-01-19 12:27:04,774 - mmdet - INFO - Iter [900/120000]  lr: 4.000e-04, eta: 6:39:24, time: 0.199, data_time: 0.003, memory: 6267, loss_mask: 0.2924, loss_texture: 0.2216, iou: 0.6570, loss: 0.5139
2024-01-19 12:27:14,760 - mmdet - INFO - Iter [950/120000]  lr: 4.000e-04, eta: 6:39:04, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.2299, loss_texture: 0.2123, iou: 0.6476, loss: 0.4421
2024-01-19 12:27:24,758 - mmdet - INFO - Exp name: segrefiner_hr.py
2024-01-19 12:27:24,758 - mmdet - INFO - Iter [1000/120000] lr: 4.000e-04, eta: 6:38:47, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.2042, loss_texture: 0.1917, iou: 0.6918, loss: 0.3960
2024-01-19 12:27:34,786 - mmdet - INFO - Iter [1050/120000] lr: 4.000e-04, eta: 6:38:34, time: 0.201, data_time: 0.004, memory: 6267, loss_mask: 0.2077, loss_texture: 0.1856, iou: 0.7152, loss: 0.3933
2024-01-19 12:27:44,821 - mmdet - INFO - Iter [1100/120000] lr: 4.000e-04, eta: 6:38:22, time: 0.201, data_time: 0.003, memory: 6267, loss_mask: 0.1809, loss_texture: 0.1987, iou: 0.7236, loss: 0.3796
2024-01-19 12:27:54,808 - mmdet - INFO - Iter [1150/120000] lr: 4.000e-04, eta: 6:38:06, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.2191, loss_texture: 0.2069, iou: 0.7322, loss: 0.4260
2024-01-19 12:28:04,823 - mmdet - INFO - Iter [1200/120000] lr: 4.000e-04, eta: 6:37:52, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.2450, loss_texture: 0.2021, iou: 0.7341, loss: 0.4471
2024-01-19 12:28:14,884 - mmdet - INFO - Iter [1250/120000] lr: 4.000e-04, eta: 6:37:43, time: 0.201, data_time: 0.003, memory: 6267, loss_mask: 0.1971, loss_texture: 0.2114, iou: 0.7124, loss: 0.4085
2024-01-19 12:28:24,896 - mmdet - INFO - Iter [1300/120000] lr: 4.000e-04, eta: 6:37:30, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1347, loss_texture: 0.1755, iou: 0.8245, loss: 0.3102
2024-01-19 12:28:35,061 - mmdet - INFO - Iter [1350/120000] lr: 4.000e-04, eta: 6:37:30, time: 0.203, data_time: 0.006, memory: 6267, loss_mask: 0.2002, loss_texture: 0.1978, iou: 0.7008, loss: 0.3980
2024-01-19 12:28:45,045 - mmdet - INFO - Iter [1400/120000] lr: 4.000e-04, eta: 6:37:15, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1808, loss_texture: 0.1898, iou: 0.7335, loss: 0.3705
2024-01-19 12:28:54,994 - mmdet - INFO - Iter [1450/120000] lr: 4.000e-04, eta: 6:36:56, time: 0.199, data_time: 0.003, memory: 6267, loss_mask: 0.1796, loss_texture: 0.1766, iou: 0.7720, loss: 0.3561
2024-01-19 12:29:04,995 - mmdet - INFO - Iter [1500/120000] lr: 4.000e-04, eta: 6:36:43, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1564, loss_texture: 0.1758, iou: 0.7233, loss: 0.3321
2024-01-19 12:29:14,931 - mmdet - INFO - Iter [1550/120000] lr: 4.000e-04, eta: 6:36:25, time: 0.199, data_time: 0.003, memory: 6267, loss_mask: 0.1863, loss_texture: 0.2102, iou: 0.7394, loss: 0.3965
2024-01-19 12:29:25,060 - mmdet - INFO - Iter [1600/120000] lr: 4.000e-04, eta: 6:36:21, time: 0.203, data_time: 0.005, memory: 6267, loss_mask: 0.2034, loss_texture: 0.2099, iou: 0.7317, loss: 0.4134
2024-01-19 12:29:35,074 - mmdet - INFO - Iter [1650/120000] lr: 4.000e-04, eta: 6:36:09, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1500, loss_texture: 0.1764, iou: 0.7995, loss: 0.3263
2024-01-19 12:29:45,054 - mmdet - INFO - Iter [1700/120000] lr: 4.000e-04, eta: 6:35:55, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1477, loss_texture: 0.1792, iou: 0.7476, loss: 0.3270
2024-01-19 12:29:55,016 - mmdet - INFO - Iter [1750/120000] lr: 4.000e-04, eta: 6:35:39, time: 0.199, data_time: 0.003, memory: 6267, loss_mask: 0.1770, loss_texture: 0.1875, iou: 0.7434, loss: 0.3645
2024-01-19 12:30:04,943 - mmdet - INFO - Iter [1800/120000] lr: 4.000e-04, eta: 6:35:22, time: 0.199, data_time: 0.003, memory: 6267, loss_mask: 0.1464, loss_texture: 0.1856, iou: 0.7543, loss: 0.3320
2024-01-19 12:30:14,854 - mmdet - INFO - Iter [1850/120000] lr: 4.000e-04, eta: 6:35:04, time: 0.198, data_time: 0.003, memory: 6267, loss_mask: 0.1349, loss_texture: 0.1745, iou: 0.7693, loss: 0.3094
2024-01-19 12:30:24,803 - mmdet - INFO - Iter [1900/120000] lr: 4.000e-04, eta: 6:34:49, time: 0.199, data_time: 0.003, memory: 6267, loss_mask: 0.1340, loss_texture: 0.1688, iou: 0.8084, loss: 0.3028
2024-01-19 12:30:34,767 - mmdet - INFO - Iter [1950/120000] lr: 4.000e-04, eta: 6:34:35, time: 0.199, data_time: 0.003, memory: 6267, loss_mask: 0.1249, loss_texture: 0.1673, iou: 0.7941, loss: 0.2922
2024-01-19 12:30:44,694 - mmdet - INFO - Exp name: segrefiner_hr.py
2024-01-19 12:30:44,694 - mmdet - INFO - Iter [2000/120000] lr: 4.000e-04, eta: 6:34:19, time: 0.199, data_time: 0.003, memory: 6267, loss_mask: 0.2065, loss_texture: 0.1790, iou: 0.7811, loss: 0.3856
2024-01-19 12:30:54,604 - mmdet - INFO - Iter [2050/120000] lr: 4.000e-04, eta: 6:34:02, time: 0.198, data_time: 0.003, memory: 6267, loss_mask: 0.1713, loss_texture: 0.2143, iou: 0.7463, loss: 0.3856
2024-01-19 12:31:04,618 - mmdet - INFO - Iter [2100/120000] lr: 4.000e-04, eta: 6:33:52, time: 0.200, data_time: 0.004, memory: 6267, loss_mask: 0.2016, loss_texture: 0.2008, iou: 0.7280, loss: 0.4023
2024-01-19 12:31:14,605 - mmdet - INFO - Iter [2150/120000] lr: 4.000e-04, eta: 6:33:40, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1621, loss_texture: 0.2051, iou: 0.7716, loss: 0.3672
2024-01-19 12:31:24,538 - mmdet - INFO - Iter [2200/120000] lr: 4.000e-04, eta: 6:33:25, time: 0.199, data_time: 0.003, memory: 6267, loss_mask: 0.1611, loss_texture: 0.1994, iou: 0.8068, loss: 0.3604
2024-01-19 12:31:34,484 - mmdet - INFO - Iter [2250/120000] lr: 4.000e-04, eta: 6:33:11, time: 0.199, data_time: 0.003, memory: 6267, loss_mask: 0.1697, loss_texture: 0.2121, iou: 0.6758, loss: 0.3818
2024-01-19 12:31:44,713 - mmdet - INFO - Iter [2300/120000] lr: 4.000e-04, eta: 6:33:12, time: 0.205, data_time: 0.008, memory: 6267, loss_mask: 0.1664, loss_texture: 0.1805, iou: 0.7210, loss: 0.3469
2024-01-19 12:31:54,648 - mmdet - INFO - Iter [2350/120000] lr: 4.000e-04, eta: 6:32:57, time: 0.199, data_time: 0.003, memory: 6267, loss_mask: 0.1260, loss_texture: 0.1762, iou: 0.7685, loss: 0.3021
2024-01-19 12:32:04,631 - mmdet - INFO - Iter [2400/120000] lr: 4.000e-04, eta: 6:32:45, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1444, loss_texture: 0.1815, iou: 0.8099, loss: 0.3260
2024-01-19 12:32:15,601 - mmdet - INFO - Iter [2450/120000] lr: 4.000e-04, eta: 6:33:21, time: 0.219, data_time: 0.021, memory: 6267, loss_mask: 0.1563, loss_texture: 0.1941, iou: 0.7882, loss: 0.3505
2024-01-19 12:32:25,657 - mmdet - INFO - Iter [2500/120000] lr: 4.000e-04, eta: 6:33:12, time: 0.201, data_time: 0.003, memory: 6267, loss_mask: 0.1376, loss_texture: 0.1658, iou: 0.7853, loss: 0.3034
2024-01-19 12:32:35,804 - mmdet - INFO - Iter [2550/120000] lr: 4.000e-04, eta: 6:33:07, time: 0.203, data_time: 0.003, memory: 6267, loss_mask: 0.1314, loss_texture: 0.1676, iou: 0.8216, loss: 0.2991
2024-01-19 12:32:45,851 - mmdet - INFO - Iter [2600/120000] lr: 4.000e-04, eta: 6:32:57, time: 0.201, data_time: 0.003, memory: 6267, loss_mask: 0.1733, loss_texture: 0.1933, iou: 0.7472, loss: 0.3666
2024-01-19 12:32:55,850 - mmdet - INFO - Iter [2650/120000] lr: 4.000e-04, eta: 6:32:45, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1992, loss_texture: 0.2001, iou: 0.6920, loss: 0.3993
2024-01-19 12:33:05,933 - mmdet - INFO - Iter [2700/120000] lr: 4.000e-04, eta: 6:32:37, time: 0.202, data_time: 0.003, memory: 6267, loss_mask: 0.1481, loss_texture: 0.1848, iou: 0.7880, loss: 0.3328
2024-01-19 12:33:15,942 - mmdet - INFO - Iter [2750/120000] lr: 4.000e-04, eta: 6:32:25, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1818, loss_texture: 0.1817, iou: 0.7667, loss: 0.3635
2024-01-19 12:33:25,938 - mmdet - INFO - Iter [2800/120000] lr: 4.000e-04, eta: 6:32:13, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1897, loss_texture: 0.1949, iou: 0.7399, loss: 0.3846
2024-01-19 12:33:36,020 - mmdet - INFO - Iter [2850/120000] lr: 4.000e-04, eta: 6:32:05, time: 0.202, data_time: 0.003, memory: 6267, loss_mask: 0.1986, loss_texture: 0.1722, iou: 0.7496, loss: 0.3707
2024-01-19 12:33:46,018 - mmdet - INFO - Iter [2900/120000] lr: 4.000e-04, eta: 6:31:53, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1509, loss_texture: 0.1899, iou: 0.7735, loss: 0.3408
2024-01-19 12:33:56,028 - mmdet - INFO - Iter [2950/120000] lr: 4.000e-04, eta: 6:31:42, time: 0.200, data_time: 0.004, memory: 6267, loss_mask: 0.1364, loss_texture: 0.1691, iou: 0.7962, loss: 0.3055
2024-01-19 12:34:06,055 - mmdet - INFO - Exp name: segrefiner_hr.py
2024-01-19 12:34:06,055 - mmdet - INFO - Iter [3000/120000] lr: 4.000e-04, eta: 6:31:31, time: 0.201, data_time: 0.003, memory: 6267, loss_mask: 0.1484, loss_texture: 0.1878, iou: 0.7773, loss: 0.3362
2024-01-19 12:34:16,073 - mmdet - INFO - Iter [3050/120000] lr: 4.000e-04, eta: 6:31:21, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1766, loss_texture: 0.2010, iou: 0.7864, loss: 0.3776
2024-01-19 12:34:26,081 - mmdet - INFO - Iter [3100/120000] lr: 4.000e-04, eta: 6:31:09, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1475, loss_texture: 0.1973, iou: 0.7756, loss: 0.3448
2024-01-19 12:34:36,077 - mmdet - INFO - Iter [3150/120000] lr: 4.000e-04, eta: 6:30:58, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1222, loss_texture: 0.1656, iou: 0.8325, loss: 0.2878
2024-01-19 12:34:46,095 - mmdet - INFO - Iter [3200/120000] lr: 4.000e-04, eta: 6:30:47, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1147, loss_texture: 0.1624, iou: 0.7541, loss: 0.2771
2024-01-19 12:34:56,057 - mmdet - INFO - Iter [3250/120000] lr: 4.000e-04, eta: 6:30:34, time: 0.199, data_time: 0.003, memory: 6267, loss_mask: 0.1926, loss_texture: 0.1761, iou: 0.7865, loss: 0.3687
2024-01-19 12:35:06,074 - mmdet - INFO - Iter [3300/120000] lr: 4.000e-04, eta: 6:30:23, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1360, loss_texture: 0.1826, iou: 0.7763, loss: 0.3186
2024-01-19 12:35:16,077 - mmdet - INFO - Iter [3350/120000] lr: 4.000e-04, eta: 6:30:12, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.3127, loss_texture: 0.1954, iou: 0.6610, loss: 0.5081
2024-01-19 12:35:26,089 - mmdet - INFO - Iter [3400/120000] lr: 4.000e-04, eta: 6:30:01, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.2593, loss_texture: 0.1952, iou: 0.6893, loss: 0.4545
2024-01-19 12:35:36,096 - mmdet - INFO - Iter [3450/120000] lr: 4.000e-04, eta: 6:29:50, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1894, loss_texture: 0.2158, iou: 0.7296, loss: 0.4052
2024-01-19 12:35:46,117 - mmdet - INFO - Iter [3500/120000] lr: 4.000e-04, eta: 6:29:40, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.2522, loss_texture: 0.2290, iou: 0.6647, loss: 0.4813
2024-01-19 12:35:56,152 - mmdet - INFO - Iter [3550/120000] lr: 4.000e-04, eta: 6:29:30, time: 0.201, data_time: 0.003, memory: 6267, loss_mask: 0.1732, loss_texture: 0.1959, iou: 0.7273, loss: 0.3691
2024-01-19 12:36:06,232 - mmdet - INFO - Iter [3600/120000] lr: 4.000e-04, eta: 6:29:21, time: 0.202, data_time: 0.003, memory: 6267, loss_mask: 0.1718, loss_texture: 0.1937, iou: 0.7334, loss: 0.3655
2024-01-19 12:36:16,231 - mmdet - INFO - Iter [3650/120000] lr: 4.000e-04, eta: 6:29:10, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1612, loss_texture: 0.1986, iou: 0.7909, loss: 0.3598
2024-01-19 12:36:26,350 - mmdet - INFO - Iter [3700/120000] lr: 4.000e-04, eta: 6:29:03, time: 0.202, data_time: 0.006, memory: 6267, loss_mask: 0.1699, loss_texture: 0.1903, iou: 0.7482, loss: 0.3603
2024-01-19 12:36:36,318 - mmdet - INFO - Iter [3750/120000] lr: 4.000e-04, eta: 6:28:51, time: 0.199, data_time: 0.003, memory: 6267, loss_mask: 0.1348, loss_texture: 0.1824, iou: 0.7632, loss: 0.3172
2024-01-19 12:36:47,253 - mmdet - INFO - Iter [3800/120000] lr: 4.000e-04, eta: 6:29:08, time: 0.219, data_time: 0.022, memory: 6267, loss_mask: 0.2047, loss_texture: 0.2051, iou: 0.7482, loss: 0.4098
2024-01-19 12:36:57,272 - mmdet - INFO - Iter [3850/120000] lr: 4.000e-04, eta: 6:28:57, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1410, loss_texture: 0.2001, iou: 0.7394, loss: 0.3412
2024-01-19 12:37:07,299 - mmdet - INFO - Iter [3900/120000] lr: 4.000e-04, eta: 6:28:47, time: 0.201, data_time: 0.003, memory: 6267, loss_mask: 0.1435, loss_texture: 0.1750, iou: 0.7415, loss: 0.3185
2024-01-19 12:37:17,327 - mmdet - INFO - Iter [3950/120000] lr: 4.000e-04, eta: 6:28:36, time: 0.201, data_time: 0.003, memory: 6267, loss_mask: 0.1412, loss_texture: 0.1831, iou: 0.7556, loss: 0.3242
2024-01-19 12:37:27,355 - mmdet - INFO - Exp name: segrefiner_hr.py
2024-01-19 12:37:27,356 - mmdet - INFO - Iter [4000/120000] lr: 4.000e-04, eta: 6:28:25, time: 0.201, data_time: 0.004, memory: 6267, loss_mask: 0.1276, loss_texture: 0.1654, iou: 0.7787, loss: 0.2930
2024-01-19 12:37:37,406 - mmdet - INFO - Iter [4050/120000] lr: 4.000e-04, eta: 6:28:16, time: 0.201, data_time: 0.003, memory: 6267, loss_mask: 0.1453, loss_texture: 0.1862, iou: 0.7722, loss: 0.3315
2024-01-19 12:37:47,268 - mmdet - INFO - Iter [4100/120000] lr: 4.000e-04, eta: 6:28:00, time: 0.197, data_time: 0.003, memory: 6267, loss_mask: 0.5625, loss_texture: 0.2051, iou: 0.1537, loss: 0.7676
2024-01-19 12:37:57,120 - mmdet - INFO - Iter [4150/120000] lr: 4.000e-04, eta: 6:27:45, time: 0.197, data_time: 0.003, memory: 6267, loss_mask: 0.6593, loss_texture: 0.2000, iou: 0.0000, loss: 0.8593
2024-01-19 12:38:06,906 - mmdet - INFO - Iter [4200/120000] lr: 4.000e-04, eta: 6:27:28, time: 0.196, data_time: 0.003, memory: 6267, loss_mask: 0.6285, loss_texture: 0.2061, iou: 0.0000, loss: 0.8345
2024-01-19 12:38:16,736 - mmdet - INFO - Iter [4250/120000] lr: 4.000e-04, eta: 6:27:12, time: 0.197, data_time: 0.003, memory: 6267, loss_mask: 0.6399, loss_texture: 0.1972, iou: 0.0000, loss: 0.8372
2024-01-19 12:38:26,524 - mmdet - INFO - Iter [4300/120000] lr: 4.000e-04, eta: 6:26:55, time: 0.196, data_time: 0.003, memory: 6267, loss_mask: 0.6509, loss_texture: 0.1880, iou: 0.0000, loss: 0.8390
2024-01-19 12:38:36,357 - mmdet - INFO - Iter [4350/120000] lr: 4.000e-04, eta: 6:26:40, time: 0.197, data_time: 0.003, memory: 6267, loss_mask: 0.6114, loss_texture: 0.2054, iou: 0.0000, loss: 0.8168
2024-01-19 12:38:46,185 - mmdet - INFO - Iter [4400/120000] lr: 4.000e-04, eta: 6:26:25, time: 0.197, data_time: 0.003, memory: 6267, loss_mask: 0.6601, loss_texture: 0.1975, iou: 0.0000, loss: 0.8576
2024-01-19 12:38:55,983 - mmdet - INFO - Iter [4450/120000] lr: 4.000e-04, eta: 6:26:09, time: 0.196, data_time: 0.003, memory: 6267, loss_mask: 0.6781, loss_texture: 0.2102, iou: 0.0000, loss: 0.8882
2024-01-19 12:39:06,238 - mmdet - INFO - Iter [4500/120000] lr: 4.000e-04, eta: 6:26:04, time: 0.205, data_time: 0.012, memory: 6267, loss_mask: 0.6450, loss_texture: 0.1818, iou: 0.0000, loss: 0.8268
2024-01-19 12:39:16,085 - mmdet - INFO - Iter [4550/120000] lr: 4.000e-04, eta: 6:25:50, time: 0.197, data_time: 0.003, memory: 6267, loss_mask: 0.6331, loss_texture: 0.2153, iou: 0.0000, loss: 0.8485
2024-01-19 12:39:26,005 - mmdet - INFO - Iter [4600/120000] lr: 4.000e-04, eta: 6:25:37, time: 0.198, data_time: 0.003, memory: 6267, loss_mask: 0.6359, loss_texture: 0.2007, iou: 0.0000, loss: 0.8366
2024-01-19 12:39:35,871 - mmdet - INFO - Iter [4650/120000] lr: 4.000e-04, eta: 6:25:23, time: 0.197, data_time: 0.003, memory: 6267, loss_mask: 0.6284, loss_texture: 0.1774, iou: 0.0000, loss: 0.8058
2024-01-19 12:39:45,796 - mmdet - INFO - Iter [4700/120000] lr: 4.000e-04, eta: 6:25:11, time: 0.198, data_time: 0.003, memory: 6267, loss_mask: 0.6485, loss_texture: 0.2023, iou: 0.0000, loss: 0.8507
2024-01-19 12:39:55,629 - mmdet - INFO - Iter [4750/120000] lr: 4.000e-04, eta: 6:24:56, time: 0.197, data_time: 0.003, memory: 6267, loss_mask: 0.6554, loss_texture: 0.1779, iou: 0.0000, loss: 0.8333
2024-01-19 12:40:05,472 - mmdet - INFO - Iter [4800/120000] lr: 4.000e-04, eta: 6:24:42, time: 0.197, data_time: 0.003, memory: 6267, loss_mask: 0.6776, loss_texture: 0.2007, iou: 0.0000, loss: 0.8783
yusuke-ai commented 5 months ago

also environmental information

mmcv-full              1.7.1
torch                  1.12.1+cu116
torchvision            0.13.1+cu116
mmdet                  2.28.1       /home/yusuke/gitrepos/SegRefiner
xiaolul2 commented 5 months ago

I also encounter the same problem....

xiaolul2 commented 5 months ago

No decline in loss when iou becomes 0. Maybe it is caused by gradient exploration. I use only 2 GPUs and change the learning rate to 1e-4 and solve this problem:)

yusuke-ai commented 5 months ago

@Liuliuliudalu Thank you for the information! Can I ask more questions?

  1. what is 14-e? 1e-14??
  2. Did you successfully trained the segmentation refiner with same quality as "segrefiner_hr_latest.pth"
xiaolul2 commented 5 months ago

@Liuliuliudalu Thank you for the information! Can I ask more questions?

  1. what is 14-e? 1e-14??
  2. Did you successfully trained the segmentation refiner with same quality as "segrefiner_hr_latest.pth"

Sorry for my typo, it is 1e-4. I'm still training the segrefiner_lr. Maybe I can reply you later after evaluation.

MengyuWang826 commented 5 months ago

Thank you all for your attention to this work and sorry for the delayed response. I believe the issue is due to training instability caused by a too-small batch size. The samples_per_gpu=1 setting in the config is only for convenient debugging, and the original results in the paper correspond to a global batch size of $4 \times 8=32$ . Therefore, please increase samples_per_gpu and the number of GPUs as much as possible. When the global batch size is small, a smaller learning rate can help stabilize training, but I'm not sure if this can reproduce the original results.

MengyuWang826 commented 5 months ago

Thank you all for your attention to this work and sorry for the delayed response. I believe the issue is due to training instability caused by a too-small batch size. The samples_per_gpu=1 setting in the config is only for convenient debugging, and the original results in the paper correspond to a global batch size of 4×8=32 . Therefore, please increase samples_per_gpu and the number of GPUs as much as possible. When the global batch size is small, a smaller learning rate can help stabilize training, but I'm not sure if this can reproduce the original results.

And through testing, I have confirmed that for the initial learning rate of 4e-4 in HR-SegRefiner, the issue of IOU dropping to 0 does not occur when the global batch size reaches 4.

yusuke-ai commented 5 months ago

@Liuliuliudalu Thank you for the answer! I couldn't train the same perfromant model as the provided ones...

@MengyuWang826 Thank you for the answer! I tried with 4e-4 and 2e-4 learning rate with batch size 16 * 4 = 64, but in either learning rate, the loss often increased a lot like the log below(around per 3000 iterations). Did you see such a trouble before?

2024-01-21 23:04:10,821 - mmdet - INFO - Iter [17450/120000]    lr: 2.000e-04, eta: 2 days, 1:07:27, time: 1.740, data_time: 0.378, memory: 70378, loss_mask: 0.0225, loss_texture: 0.0297, iou: 0.9646, loss: 0.0522
2024-01-21 23:05:35,286 - mmdet - INFO - Iter [17500/120000]    lr: 2.000e-04, eta: 2 days, 1:05:50, time: 1.689, data_time: 0.182, memory: 70378, loss_mask: 0.0204, loss_texture: 0.0279, iou: 0.9670, loss: 0.0483
2024-01-21 23:07:03,462 - mmdet - INFO - Iter [17550/120000]    lr: 2.000e-04, eta: 2 days, 1:04:36, time: 1.764, data_time: 0.461, memory: 70378, loss_mask: 0.0200, loss_texture: 0.0282, iou: 0.9661, loss: 0.0482
2024-01-21 23:08:29,160 - mmdet - INFO - Iter [17600/120000]    lr: 2.000e-04, eta: 2 days, 1:03:06, time: 1.714, data_time: 0.355, memory: 70378, loss_mask: 0.0224, loss_texture: 0.0303, iou: 0.9648, loss: 0.0527
2024-01-21 23:09:54,733 - mmdet - INFO - Iter [17650/120000]    lr: 2.000e-04, eta: 2 days, 1:01:36, time: 1.712, data_time: 0.475, memory: 70378, loss_mask: 0.0207, loss_texture: 0.0282, iou: 0.9687, loss: 0.0489
2024-01-21 23:11:20,855 - mmdet - INFO - Iter [17700/120000]    lr: 2.000e-04, eta: 2 days, 1:00:10, time: 1.722, data_time: 0.323, memory: 70378, loss_mask: 0.0250, loss_texture: 0.0313, iou: 0.9617, loss: 0.0564
2024-01-21 23:12:49,212 - mmdet - INFO - Iter [17750/120000]    lr: 2.000e-04, eta: 2 days, 0:58:56, time: 1.767, data_time: 0.080, memory: 70378, loss_mask: 0.4141, loss_texture: 0.1946, iou: 0.5304, loss: 0.6087
2024-01-21 23:14:12,939 - mmdet - INFO - Iter [17800/120000]    lr: 2.000e-04, eta: 2 days, 0:57:15, time: 1.675, data_time: 0.184, memory: 70378, loss_mask: 0.2365, loss_texture: 0.1971, iou: 0.6378, loss: 0.4336
2024-01-21 23:15:35,261 - mmdet - INFO - Iter [17850/120000]    lr: 2.000e-04, eta: 2 days, 0:55:27, time: 1.646, data_time: 0.153, memory: 70378, loss_mask: 0.2279, loss_texture: 0.1967, iou: 0.6453, loss: 0.4246
2024-01-21 23:17:04,512 - mmdet - INFO - Iter [17900/120000]    lr: 2.000e-04, eta: 2 days, 0:54:18, time: 1.785, data_time: 0.180, memory: 70378, loss_mask: 0.2043, loss_texture: 0.1948, iou: 0.6626, loss: 0.3991
MengyuWang826 commented 5 months ago

@Liuliuliudalu Thank you for the answer! I couldn't train the same perfromant model as the provided ones...

@MengyuWang826 Thank you for the answer! I tried with 4e-4 and 2e-4 learning rate with batch size 16 * 4 = 64, but in either learning rate, the loss often increased a lot like the log below(around per 3000 iterations). Did you see such a trouble before?

2024-01-21 23:04:10,821 - mmdet - INFO - Iter [17450/120000]  lr: 2.000e-04, eta: 2 days, 1:07:27, time: 1.740, data_time: 0.378, memory: 70378, loss_mask: 0.0225, loss_texture: 0.0297, iou: 0.9646, loss: 0.0522
2024-01-21 23:05:35,286 - mmdet - INFO - Iter [17500/120000]  lr: 2.000e-04, eta: 2 days, 1:05:50, time: 1.689, data_time: 0.182, memory: 70378, loss_mask: 0.0204, loss_texture: 0.0279, iou: 0.9670, loss: 0.0483
2024-01-21 23:07:03,462 - mmdet - INFO - Iter [17550/120000]  lr: 2.000e-04, eta: 2 days, 1:04:36, time: 1.764, data_time: 0.461, memory: 70378, loss_mask: 0.0200, loss_texture: 0.0282, iou: 0.9661, loss: 0.0482
2024-01-21 23:08:29,160 - mmdet - INFO - Iter [17600/120000]  lr: 2.000e-04, eta: 2 days, 1:03:06, time: 1.714, data_time: 0.355, memory: 70378, loss_mask: 0.0224, loss_texture: 0.0303, iou: 0.9648, loss: 0.0527
2024-01-21 23:09:54,733 - mmdet - INFO - Iter [17650/120000]  lr: 2.000e-04, eta: 2 days, 1:01:36, time: 1.712, data_time: 0.475, memory: 70378, loss_mask: 0.0207, loss_texture: 0.0282, iou: 0.9687, loss: 0.0489
2024-01-21 23:11:20,855 - mmdet - INFO - Iter [17700/120000]  lr: 2.000e-04, eta: 2 days, 1:00:10, time: 1.722, data_time: 0.323, memory: 70378, loss_mask: 0.0250, loss_texture: 0.0313, iou: 0.9617, loss: 0.0564
2024-01-21 23:12:49,212 - mmdet - INFO - Iter [17750/120000]  lr: 2.000e-04, eta: 2 days, 0:58:56, time: 1.767, data_time: 0.080, memory: 70378, loss_mask: 0.4141, loss_texture: 0.1946, iou: 0.5304, loss: 0.6087
2024-01-21 23:14:12,939 - mmdet - INFO - Iter [17800/120000]  lr: 2.000e-04, eta: 2 days, 0:57:15, time: 1.675, data_time: 0.184, memory: 70378, loss_mask: 0.2365, loss_texture: 0.1971, iou: 0.6378, loss: 0.4336
2024-01-21 23:15:35,261 - mmdet - INFO - Iter [17850/120000]  lr: 2.000e-04, eta: 2 days, 0:55:27, time: 1.646, data_time: 0.153, memory: 70378, loss_mask: 0.2279, loss_texture: 0.1967, iou: 0.6453, loss: 0.4246
2024-01-21 23:17:04,512 - mmdet - INFO - Iter [17900/120000]  lr: 2.000e-04, eta: 2 days, 0:54:18, time: 1.785, data_time: 0.180, memory: 70378, loss_mask: 0.2043, loss_texture: 0.1948, iou: 0.6626, loss: 0.3991

This is a bit strange. When I have enough free GPUs available, I will check and retrain it.

yusuke-ai commented 5 months ago

@MengyuWang826 Thank you!

Taocunguo commented 3 months ago

还有环境信息

mmcv-full              1.7.1
torch                  1.12.1+cu116
torchvision            0.13.1+cu116
mmdet                  2.28.1       /home/yusuke/gitrepos/SegRefiner

我遇到了类似的问题，训练配置和日志如下。

训练配置

checkpoint_config = dict(
    interval=5000, by_epoch=False, save_last=True, max_keep_ckpts=20)
log_config = dict(
    interval=50, hooks=[dict(type='TextLoggerHook', by_epoch=False)])
dist_params = dict(backend='nccl')
log_level = 'INFO'
load_from = None
resume_from = 'segrefiner_hr_latest.pth'
workflow = [('train', 5000)]
opencv_num_threads = 0
mp_start_method = 'fork'
auto_scale_lr = dict(enable=False, base_batch_size=16)
object_size = 256
task = 'instance'
model = dict(
    type='SegRefiner',
    task='instance',
    step=6,
    denoise_model=dict(
        type='DenoiseUNet',
        in_channels=4,
        out_channels=1,
        model_channels=128,
        num_res_blocks=2,
        num_heads=4,
        num_heads_upsample=-1,
        attention_strides=(16, 32),
        learn_time_embd=True,
        channel_mult=(1, 1, 2, 2, 4, 4),
        dropout=0.0),
    diffusion_cfg=dict(
        betas=dict(type='linear', start=0.8, stop=0, num_timesteps=6),
        diff_iter=False),
    test_cfg=dict())
img_norm_cfg = dict(
    mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_rgb=True)
train_pipeline = [
    dict(type='LoadImageFromFile'),
    dict(
        type='LoadAnnotations',
        with_bbox=False,
        with_label=False,
        with_mask=True),
    dict(type='LoadPatchData', object_size=256, patch_size=256),
    dict(type='Resize', img_scale=(256, 256), keep_ratio=False),
    dict(type='RandomFlip', flip_ratio=0.5),
    dict(
        type='Normalize',
        mean=[123.675, 116.28, 103.53],
        std=[58.395, 57.12, 57.375],
        to_rgb=True),
    dict(type='DefaultFormatBundle'),
    dict(
        type='Collect',
        keys=[
            'object_img', 'object_gt_masks', 'object_coarse_masks',
            'patch_img', 'patch_gt_masks', 'patch_coarse_masks'
        ])
]
dataset_type = 'HRCollectionDataset'
img_root = '/share/project/datasets/MSCOCO/coco2017/'
ann_root = '/share/project/datasets/LVIS/'
train_dataloader = dict(samples_per_gpu=1, workers_per_gpu=1)
data = dict(
    train=dict(
        type='HRCollectionDataset',
        pipeline=[
            dict(type='LoadImageFromFile'),
            dict(
                type='LoadAnnotations',
                with_bbox=False,
                with_label=False,
                with_mask=True),
            dict(type='LoadPatchData', object_size=256, patch_size=256),
            dict(type='Resize', img_scale=(256, 256), keep_ratio=False),
            dict(type='RandomFlip', flip_ratio=0.5),
            dict(
                type='Normalize',
                mean=[123.675, 116.28, 103.53],
                std=[58.395, 57.12, 57.375],
                to_rgb=True),
            dict(type='DefaultFormatBundle'),
            dict(
                type='Collect',
                keys=[
                    'object_img', 'object_gt_masks', 'object_coarse_masks',
                    'patch_img', 'patch_gt_masks', 'patch_coarse_masks'
                ])
        ],
        data_root='data/',
        collection_datasets=['thin', 'dis'],
        collection_json='data/collection_hr.json'),
    train_dataloader=dict(samples_per_gpu=1, workers_per_gpu=1),
    val=dict(),
    test=dict())
optimizer = dict(
    type='AdamW', lr=0.0004, weight_decay=0, eps=1e-08, betas=(0.9, 0.999))
optimizer_config = dict(grad_clip=None)
max_iters = 120000
runner = dict(type='IterBasedRunner', max_iters=120000)
lr_config = dict(
    policy='step',
    gamma=0.5,
    by_epoch=False,
    step=[80000, 100000],
    warmup='linear',
    warmup_by_epoch=False,
    warmup_ratio=1.0,
    warmup_iters=10)
interval = 5000
data_root = 'data/'
work_dir = './work_dirs/segrefiner_hr'
auto_resume = False
gpu_ids = [0]

日志

2024-01-19 12:24:03,655 - mmdet - INFO - workflow: [('train', 5000)], max: 120000 iters
2024-01-19 12:24:03,655 - mmdet - INFO - Checkpoints will be saved to /home/yusuke/gitrepos/SegRefiner/work_dirs/segrefiner_hr by HardDiskBackend.
2024-01-19 12:24:15,210 - mmdet - INFO - Iter [50/120000] lr: 4.000e-04, eta: 7:41:07, time: 0.231, data_time: 0.005, memory: 6267, loss_mask: 0.0749, loss_texture: 0.0805, iou: 0.9283, loss: 0.1554
2024-01-19 12:24:25,180 - mmdet - INFO - Iter [100/120000]    lr: 4.000e-04, eta: 7:09:42, time: 0.199, data_time: 0.003, memory: 6267, loss_mask: 0.0736, loss_texture: 0.0857, iou: 0.9084, loss: 0.1592
2024-01-19 12:24:35,107 - mmdet - INFO - Iter [150/120000]    lr: 4.000e-04, eta: 6:58:32, time: 0.199, data_time: 0.003, memory: 6267, loss_mask: 0.4370, loss_texture: 0.2199, iou: 0.5518, loss: 0.6569
2024-01-19 12:24:45,140 - mmdet - INFO - Iter [200/120000]    lr: 4.000e-04, eta: 6:53:55, time: 0.201, data_time: 0.003, memory: 6267, loss_mask: 0.2852, loss_texture: 0.2145, iou: 0.5794, loss: 0.4996
2024-01-19 12:24:55,056 - mmdet - INFO - Iter [250/120000]    lr: 4.000e-04, eta: 6:50:09, time: 0.198, data_time: 0.003, memory: 6267, loss_mask: 0.2346, loss_texture: 0.2139, iou: 0.6228, loss: 0.4484
2024-01-19 12:25:05,039 - mmdet - INFO - Iter [300/120000]    lr: 4.000e-04, eta: 6:48:02, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.2050, loss_texture: 0.1940, iou: 0.6370, loss: 0.3990
2024-01-19 12:25:14,964 - mmdet - INFO - Iter [350/120000]    lr: 4.000e-04, eta: 6:46:09, time: 0.198, data_time: 0.003, memory: 6267, loss_mask: 0.2655, loss_texture: 0.2266, iou: 0.6330, loss: 0.4921
2024-01-19 12:25:24,949 - mmdet - INFO - Iter [400/120000]    lr: 4.000e-04, eta: 6:44:59, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.2196, loss_texture: 0.2139, iou: 0.7239, loss: 0.4336
2024-01-19 12:25:34,935 - mmdet - INFO - Iter [450/120000]    lr: 4.000e-04, eta: 6:44:03, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1660, loss_texture: 0.1727, iou: 0.7408, loss: 0.3387
2024-01-19 12:25:44,947 - mmdet - INFO - Iter [500/120000]    lr: 4.000e-04, eta: 6:43:22, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.2036, loss_texture: 0.2130, iou: 0.7069, loss: 0.4167
2024-01-19 12:25:54,943 - mmdet - INFO - Iter [550/120000]    lr: 4.000e-04, eta: 6:42:44, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1658, loss_texture: 0.1909, iou: 0.7922, loss: 0.3566
2024-01-19 12:26:04,933 - mmdet - INFO - Iter [600/120000]    lr: 4.000e-04, eta: 6:42:09, time: 0.200, data_time: 0.004, memory: 6267, loss_mask: 0.1983, loss_texture: 0.2182, iou: 0.7171, loss: 0.4165
2024-01-19 12:26:14,901 - mmdet - INFO - Iter [650/120000]    lr: 4.000e-04, eta: 6:41:33, time: 0.199, data_time: 0.003, memory: 6267, loss_mask: 0.1579, loss_texture: 0.1868, iou: 0.7118, loss: 0.3447
2024-01-19 12:26:24,843 - mmdet - INFO - Iter [700/120000]    lr: 4.000e-04, eta: 6:40:57, time: 0.199, data_time: 0.003, memory: 6267, loss_mask: 0.2049, loss_texture: 0.2065, iou: 0.7138, loss: 0.4114
2024-01-19 12:26:34,789 - mmdet - INFO - Iter [750/120000]    lr: 4.000e-04, eta: 6:40:25, time: 0.199, data_time: 0.003, memory: 6267, loss_mask: 0.2391, loss_texture: 0.2162, iou: 0.6966, loss: 0.4553
2024-01-19 12:26:44,700 - mmdet - INFO - Iter [800/120000]    lr: 4.000e-04, eta: 6:39:51, time: 0.198, data_time: 0.003, memory: 6267, loss_mask: 0.1903, loss_texture: 0.2007, iou: 0.6805, loss: 0.3910
2024-01-19 12:26:54,823 - mmdet - INFO - Iter [850/120000]    lr: 4.000e-04, eta: 6:39:49, time: 0.202, data_time: 0.006, memory: 6267, loss_mask: 0.2275, loss_texture: 0.2216, iou: 0.6903, loss: 0.4491
2024-01-19 12:27:04,774 - mmdet - INFO - Iter [900/120000]    lr: 4.000e-04, eta: 6:39:24, time: 0.199, data_time: 0.003, memory: 6267, loss_mask: 0.2924, loss_texture: 0.2216, iou: 0.6570, loss: 0.5139
2024-01-19 12:27:14,760 - mmdet - INFO - Iter [950/120000]    lr: 4.000e-04, eta: 6:39:04, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.2299, loss_texture: 0.2123, iou: 0.6476, loss: 0.4421
2024-01-19 12:27:24,758 - mmdet - INFO - Exp name: segrefiner_hr.py
2024-01-19 12:27:24,758 - mmdet - INFO - Iter [1000/120000]   lr: 4.000e-04, eta: 6:38:47, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.2042, loss_texture: 0.1917, iou: 0.6918, loss: 0.3960
2024-01-19 12:27:34,786 - mmdet - INFO - Iter [1050/120000]   lr: 4.000e-04, eta: 6:38:34, time: 0.201, data_time: 0.004, memory: 6267, loss_mask: 0.2077, loss_texture: 0.1856, iou: 0.7152, loss: 0.3933
2024-01-19 12:27:44,821 - mmdet - INFO - Iter [1100/120000]   lr: 4.000e-04, eta: 6:38:22, time: 0.201, data_time: 0.003, memory: 6267, loss_mask: 0.1809, loss_texture: 0.1987, iou: 0.7236, loss: 0.3796
2024-01-19 12:27:54,808 - mmdet - INFO - Iter [1150/120000]   lr: 4.000e-04, eta: 6:38:06, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.2191, loss_texture: 0.2069, iou: 0.7322, loss: 0.4260
2024-01-19 12:28:04,823 - mmdet - INFO - Iter [1200/120000]   lr: 4.000e-04, eta: 6:37:52, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.2450, loss_texture: 0.2021, iou: 0.7341, loss: 0.4471
2024-01-19 12:28:14,884 - mmdet - INFO - Iter [1250/120000]   lr: 4.000e-04, eta: 6:37:43, time: 0.201, data_time: 0.003, memory: 6267, loss_mask: 0.1971, loss_texture: 0.2114, iou: 0.7124, loss: 0.4085
2024-01-19 12:28:24,896 - mmdet - INFO - Iter [1300/120000]   lr: 4.000e-04, eta: 6:37:30, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1347, loss_texture: 0.1755, iou: 0.8245, loss: 0.3102
2024-01-19 12:28:35,061 - mmdet - INFO - Iter [1350/120000]   lr: 4.000e-04, eta: 6:37:30, time: 0.203, data_time: 0.006, memory: 6267, loss_mask: 0.2002, loss_texture: 0.1978, iou: 0.7008, loss: 0.3980
2024-01-19 12:28:45,045 - mmdet - INFO - Iter [1400/120000]   lr: 4.000e-04, eta: 6:37:15, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1808, loss_texture: 0.1898, iou: 0.7335, loss: 0.3705
2024-01-19 12:28:54,994 - mmdet - INFO - Iter [1450/120000]   lr: 4.000e-04, eta: 6:36:56, time: 0.199, data_time: 0.003, memory: 6267, loss_mask: 0.1796, loss_texture: 0.1766, iou: 0.7720, loss: 0.3561
2024-01-19 12:29:04,995 - mmdet - INFO - Iter [1500/120000]   lr: 4.000e-04, eta: 6:36:43, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1564, loss_texture: 0.1758, iou: 0.7233, loss: 0.3321
2024-01-19 12:29:14,931 - mmdet - INFO - Iter [1550/120000]   lr: 4.000e-04, eta: 6:36:25, time: 0.199, data_time: 0.003, memory: 6267, loss_mask: 0.1863, loss_texture: 0.2102, iou: 0.7394, loss: 0.3965
2024-01-19 12:29:25,060 - mmdet - INFO - Iter [1600/120000]   lr: 4.000e-04, eta: 6:36:21, time: 0.203, data_time: 0.005, memory: 6267, loss_mask: 0.2034, loss_texture: 0.2099, iou: 0.7317, loss: 0.4134
2024-01-19 12:29:35,074 - mmdet - INFO - Iter [1650/120000]   lr: 4.000e-04, eta: 6:36:09, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1500, loss_texture: 0.1764, iou: 0.7995, loss: 0.3263
2024-01-19 12:29:45,054 - mmdet - INFO - Iter [1700/120000]   lr: 4.000e-04, eta: 6:35:55, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1477, loss_texture: 0.1792, iou: 0.7476, loss: 0.3270
2024-01-19 12:29:55,016 - mmdet - INFO - Iter [1750/120000]   lr: 4.000e-04, eta: 6:35:39, time: 0.199, data_time: 0.003, memory: 6267, loss_mask: 0.1770, loss_texture: 0.1875, iou: 0.7434, loss: 0.3645
2024-01-19 12:30:04,943 - mmdet - INFO - Iter [1800/120000]   lr: 4.000e-04, eta: 6:35:22, time: 0.199, data_time: 0.003, memory: 6267, loss_mask: 0.1464, loss_texture: 0.1856, iou: 0.7543, loss: 0.3320
2024-01-19 12:30:14,854 - mmdet - INFO - Iter [1850/120000]   lr: 4.000e-04, eta: 6:35:04, time: 0.198, data_time: 0.003, memory: 6267, loss_mask: 0.1349, loss_texture: 0.1745, iou: 0.7693, loss: 0.3094
2024-01-19 12:30:24,803 - mmdet - INFO - Iter [1900/120000]   lr: 4.000e-04, eta: 6:34:49, time: 0.199, data_time: 0.003, memory: 6267, loss_mask: 0.1340, loss_texture: 0.1688, iou: 0.8084, loss: 0.3028
2024-01-19 12:30:34,767 - mmdet - INFO - Iter [1950/120000]   lr: 4.000e-04, eta: 6:34:35, time: 0.199, data_time: 0.003, memory: 6267, loss_mask: 0.1249, loss_texture: 0.1673, iou: 0.7941, loss: 0.2922
2024-01-19 12:30:44,694 - mmdet - INFO - Exp name: segrefiner_hr.py
2024-01-19 12:30:44,694 - mmdet - INFO - Iter [2000/120000]   lr: 4.000e-04, eta: 6:34:19, time: 0.199, data_time: 0.003, memory: 6267, loss_mask: 0.2065, loss_texture: 0.1790, iou: 0.7811, loss: 0.3856
2024-01-19 12:30:54,604 - mmdet - INFO - Iter [2050/120000]   lr: 4.000e-04, eta: 6:34:02, time: 0.198, data_time: 0.003, memory: 6267, loss_mask: 0.1713, loss_texture: 0.2143, iou: 0.7463, loss: 0.3856
2024-01-19 12:31:04,618 - mmdet - INFO - Iter [2100/120000]   lr: 4.000e-04, eta: 6:33:52, time: 0.200, data_time: 0.004, memory: 6267, loss_mask: 0.2016, loss_texture: 0.2008, iou: 0.7280, loss: 0.4023
2024-01-19 12:31:14,605 - mmdet - INFO - Iter [2150/120000]   lr: 4.000e-04, eta: 6:33:40, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1621, loss_texture: 0.2051, iou: 0.7716, loss: 0.3672
2024-01-19 12:31:24,538 - mmdet - INFO - Iter [2200/120000]   lr: 4.000e-04, eta: 6:33:25, time: 0.199, data_time: 0.003, memory: 6267, loss_mask: 0.1611, loss_texture: 0.1994, iou: 0.8068, loss: 0.3604
2024-01-19 12:31:34,484 - mmdet - INFO - Iter [2250/120000]   lr: 4.000e-04, eta: 6:33:11, time: 0.199, data_time: 0.003, memory: 6267, loss_mask: 0.1697, loss_texture: 0.2121, iou: 0.6758, loss: 0.3818
2024-01-19 12:31:44,713 - mmdet - INFO - Iter [2300/120000]   lr: 4.000e-04, eta: 6:33:12, time: 0.205, data_time: 0.008, memory: 6267, loss_mask: 0.1664, loss_texture: 0.1805, iou: 0.7210, loss: 0.3469
2024-01-19 12:31:54,648 - mmdet - INFO - Iter [2350/120000]   lr: 4.000e-04, eta: 6:32:57, time: 0.199, data_time: 0.003, memory: 6267, loss_mask: 0.1260, loss_texture: 0.1762, iou: 0.7685, loss: 0.3021
2024-01-19 12:32:04,631 - mmdet - INFO - Iter [2400/120000]   lr: 4.000e-04, eta: 6:32:45, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1444, loss_texture: 0.1815, iou: 0.8099, loss: 0.3260
2024-01-19 12:32:15,601 - mmdet - INFO - Iter [2450/120000]   lr: 4.000e-04, eta: 6:33:21, time: 0.219, data_time: 0.021, memory: 6267, loss_mask: 0.1563, loss_texture: 0.1941, iou: 0.7882, loss: 0.3505
2024-01-19 12:32:25,657 - mmdet - INFO - Iter [2500/120000]   lr: 4.000e-04, eta: 6:33:12, time: 0.201, data_time: 0.003, memory: 6267, loss_mask: 0.1376, loss_texture: 0.1658, iou: 0.7853, loss: 0.3034
2024-01-19 12:32:35,804 - mmdet - INFO - Iter [2550/120000]   lr: 4.000e-04, eta: 6:33:07, time: 0.203, data_time: 0.003, memory: 6267, loss_mask: 0.1314, loss_texture: 0.1676, iou: 0.8216, loss: 0.2991
2024-01-19 12:32:45,851 - mmdet - INFO - Iter [2600/120000]   lr: 4.000e-04, eta: 6:32:57, time: 0.201, data_time: 0.003, memory: 6267, loss_mask: 0.1733, loss_texture: 0.1933, iou: 0.7472, loss: 0.3666
2024-01-19 12:32:55,850 - mmdet - INFO - Iter [2650/120000]   lr: 4.000e-04, eta: 6:32:45, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1992, loss_texture: 0.2001, iou: 0.6920, loss: 0.3993
2024-01-19 12:33:05,933 - mmdet - INFO - Iter [2700/120000]   lr: 4.000e-04, eta: 6:32:37, time: 0.202, data_time: 0.003, memory: 6267, loss_mask: 0.1481, loss_texture: 0.1848, iou: 0.7880, loss: 0.3328
2024-01-19 12:33:15,942 - mmdet - INFO - Iter [2750/120000]   lr: 4.000e-04, eta: 6:32:25, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1818, loss_texture: 0.1817, iou: 0.7667, loss: 0.3635
2024-01-19 12:33:25,938 - mmdet - INFO - Iter [2800/120000]   lr: 4.000e-04, eta: 6:32:13, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1897, loss_texture: 0.1949, iou: 0.7399, loss: 0.3846
2024-01-19 12:33:36,020 - mmdet - INFO - Iter [2850/120000]   lr: 4.000e-04, eta: 6:32:05, time: 0.202, data_time: 0.003, memory: 6267, loss_mask: 0.1986, loss_texture: 0.1722, iou: 0.7496, loss: 0.3707
2024-01-19 12:33:46,018 - mmdet - INFO - Iter [2900/120000]   lr: 4.000e-04, eta: 6:31:53, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1509, loss_texture: 0.1899, iou: 0.7735, loss: 0.3408
2024-01-19 12:33:56,028 - mmdet - INFO - Iter [2950/120000]   lr: 4.000e-04, eta: 6:31:42, time: 0.200, data_time: 0.004, memory: 6267, loss_mask: 0.1364, loss_texture: 0.1691, iou: 0.7962, loss: 0.3055
2024-01-19 12:34:06,055 - mmdet - INFO - Exp name: segrefiner_hr.py
2024-01-19 12:34:06,055 - mmdet - INFO - Iter [3000/120000]   lr: 4.000e-04, eta: 6:31:31, time: 0.201, data_time: 0.003, memory: 6267, loss_mask: 0.1484, loss_texture: 0.1878, iou: 0.7773, loss: 0.3362
2024-01-19 12:34:16,073 - mmdet - INFO - Iter [3050/120000]   lr: 4.000e-04, eta: 6:31:21, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1766, loss_texture: 0.2010, iou: 0.7864, loss: 0.3776
2024-01-19 12:34:26,081 - mmdet - INFO - Iter [3100/120000]   lr: 4.000e-04, eta: 6:31:09, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1475, loss_texture: 0.1973, iou: 0.7756, loss: 0.3448
2024-01-19 12:34:36,077 - mmdet - INFO - Iter [3150/120000]   lr: 4.000e-04, eta: 6:30:58, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1222, loss_texture: 0.1656, iou: 0.8325, loss: 0.2878
2024-01-19 12:34:46,095 - mmdet - INFO - Iter [3200/120000]   lr: 4.000e-04, eta: 6:30:47, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1147, loss_texture: 0.1624, iou: 0.7541, loss: 0.2771
2024-01-19 12:34:56,057 - mmdet - INFO - Iter [3250/120000]   lr: 4.000e-04, eta: 6:30:34, time: 0.199, data_time: 0.003, memory: 6267, loss_mask: 0.1926, loss_texture: 0.1761, iou: 0.7865, loss: 0.3687
2024-01-19 12:35:06,074 - mmdet - INFO - Iter [3300/120000]   lr: 4.000e-04, eta: 6:30:23, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1360, loss_texture: 0.1826, iou: 0.7763, loss: 0.3186
2024-01-19 12:35:16,077 - mmdet - INFO - Iter [3350/120000]   lr: 4.000e-04, eta: 6:30:12, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.3127, loss_texture: 0.1954, iou: 0.6610, loss: 0.5081
2024-01-19 12:35:26,089 - mmdet - INFO - Iter [3400/120000]   lr: 4.000e-04, eta: 6:30:01, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.2593, loss_texture: 0.1952, iou: 0.6893, loss: 0.4545
2024-01-19 12:35:36,096 - mmdet - INFO - Iter [3450/120000]   lr: 4.000e-04, eta: 6:29:50, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1894, loss_texture: 0.2158, iou: 0.7296, loss: 0.4052
2024-01-19 12:35:46,117 - mmdet - INFO - Iter [3500/120000]   lr: 4.000e-04, eta: 6:29:40, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.2522, loss_texture: 0.2290, iou: 0.6647, loss: 0.4813
2024-01-19 12:35:56,152 - mmdet - INFO - Iter [3550/120000]   lr: 4.000e-04, eta: 6:29:30, time: 0.201, data_time: 0.003, memory: 6267, loss_mask: 0.1732, loss_texture: 0.1959, iou: 0.7273, loss: 0.3691
2024-01-19 12:36:06,232 - mmdet - INFO - Iter [3600/120000]   lr: 4.000e-04, eta: 6:29:21, time: 0.202, data_time: 0.003, memory: 6267, loss_mask: 0.1718, loss_texture: 0.1937, iou: 0.7334, loss: 0.3655
2024-01-19 12:36:16,231 - mmdet - INFO - Iter [3650/120000]   lr: 4.000e-04, eta: 6:29:10, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1612, loss_texture: 0.1986, iou: 0.7909, loss: 0.3598
2024-01-19 12:36:26,350 - mmdet - INFO - Iter [3700/120000]   lr: 4.000e-04, eta: 6:29:03, time: 0.202, data_time: 0.006, memory: 6267, loss_mask: 0.1699, loss_texture: 0.1903, iou: 0.7482, loss: 0.3603
2024-01-19 12:36:36,318 - mmdet - INFO - Iter [3750/120000]   lr: 4.000e-04, eta: 6:28:51, time: 0.199, data_time: 0.003, memory: 6267, loss_mask: 0.1348, loss_texture: 0.1824, iou: 0.7632, loss: 0.3172
2024-01-19 12:36:47,253 - mmdet - INFO - Iter [3800/120000]   lr: 4.000e-04, eta: 6:29:08, time: 0.219, data_time: 0.022, memory: 6267, loss_mask: 0.2047, loss_texture: 0.2051, iou: 0.7482, loss: 0.4098
2024-01-19 12:36:57,272 - mmdet - INFO - Iter [3850/120000]   lr: 4.000e-04, eta: 6:28:57, time: 0.200, data_time: 0.003, memory: 6267, loss_mask: 0.1410, loss_texture: 0.2001, iou: 0.7394, loss: 0.3412
2024-01-19 12:37:07,299 - mmdet - INFO - Iter [3900/120000]   lr: 4.000e-04, eta: 6:28:47, time: 0.201, data_time: 0.003, memory: 6267, loss_mask: 0.1435, loss_texture: 0.1750, iou: 0.7415, loss: 0.3185
2024-01-19 12:37:17,327 - mmdet - INFO - Iter [3950/120000]   lr: 4.000e-04, eta: 6:28:36, time: 0.201, data_time: 0.003, memory: 6267, loss_mask: 0.1412, loss_texture: 0.1831, iou: 0.7556, loss: 0.3242
2024-01-19 12:37:27,355 - mmdet - INFO - Exp name: segrefiner_hr.py
2024-01-19 12:37:27,356 - mmdet - INFO - Iter [4000/120000]   lr: 4.000e-04, eta: 6:28:25, time: 0.201, data_time: 0.004, memory: 6267, loss_mask: 0.1276, loss_texture: 0.1654, iou: 0.7787, loss: 0.2930
2024-01-19 12:37:37,406 - mmdet - INFO - Iter [4050/120000]   lr: 4.000e-04, eta: 6:28:16, time: 0.201, data_time: 0.003, memory: 6267, loss_mask: 0.1453, loss_texture: 0.1862, iou: 0.7722, loss: 0.3315
2024-01-19 12:37:47,268 - mmdet - INFO - Iter [4100/120000]   lr: 4.000e-04, eta: 6:28:00, time: 0.197, data_time: 0.003, memory: 6267, loss_mask: 0.5625, loss_texture: 0.2051, iou: 0.1537, loss: 0.7676
2024-01-19 12:37:57,120 - mmdet - INFO - Iter [4150/120000]   lr: 4.000e-04, eta: 6:27:45, time: 0.197, data_time: 0.003, memory: 6267, loss_mask: 0.6593, loss_texture: 0.2000, iou: 0.0000, loss: 0.8593
2024-01-19 12:38:06,906 - mmdet - INFO - Iter [4200/120000]   lr: 4.000e-04, eta: 6:27:28, time: 0.196, data_time: 0.003, memory: 6267, loss_mask: 0.6285, loss_texture: 0.2061, iou: 0.0000, loss: 0.8345
2024-01-19 12:38:16,736 - mmdet - INFO - Iter [4250/120000]   lr: 4.000e-04, eta: 6:27:12, time: 0.197, data_time: 0.003, memory: 6267, loss_mask: 0.6399, loss_texture: 0.1972, iou: 0.0000, loss: 0.8372
2024-01-19 12:38:26,524 - mmdet - INFO - Iter [4300/120000]   lr: 4.000e-04, eta: 6:26:55, time: 0.196, data_time: 0.003, memory: 6267, loss_mask: 0.6509, loss_texture: 0.1880, iou: 0.0000, loss: 0.8390
2024-01-19 12:38:36,357 - mmdet - INFO - Iter [4350/120000]   lr: 4.000e-04, eta: 6:26:40, time: 0.197, data_time: 0.003, memory: 6267, loss_mask: 0.6114, loss_texture: 0.2054, iou: 0.0000, loss: 0.8168
2024-01-19 12:38:46,185 - mmdet - INFO - Iter [4400/120000]   lr: 4.000e-04, eta: 6:26:25, time: 0.197, data_time: 0.003, memory: 6267, loss_mask: 0.6601, loss_texture: 0.1975, iou: 0.0000, loss: 0.8576
2024-01-19 12:38:55,983 - mmdet - INFO - Iter [4450/120000]   lr: 4.000e-04, eta: 6:26:09, time: 0.196, data_time: 0.003, memory: 6267, loss_mask: 0.6781, loss_texture: 0.2102, iou: 0.0000, loss: 0.8882
2024-01-19 12:39:06,238 - mmdet - INFO - Iter [4500/120000]   lr: 4.000e-04, eta: 6:26:04, time: 0.205, data_time: 0.012, memory: 6267, loss_mask: 0.6450, loss_texture: 0.1818, iou: 0.0000, loss: 0.8268
2024-01-19 12:39:16,085 - mmdet - INFO - Iter [4550/120000]   lr: 4.000e-04, eta: 6:25:50, time: 0.197, data_time: 0.003, memory: 6267, loss_mask: 0.6331, loss_texture: 0.2153, iou: 0.0000, loss: 0.8485
2024-01-19 12:39:26,005 - mmdet - INFO - Iter [4600/120000]   lr: 4.000e-04, eta: 6:25:37, time: 0.198, data_time: 0.003, memory: 6267, loss_mask: 0.6359, loss_texture: 0.2007, iou: 0.0000, loss: 0.8366
2024-01-19 12:39:35,871 - mmdet - INFO - Iter [4650/120000]   lr: 4.000e-04, eta: 6:25:23, time: 0.197, data_time: 0.003, memory: 6267, loss_mask: 0.6284, loss_texture: 0.1774, iou: 0.0000, loss: 0.8058
2024-01-19 12:39:45,796 - mmdet - INFO - Iter [4700/120000]   lr: 4.000e-04, eta: 6:25:11, time: 0.198, data_time: 0.003, memory: 6267, loss_mask: 0.6485, loss_texture: 0.2023, iou: 0.0000, loss: 0.8507
2024-01-19 12:39:55,629 - mmdet - INFO - Iter [4750/120000]   lr: 4.000e-04, eta: 6:24:56, time: 0.197, data_time: 0.003, memory: 6267, loss_mask: 0.6554, loss_texture: 0.1779, iou: 0.0000, loss: 0.8333
2024-01-19 12:40:05,472 - mmdet - INFO - Iter [4800/120000]   lr: 4.000e-04, eta: 6:24:42, time: 0.197, data_time: 0.003, memory: 6267, loss_mask: 0.6776, loss_texture: 0.2007, iou: 0.0000, loss: 0.8783

Hello, I also encountered the same problem, may I ask you to solve it？

yusuke-ai commented 2 months ago

@MengyuWang826 Hi! Did you found out if there's the issue in the training process?