search

open-mmlab / mmocr

OpenMMLab Text Detection, Recognition and Understanding Toolbox
https://mmocr.readthedocs.io/en/dev-1.x/
Apache License 2.0
4.35k stars 751 forks source link

AssertionError while training textsnake model #339

Open mobassir94 opened 3 years ago

mobassir94 commented 3 years ago

i was training textsnake and after train epoch i get assertion error,please check my model and train+error log attached : 

QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-apsisdev'
QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-apsisdev'
Config:
optimizer = dict(type='SGD', lr=0.000125, momentum=0.9, weight_decay=0.0001)
optimizer_config = dict(grad_clip=None)
lr_config = dict(
    policy='poly', power=0.9, min_lr=1e-07, by_epoch=True, warmup=None)
total_epochs = 1200
checkpoint_config = dict(interval=1)
log_config = dict(interval=1, hooks=[dict(type='TextLoggerHook')])
dist_params = dict(backend='nccl')
log_level = 'INFO'
load_from = None
resume_from = None
workflow = [('train', 1)]
model = dict(
    type='TextSnake',
    pretrained='torchvision://resnet50',
    backbone=dict(
        type='ResNet',
        depth=50,
        num_stages=4,
        out_indices=(0, 1, 2, 3),
        frozen_stages=-1,
        norm_cfg=dict(type='BN', requires_grad=True),
        norm_eval=True,
        style='caffe'),
    neck=dict(
        type='FPN_UNet', in_channels=[256, 512, 1024, 2048], out_channels=32),
    bbox_head=dict(
        type='TextSnakeHead',
        in_channels=32,
        text_repr_type='poly',
        loss=dict(type='TextSnakeLoss')),
    train_cfg=None,
    test_cfg=None)
dataset_type = 'IcdarDataset'
data_root = '/home/apsisdev/IMPORTANT/totaltext'
img_norm_cfg = dict(
    mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_rgb=True)
train_pipeline = [
    dict(type='LoadImageFromFile'),
    dict(
        type='LoadTextAnnotations',
        with_bbox=True,
        with_mask=True,
        poly2mask=False),
    dict(type='ColorJitter', brightness=0.12549019607843137, saturation=0.5),
    dict(
        type='Normalize',
        mean=[123.675, 116.28, 103.53],
        std=[58.395, 57.12, 57.375],
        to_rgb=True),
    dict(
        type='RandomCropPolyInstances',
        instance_key='gt_masks',
        crop_ratio=0.65,
        min_side_ratio=0.3),
    dict(
        type='RandomRotatePolyInstances',
        rotate_ratio=0.5,
        max_angle=20,
        pad_with_fixed_color=False),
    dict(
        type='ScaleAspectJitter',
        img_scale=[(3000, 736)],
        ratio_range=(0.7, 1.3),
        aspect_ratio_range=(0.9, 1.1),
        multiscale_mode='value',
        long_size_bound=800,
        short_size_bound=480,
        resize_type='long_short_bound',
        keep_ratio=False),
    dict(type='SquareResizePad', target_size=800, pad_ratio=0.6),
    dict(type='RandomFlip', flip_ratio=0.5, direction='horizontal'),
    dict(type='TextSnakeTargets'),
    dict(type='Pad', size_divisor=32),
    dict(
        type='CustomFormatBundle',
        keys=[
            'gt_text_mask', 'gt_center_region_mask', 'gt_mask',
            'gt_radius_map', 'gt_sin_map', 'gt_cos_map'
        ],
        visualize=dict(flag=False, boundary_key='gt_text_mask')),
    dict(
        type='Collect',
        keys=[
            'img', 'gt_text_mask', 'gt_center_region_mask', 'gt_mask',
            'gt_radius_map', 'gt_sin_map', 'gt_cos_map'
        ])
]
test_pipeline = [
    dict(type='LoadImageFromFile'),
    dict(
        type='MultiScaleFlipAug',
        img_scale=(1333, 736),
        flip=False,
        transforms=[
            dict(type='Resize', img_scale=(1333, 736), keep_ratio=True),
            dict(
                type='Normalize',
                mean=[123.675, 116.28, 103.53],
                std=[58.395, 57.12, 57.375],
                to_rgb=True),
            dict(type='Pad', size_divisor=32),
            dict(type='ImageToTensor', keys=['img']),
            dict(type='Collect', keys=['img'])
        ])
]
data = dict(
    samples_per_gpu=12,
    workers_per_gpu=4,
    train=dict(
        type='IcdarDataset',
        ann_file='/home/apsisdev/IMPORTANT/totaltext/instances_training.json',
        img_prefix='/home/apsisdev/IMPORTANT/totaltext/imgs',
        pipeline=[
            dict(type='LoadImageFromFile'),
            dict(
                type='LoadTextAnnotations',
                with_bbox=True,
                with_mask=True,
                poly2mask=False),
            dict(
                type='ColorJitter',
                brightness=0.12549019607843137,
                saturation=0.5),
            dict(
                type='Normalize',
                mean=[123.675, 116.28, 103.53],
                std=[58.395, 57.12, 57.375],
                to_rgb=True),
            dict(
                type='RandomCropPolyInstances',
                instance_key='gt_masks',
                crop_ratio=0.65,
                min_side_ratio=0.3),
            dict(
                type='RandomRotatePolyInstances',
                rotate_ratio=0.5,
                max_angle=20,
                pad_with_fixed_color=False),
            dict(
                type='ScaleAspectJitter',
                img_scale=[(3000, 736)],
                ratio_range=(0.7, 1.3),
                aspect_ratio_range=(0.9, 1.1),
                multiscale_mode='value',
                long_size_bound=800,
                short_size_bound=480,
                resize_type='long_short_bound',
                keep_ratio=False),
            dict(type='SquareResizePad', target_size=800, pad_ratio=0.6),
            dict(type='RandomFlip', flip_ratio=0.5, direction='horizontal'),
            dict(type='TextSnakeTargets'),
            dict(type='Pad', size_divisor=32),
            dict(
                type='CustomFormatBundle',
                keys=[
                    'gt_text_mask', 'gt_center_region_mask', 'gt_mask',
                    'gt_radius_map', 'gt_sin_map', 'gt_cos_map'
                ],
                visualize=dict(flag=False, boundary_key='gt_text_mask')),
            dict(
                type='Collect',
                keys=[
                    'img', 'gt_text_mask', 'gt_center_region_mask', 'gt_mask',
                    'gt_radius_map', 'gt_sin_map', 'gt_cos_map'
                ])
        ]),
    val=dict(
        type='IcdarDataset',
        ann_file='/home/apsisdev/IMPORTANT/totaltext/instances_test.json',
        img_prefix='/home/apsisdev/IMPORTANT/totaltext/imgs',
        pipeline=[
            dict(type='LoadImageFromFile'),
            dict(
                type='MultiScaleFlipAug',
                img_scale=(1333, 736),
                flip=False,
                transforms=[
                    dict(
                        type='Resize', img_scale=(1333, 736), keep_ratio=True),
                    dict(
                        type='Normalize',
                        mean=[123.675, 116.28, 103.53],
                        std=[58.395, 57.12, 57.375],
                        to_rgb=True),
                    dict(type='Pad', size_divisor=32),
                    dict(type='ImageToTensor', keys=['img']),
                    dict(type='Collect', keys=['img'])
                ])
        ]),
    test=dict(
        type='IcdarDataset',
        ann_file='/home/apsisdev/IMPORTANT/totaltext/instances_test.json',
        img_prefix='/home/apsisdev/IMPORTANT/totaltext/imgs',
        pipeline=[
            dict(type='LoadImageFromFile'),
            dict(
                type='MultiScaleFlipAug',
                img_scale=(1333, 736),
                flip=False,
                transforms=[
                    dict(
                        type='Resize', img_scale=(1333, 736), keep_ratio=True),
                    dict(
                        type='Normalize',
                        mean=[123.675, 116.28, 103.53],
                        std=[58.395, 57.12, 57.375],
                        to_rgb=True),
                    dict(type='Pad', size_divisor=32),
                    dict(type='ImageToTensor', keys=['img']),
                    dict(type='Collect', keys=['img'])
                ])
        ]))
evaluation = dict(interval=10, metric='hmean-iou')
work_dir = '../outputs/detection_model'
seed = 0
gpu_ids = range(0, 1)

loading annotations into memory...
Done (t=0.00s)
creating index...
index created!
2021-06-30 13:21:39,966 - mmdet - INFO - load model from: torchvision://resnet50
2021-06-30 13:21:39,966 - mmdet - INFO - Use load_from_torchvision loader
2021-06-30 13:21:40,021 - mmdet - WARNING - The model and loaded state dict do not match exactly

unexpected key in source state_dict: fc.weight, fc.bias

-------------->>>>>>>>> Training FILE------------------->>>>>>>>>
 /home/apsisdev/IMPORTANT/mmocr/configs/textdet/textsnake/textsnake_r50_fpn_unet_1200e_ctw1500.py

/home/apsisdev/IMPORTANT/mmocr/mmocr/apis/train.py:74: UserWarning: config is now expected to have a `runner` section, please set `runner` in your config.
  warnings.warn(
loading annotations into memory...
Done (t=0.00s)
creating index...
index created!
2021-06-30 13:21:42,119 - mmocr - INFO - Start running, host: apsisdev@ML, work_dir: /home/apsisdev/IMPORTANT/mmocr/outputs/detection_model
2021-06-30 13:21:42,119 - mmocr - INFO - workflow: [('train', 1)], max: 100 epochs
2021-06-30 13:21:46,941 - mmocr - INFO - Epoch [1][1/6] lr: 1.250e-04, eta: 0:46:08, time: 4.623, data_time: 4.109, memory: 19933, loss_text: 0.6932, loss_center: 0.6932, loss_radius: 0.5000, loss_sin: 0.3074, loss_cos: 0.6727, loss: 2.8666
2021-06-30 13:21:47,641 - mmocr - INFO - Epoch [1][2/6] lr: 1.250e-04, eta: 0:26:54, time: 0.776, data_time: 0.207, memory: 20230, loss_text: 0.6932, loss_center: 0.6932, loss_radius: 0.5000, loss_sin: 0.0280, loss_cos: 0.0010, loss: 1.9155
2021-06-30 13:21:48,416 - mmocr - INFO - Epoch [1][3/6] lr: 1.250e-04, eta: 0:20:28, time: 0.775, data_time: 0.213, memory: 20230, loss_text: 0.6931, loss_center: 0.6932, loss_radius: 0.5000, loss_sin: 0.0067, loss_cos: 0.0002, loss: 1.8932
2021-06-30 13:21:49,187 - mmocr - INFO - Epoch [1][4/6] lr: 1.250e-04, eta: 0:17:14, time: 0.771, data_time: 0.213, memory: 20230, loss_text: 0.6932, loss_center: 0.6932, loss_radius: 0.5000, loss_sin: 0.0135, loss_cos: 0.0014, loss: 1.9012
2021-06-30 13:21:49,956 - mmocr - INFO - Epoch [1][5/6] lr: 1.250e-04, eta: 0:15:17, time: 0.769, data_time: 0.215, memory: 20230, loss_text: 0.6932, loss_center: 0.6932, loss_radius: 0.5000, loss_sin: 0.0053, loss_cos: 0.0001, loss: 1.8918
2021-06-30 13:21:50,714 - mmocr - INFO - Epoch [1][6/6] lr: 1.250e-04, eta: 0:13:58, time: 0.758, data_time: 0.214, memory: 20230, loss_text: 0.6932, loss_center: 0.6931, loss_radius: 0.5000, loss_sin: 0.0158, loss_cos: 0.0003, loss: 1.9025
2021-06-30 13:21:50,780 - mmocr - INFO - Saving checkpoint at 1 epochs
2021-06-30 13:21:56,002 - mmocr - INFO - Epoch [2][1/6] lr: 1.239e-04, eta: 0:18:35, time: 4.693, data_time: 4.122, memory: 20230, loss_text: 0.6931, loss_center: 0.6931, loss_radius: 0.5000, loss_sin: 0.0105, loss_cos: 0.0003, loss: 1.8970
2021-06-30 13:21:56,683 - mmocr - INFO - Epoch [2][2/6] lr: 1.239e-04, eta: 0:17:04, time: 0.681, data_time: 0.131, memory: 20230, loss_text: 0.6931, loss_center: 0.6931, loss_radius: 0.5000, loss_sin: 0.0049, loss_cos: 0.0000, loss: 1.8911
2021-06-30 13:21:57,363 - mmocr - INFO - Epoch [2][3/6] lr: 1.239e-04, eta: 0:15:53, time: 0.681, data_time: 0.130, memory: 20230, loss_text: 0.6930, loss_center: 0.6930, loss_radius: 0.5000, loss_sin: 0.0024, loss_cos: 0.0000, loss: 1.8885
2021-06-30 13:21:58,047 - mmocr - INFO - Epoch [2][4/6] lr: 1.239e-04, eta: 0:14:57, time: 0.684, data_time: 0.130, memory: 20230, loss_text: 0.6930, loss_center: 0.6930, loss_radius: 0.5000, loss_sin: 0.0157, loss_cos: 0.0003, loss: 1.9020
2021-06-30 13:21:58,909 - mmocr - INFO - Epoch [2][5/6] lr: 1.239e-04, eta: 0:14:20, time: 0.862, data_time: 0.304, memory: 20230, loss_text: 0.6929, loss_center: 0.6929, loss_radius: 0.5000, loss_sin: 0.0301, loss_cos: 0.0009, loss: 1.9169
2021-06-30 13:21:59,587 - mmocr - INFO - Epoch [2][6/6] lr: 1.239e-04, eta: 0:13:40, time: 0.678, data_time: 0.130, memory: 20230, loss_text: 0.6929, loss_center: 0.6929, loss_radius: 0.5000, loss_sin: 0.0297, loss_cos: 0.0016, loss: 1.9172
2021-06-30 13:21:59,657 - mmocr - INFO - Saving checkpoint at 2 epochs
2021-06-30 13:22:04,842 - mmocr - INFO - Epoch [3][1/6] lr: 1.227e-04, eta: 0:16:06, time: 4.655, data_time: 4.064, memory: 20230, loss_text: 0.6928, loss_center: 0.6928, loss_radius: 0.5000, loss_sin: 0.0117, loss_cos: 0.0007, loss: 1.8980
2021-06-30 13:22:05,513 - mmocr - INFO - Epoch [3][2/6] lr: 1.227e-04, eta: 0:15:23, time: 0.671, data_time: 0.131, memory: 20230, loss_text: 0.6929, loss_center: 0.6928, loss_radius: 0.5000, loss_sin: 0.0115, loss_cos: 0.0002, loss: 1.8974
2021-06-30 13:22:06,275 - mmocr - INFO - Epoch [3][3/6] lr: 1.227e-04, eta: 0:14:50, time: 0.763, data_time: 0.219, memory: 20230, loss_text: 0.6928, loss_center: 0.6928, loss_radius: 0.5000, loss_sin: 0.0050, loss_cos: 0.0000, loss: 1.8906
2021-06-30 13:22:06,943 - mmocr - INFO - Epoch [3][4/6] lr: 1.227e-04, eta: 0:14:17, time: 0.668, data_time: 0.130, memory: 20230, loss_text: 0.6928, loss_center: 0.6927, loss_radius: 0.5000, loss_sin: 0.0110, loss_cos: 0.0002, loss: 1.8967
2021-06-30 13:22:07,809 - mmocr - INFO - Epoch [3][5/6] lr: 1.227e-04, eta: 0:13:55, time: 0.866, data_time: 0.307, memory: 20230, loss_text: 0.6926, loss_center: 0.6926, loss_radius: 0.5000, loss_sin: 0.0216, loss_cos: 0.0032, loss: 1.9100
2021-06-30 13:22:08,479 - mmocr - INFO - Epoch [3][6/6] lr: 1.227e-04, eta: 0:13:29, time: 0.670, data_time: 0.130, memory: 20230, loss_text: 0.6926, loss_center: 0.6926, loss_radius: 0.5000, loss_sin: 0.0009, loss_cos: 0.0000, loss: 1.8861
2021-06-30 13:22:08,550 - mmocr - INFO - Saving checkpoint at 3 epochs
2021-06-30 13:22:13,483 - mmocr - INFO - Epoch [4][1/6] lr: 1.216e-04, eta: 0:14:59, time: 4.376, data_time: 3.809, memory: 20230, loss_text: 0.6925, loss_center: 0.6925, loss_radius: 0.5000, loss_sin: 0.0132, loss_cos: 0.0005, loss: 1.8986
2021-06-30 13:22:14,296 - mmocr - INFO - Epoch [4][2/6] lr: 1.216e-04, eta: 0:14:36, time: 0.813, data_time: 0.255, memory: 20230, loss_text: 0.6924, loss_center: 0.6924, loss_radius: 0.5000, loss_sin: 0.0053, loss_cos: 0.0001, loss: 1.8903
2021-06-30 13:22:15,142 - mmocr - INFO - Epoch [4][3/6] lr: 1.216e-04, eta: 0:14:16, time: 0.847, data_time: 0.298, memory: 20230, loss_text: 0.6924, loss_center: 0.6924, loss_radius: 0.5000, loss_sin: 0.0304, loss_cos: 0.0008, loss: 1.9161
2021-06-30 13:22:15,831 - mmocr - INFO - Epoch [4][4/6] lr: 1.216e-04, eta: 0:13:54, time: 0.689, data_time: 0.130, memory: 20230, loss_text: 0.6923, loss_center: 0.6923, loss_radius: 0.5000, loss_sin: 0.0034, loss_cos: 0.0001, loss: 1.8881
2021-06-30 13:22:16,683 - mmocr - INFO - Epoch [4][5/6] lr: 1.216e-04, eta: 0:13:38, time: 0.851, data_time: 0.296, memory: 20230, loss_text: 0.6923, loss_center: 0.6923, loss_radius: 0.5000, loss_sin: 0.0051, loss_cos: 0.0001, loss: 1.8897
2021-06-30 13:22:17,362 - mmocr - INFO - Epoch [4][6/6] lr: 1.216e-04, eta: 0:13:19, time: 0.679, data_time: 0.130, memory: 20230, loss_text: 0.6922, loss_center: 0.6922, loss_radius: 0.5000, loss_sin: 0.0333, loss_cos: 0.0014, loss: 1.9192
2021-06-30 13:22:17,456 - mmocr - INFO - Saving checkpoint at 4 epochs
2021-06-30 13:22:22,713 - mmocr - INFO - Epoch [5][1/6] lr: 1.205e-04, eta: 0:14:34, time: 4.731, data_time: 4.160, memory: 20230, loss_text: 0.6921, loss_center: 0.6922, loss_radius: 0.5000, loss_sin: 0.0182, loss_cos: 0.0006, loss: 1.9031
2021-06-30 13:22:23,402 - mmocr - INFO - Epoch [5][2/6] lr: 1.205e-04, eta: 0:14:14, time: 0.688, data_time: 0.132, memory: 20230, loss_text: 0.6921, loss_center: 0.6921, loss_radius: 0.5000, loss_sin: 0.0117, loss_cos: 0.0013, loss: 1.8972
2021-06-30 13:22:24,086 - mmocr - INFO - Epoch [5][3/6] lr: 1.205e-04, eta: 0:13:56, time: 0.684, data_time: 0.131, memory: 20230, loss_text: 0.6920, loss_center: 0.6921, loss_radius: 0.5000, loss_sin: 0.0175, loss_cos: 0.0006, loss: 1.9021
2021-06-30 13:22:24,850 - mmocr - INFO - Epoch [5][4/6] lr: 1.205e-04, eta: 0:13:40, time: 0.764, data_time: 0.222, memory: 20230, loss_text: 0.6919, loss_center: 0.6920, loss_radius: 0.5000, loss_sin: 0.0269, loss_cos: 0.0009, loss: 1.9117
2021-06-30 13:22:25,707 - mmocr - INFO - Epoch [5][5/6] lr: 1.205e-04, eta: 0:13:27, time: 0.857, data_time: 0.301, memory: 20230, loss_text: 0.6918, loss_center: 0.6919, loss_radius: 0.5000, loss_sin: 0.0158, loss_cos: 0.0008, loss: 1.9002
2021-06-30 13:22:26,383 - mmocr - INFO - Epoch [5][6/6] lr: 1.205e-04, eta: 0:13:12, time: 0.677, data_time: 0.130, memory: 20230, loss_text: 0.6918, loss_center: 0.6919, loss_radius: 0.5000, loss_sin: 0.0267, loss_cos: 0.0012, loss: 1.9115
2021-06-30 13:22:26,454 - mmocr - INFO - Saving checkpoint at 5 epochs
2021-06-30 13:22:31,640 - mmocr - INFO - Epoch [6][1/6] lr: 1.194e-04, eta: 0:14:11, time: 4.683, data_time: 4.121, memory: 20230, loss_text: 0.6917, loss_center: 0.6919, loss_radius: 0.5000, loss_sin: 0.0082, loss_cos: 0.0005, loss: 1.8922
2021-06-30 13:22:32,327 - mmocr - INFO - Epoch [6][2/6] lr: 1.194e-04, eta: 0:13:55, time: 0.686, data_time: 0.131, memory: 20230, loss_text: 0.6916, loss_center: 0.6917, loss_radius: 0.5000, loss_sin: 0.0175, loss_cos: 0.0005, loss: 1.9013
2021-06-30 13:22:33,087 - mmocr - INFO - Epoch [6][3/6] lr: 1.194e-04, eta: 0:13:41, time: 0.760, data_time: 0.215, memory: 20230, loss_text: 0.6916, loss_center: 0.6917, loss_radius: 0.5000, loss_sin: 0.0223, loss_cos: 0.0014, loss: 1.9070
2021-06-30 13:22:33,772 - mmocr - INFO - Epoch [6][4/6] lr: 1.194e-04, eta: 0:13:27, time: 0.685, data_time: 0.130, memory: 20230, loss_text: 0.6915, loss_center: 0.6916, loss_radius: 0.5000, loss_sin: 0.0215, loss_cos: 0.0006, loss: 1.9052
2021-06-30 13:22:34,618 - mmocr - INFO - Epoch [6][5/6] lr: 1.194e-04, eta: 0:13:16, time: 0.846, data_time: 0.298, memory: 20230, loss_text: 0.6914, loss_center: 0.6916, loss_radius: 0.5000, loss_sin: 0.0080, loss_cos: 0.0002, loss: 1.8911
2021-06-30 13:22:35,298 - mmocr - INFO - Epoch [6][6/6] lr: 1.194e-04, eta: 0:13:03, time: 0.679, data_time: 0.130, memory: 20230, loss_text: 0.6914, loss_center: 0.6915, loss_radius: 0.5000, loss_sin: 0.0208, loss_cos: 0.0007, loss: 1.9043
2021-06-30 13:22:35,369 - mmocr - INFO - Saving checkpoint at 6 epochs
2021-06-30 13:22:40,508 - mmocr - INFO - Epoch [7][1/6] lr: 1.182e-04, eta: 0:13:51, time: 4.630, data_time: 4.057, memory: 20230, loss_text: 0.6912, loss_center: 0.6915, loss_radius: 0.5000, loss_sin: 0.0140, loss_cos: 0.0033, loss: 1.9000
2021-06-30 13:22:41,204 - mmocr - INFO - Epoch [7][2/6] lr: 1.182e-04, eta: 0:13:38, time: 0.696, data_time: 0.131, memory: 20230, loss_text: 0.6912, loss_center: 0.6915, loss_radius: 0.5000, loss_sin: 0.0114, loss_cos: 0.0002, loss: 1.8942
2021-06-30 13:22:41,898 - mmocr - INFO - Epoch [7][3/6] lr: 1.182e-04, eta: 0:13:26, time: 0.694, data_time: 0.131, memory: 20230, loss_text: 0.6911, loss_center: 0.6913, loss_radius: 0.5000, loss_sin: 0.0194, loss_cos: 0.0003, loss: 1.9021
2021-06-30 13:22:42,665 - mmocr - INFO - Epoch [7][4/6] lr: 1.182e-04, eta: 0:13:15, time: 0.768, data_time: 0.219, memory: 20230, loss_text: 0.6911, loss_center: 0.6912, loss_radius: 0.5000, loss_sin: 0.0135, loss_cos: 0.0003, loss: 1.8960
2021-06-30 13:22:43,505 - mmocr - INFO - Epoch [7][5/6] lr: 1.182e-04, eta: 0:13:06, time: 0.840, data_time: 0.292, memory: 20230, loss_text: 0.6910, loss_center: 0.6912, loss_radius: 0.5000, loss_sin: 0.0113, loss_cos: 0.0004, loss: 1.8939
2021-06-30 13:22:44,186 - mmocr - INFO - Epoch [7][6/6] lr: 1.182e-04, eta: 0:12:55, time: 0.680, data_time: 0.131, memory: 20230, loss_text: 0.6909, loss_center: 0.6911, loss_radius: 0.5000, loss_sin: 0.0030, loss_cos: 0.0001, loss: 1.8851
2021-06-30 13:22:44,256 - mmocr - INFO - Saving checkpoint at 7 epochs
2021-06-30 13:22:49,429 - mmocr - INFO - Epoch [8][1/6] lr: 1.171e-04, eta: 0:13:36, time: 4.653, data_time: 4.095, memory: 20230, loss_text: 0.6908, loss_center: 0.6912, loss_radius: 0.5000, loss_sin: 0.0115, loss_cos: 0.0003, loss: 1.8938
2021-06-30 13:22:50,133 - mmocr - INFO - Epoch [8][2/6] lr: 1.171e-04, eta: 0:13:24, time: 0.704, data_time: 0.135, memory: 20230, loss_text: 0.6907, loss_center: 0.6910, loss_radius: 0.5000, loss_sin: 0.0110, loss_cos: 0.0008, loss: 1.8936
2021-06-30 13:22:50,813 - mmocr - INFO - Epoch [8][3/6] lr: 1.171e-04, eta: 0:13:14, time: 0.680, data_time: 0.131, memory: 20230, loss_text: 0.6907, loss_center: 0.6909, loss_radius: 0.5000, loss_sin: 0.0145, loss_cos: 0.0003, loss: 1.8964
2021-06-30 13:22:51,496 - mmocr - INFO - Epoch [8][4/6] lr: 1.171e-04, eta: 0:13:03, time: 0.683, data_time: 0.130, memory: 20230, loss_text: 0.6906, loss_center: 0.6907, loss_radius: 0.5000, loss_sin: 0.0117, loss_cos: 0.0002, loss: 1.8932
2021-06-30 13:22:52,351 - mmocr - INFO - Epoch [8][5/6] lr: 1.171e-04, eta: 0:12:55, time: 0.855, data_time: 0.301, memory: 20230, loss_text: 0.6905, loss_center: 0.6909, loss_radius: 0.5000, loss_sin: 0.0344, loss_cos: 0.0010, loss: 1.9169
2021-06-30 13:22:53,023 - mmocr - INFO - Epoch [8][6/6] lr: 1.171e-04, eta: 0:12:45, time: 0.673, data_time: 0.130, memory: 20230, loss_text: 0.6905, loss_center: 0.6907, loss_radius: 0.5000, loss_sin: 0.0176, loss_cos: 0.0003, loss: 1.8991
2021-06-30 13:22:53,093 - mmocr - INFO - Saving checkpoint at 8 epochs
2021-06-30 13:22:58,334 - mmocr - INFO - Epoch [9][1/6] lr: 1.160e-04, eta: 0:13:21, time: 4.680, data_time: 4.114, memory: 20230, loss_text: 0.6904, loss_center: 0.6909, loss_radius: 0.4999, loss_sin: 0.0072, loss_cos: 0.0001, loss: 1.8885
2021-06-30 13:22:59,036 - mmocr - INFO - Epoch [9][2/6] lr: 1.160e-04, eta: 0:13:11, time: 0.701, data_time: 0.132, memory: 20230, loss_text: 0.6903, loss_center: 0.6907, loss_radius: 0.5000, loss_sin: 0.0193, loss_cos: 0.0004, loss: 1.9007
2021-06-30 13:22:59,727 - mmocr - INFO - Epoch [9][3/6] lr: 1.160e-04, eta: 0:13:02, time: 0.691, data_time: 0.131, memory: 20230, loss_text: 0.6903, loss_center: 0.6906, loss_radius: 0.5000, loss_sin: 0.0028, loss_cos: 0.0001, loss: 1.8836
2021-06-30 13:23:00,408 - mmocr - INFO - Epoch [9][4/6] lr: 1.160e-04, eta: 0:12:52, time: 0.681, data_time: 0.131, memory: 20230, loss_text: 0.6902, loss_center: 0.6907, loss_radius: 0.4999, loss_sin: 0.0104, loss_cos: 0.0002, loss: 1.8914
2021-06-30 13:23:01,184 - mmocr - INFO - Epoch [9][5/6] lr: 1.160e-04, eta: 0:12:45, time: 0.776, data_time: 0.221, memory: 20230, loss_text: 0.6901, loss_center: 0.6906, loss_radius: 0.4999, loss_sin: 0.0041, loss_cos: 0.0000, loss: 1.8849
2021-06-30 13:23:01,972 - mmocr - INFO - Epoch [9][6/6] lr: 1.160e-04, eta: 0:12:37, time: 0.788, data_time: 0.221, memory: 20230, loss_text: 0.6900, loss_center: 0.6904, loss_radius: 0.4999, loss_sin: 0.0142, loss_cos: 0.0016, loss: 1.8962
2021-06-30 13:23:02,041 - mmocr - INFO - Saving checkpoint at 9 epochs
2021-06-30 13:23:07,660 - mmocr - INFO - Epoch [10][1/6]    lr: 1.148e-04, eta: 0:13:09, time: 4.756, data_time: 4.176, memory: 20230, loss_text: 0.6900, loss_center: 0.6902, loss_radius: 0.5000, loss_sin: 0.0257, loss_cos: 0.0028, loss: 1.9087
2021-06-30 13:23:08,343 - mmocr - INFO - Epoch [10][2/6]    lr: 1.148e-04, eta: 0:13:00, time: 0.684, data_time: 0.131, memory: 20230, loss_text: 0.6899, loss_center: 0.6904, loss_radius: 0.4999, loss_sin: 0.0077, loss_cos: 0.0001, loss: 1.8881
2021-06-30 13:23:09,020 - mmocr - INFO - Epoch [10][3/6]    lr: 1.148e-04, eta: 0:12:51, time: 0.676, data_time: 0.130, memory: 20230, loss_text: 0.6899, loss_center: 0.6901, loss_radius: 0.4999, loss_sin: 0.0162, loss_cos: 0.0006, loss: 1.8967
2021-06-30 13:23:09,691 - mmocr - INFO - Epoch [10][4/6]    lr: 1.148e-04, eta: 0:12:43, time: 0.671, data_time: 0.131, memory: 20230, loss_text: 0.6898, loss_center: 0.6903, loss_radius: 0.4999, loss_sin: 0.0231, loss_cos: 0.0006, loss: 1.9037
2021-06-30 13:23:10,526 - mmocr - INFO - Epoch [10][5/6]    lr: 1.148e-04, eta: 0:12:36, time: 0.835, data_time: 0.292, memory: 20230, loss_text: 0.6897, loss_center: 0.6903, loss_radius: 0.4999, loss_sin: 0.0067, loss_cos: 0.0001, loss: 1.8868
2021-06-30 13:23:11,202 - mmocr - INFO - Epoch [10][6/6]    lr: 1.148e-04, eta: 0:12:28, time: 0.676, data_time: 0.130, memory: 20230, loss_text: 0.6896, loss_center: 0.6902, loss_radius: 0.4999, loss_sin: 0.0035, loss_cos: 0.0001, loss: 1.8834
[>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 6/6, 0.0 task/s, elapsed: 3535s, ETA:     0sTraceback (most recent call last):
  File "train_any_mmocr_detector.py", line 154, in <module>
    train_detector(model, datasets, cfg, distributed=False, validate=True)
  File "/home/apsisdev/IMPORTANT/mmocr/mmocr/apis/train.py", line 149, in train_detector
    runner.run(data_loaders, cfg.workflow)
  File "/home/apsisdev/mmcv/mmcv/runner/epoch_based_runner.py", line 125, in run
    epoch_runner(data_loaders[i], **kwargs)
  File "/home/apsisdev/mmcv/mmcv/runner/epoch_based_runner.py", line 54, in train
    self.call_hook('after_train_epoch')
  File "/home/apsisdev/mmcv/mmcv/runner/base_runner.py", line 307, in call_hook
    getattr(hook, fn_name)(self)
  File "/home/apsisdev/anaconda3/envs/ocr/lib/python3.8/site-packages/mmdet/core/evaluation/eval_hooks.py", line 147, in after_train_epoch
    key_score = self.evaluate(runner, results)
  File "/home/apsisdev/anaconda3/envs/ocr/lib/python3.8/site-packages/mmdet/core/evaluation/eval_hooks.py", line 176, in evaluate
    eval_res = self.dataloader.dataset.evaluate(
  File "/home/apsisdev/IMPORTANT/mmocr/mmocr/datasets/icdar_dataset.py", line 149, in evaluate
    eval_results = eval_hmean(
  File "/home/apsisdev/IMPORTANT/mmocr/mmocr/core/evaluation/hmean.py", line 112, in eval_hmean
    assert utils.valid_boundary(texts[0], False)
AssertionError

i have 2 questions :

  1. how do i solve this AssertionError as shown above
  2. this step takes a lot of time : [>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 6/6, 0.0 task/s, elapsed: 3535s, ETA: 0sTraceback (most recent call last): how can i speed it up? seems like this step is working on cpu? i only have 7 image in test/validation set
HolyCrap96 commented 3 years ago

@mobassir94 Hi, are you training textsnake on your own dataset?

mobassir94 commented 3 years ago

yes @HolyCrap96 i was trying to train textsnake on my own dataset which is similar to the toy dataset format provided by mmocr team in their documentation,my dataset is for bengali language