search

microsoft / SoftTeacher

Semi-Supervised Learning, Object Detection, ICCV2021
MIT License
899 stars 123 forks source link

No evaluation seems to be happening on validation set with custom dataset #181

Open sharmalakshay93 opened 2 years ago

sharmalakshay93 commented 2 years ago

Hi, I don't see any results on the validation set. I've created a custom dataset with 1 class. On the stdout, I'm seeing this every 4000 iterations:

2022-03-24 18:24:46,351 - mmdet.ssod - INFO - Iter [12000/720000]       lr: 1.000e-02, eta: 2 days, 21:06:52, time: 0.365, data_time: 0.014, memory: 2890, ema_momentum: 0.9990, sup_loss_rpn_cls: 0.0000, sup_loss_rpn_bbox: 0.0000, sup_loss_cls: 0.0000, sup_acc: 100.0000, sup_loss_bbox: 0.0000, unsup_loss_rpn_cls: 0.0000, unsup_loss_rpn_bbox: 0.0000, unsup_loss_cls: 0.0000, unsup_acc: 100.0000, unsup_loss_bbox: 0.0000, loss: 0.0000
[                                                  ] 0/200, elapsed: 0s, ETA:/vc_data/users/lashar/fhl_spring2022/reinstalled/SoftTeacher/thirdparty/mmdetection/mmdet/datasets/pipelines/formating.py:7: UserWarning: DeprecationWarning: mmdet.datasets.pipelines.formating will be deprecated, please replace it with mmdet.datasets.pipelines.formatting.
  warnings.warn('DeprecationWarning: mmdet.datasets.pipelines.formating will be '
/vc_data/users/lashar/fhl_spring2022/reinstalled/SoftTeacher/thirdparty/mmdetection/mmdet/datasets/pipelines/formating.py:7: UserWarning: DeprecationWarning: mmdet.datasets.pipelines.formating will be deprecated, please replace it with mmdet.datasets.pipelines.formatting.
  warnings.warn('DeprecationWarning: mmdet.datasets.pipelines.formating will be '
/vc_data/users/lashar/fhl_spring2022/reinstalled/SoftTeacher/thirdparty/mmdetection/mmdet/datasets/pipelines/formating.py:7: UserWarning: DeprecationWarning: mmdet.datasets.pipelines.formating will be deprecated, please replace it with mmdet.datasets.pipelines.formatting.
  warnings.warn('DeprecationWarning: mmdet.datasets.pipelines.formating will be '
[                                                  ] 2/200, 0.2 task/s, elapsed: 10s, ETA:  1013s/vc_data/users/lashar/fhl_spring2022/reinstalled/SoftTeacher/thirdparty/mmdetection/mmdet/datasets/pipelines/formating.py:7: UserWarning: DeprecationWarning: mmdet.datasets.pipelines.formating will be deprecated, please replace it with mmdet.datasets.pipelines.formatting.
  warnings.warn('DeprecationWarning: mmdet.datasets.pipelines.formating will be '
[>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 200/200, 15.3 task/s, elapsed: 13s, ETA:     0s2022-03-24 18:25:00,201 - mmdet.ssod - INFO - Evaluating bbox...
Loading and preparing results...
2022-03-24 18:25:00,202 - mmdet.ssod - ERROR - The testing results of the whole dataset is empty.
[                                                  ] 0/200, elapsed: 0s, ETA:/vc_data/users/lashar/fhl_spring2022/reinstalled/SoftTeacher/thirdparty/mmdetection/mmdet/datasets/pipelines/formating.py:7: UserWarning: DeprecationWarning: mmdet.datasets.pipelines.formating will be deprecated, please replace it with mmdet.datasets.pipelines.formatting.
  warnings.warn('DeprecationWarning: mmdet.datasets.pipelines.formating will be '
/vc_data/users/lashar/fhl_spring2022/reinstalled/SoftTeacher/thirdparty/mmdetection/mmdet/datasets/pipelines/formating.py:7: UserWarning: DeprecationWarning: mmdet.datasets.pipelines.formating will be deprecated, please replace it with mmdet.datasets.pipelines.formatting.
  warnings.warn('DeprecationWarning: mmdet.datasets.pipelines.formating will be '
/vc_data/users/lashar/fhl_spring2022/reinstalled/SoftTeacher/thirdparty/mmdetection/mmdet/datasets/pipelines/formating.py:7: UserWarning: DeprecationWarning: mmdet.datasets.pipelines.formating will be deprecated, please replace it with mmdet.datasets.pipelines.formatting.
  warnings.warn('DeprecationWarning: mmdet.datasets.pipelines.formating will be '
[                                                  ] 2/200, 0.2 task/s, elapsed: 10s, ETA:  1013s/vc_data/users/lashar/fhl_spring2022/reinstalled/SoftTeacher/thirdparty/mmdetection/mmdet/datasets/pipelines/formating.py:7: UserWarning: DeprecationWarning: mmdet.datasets.pipelines.formating will be deprecated, please replace it with mmdet.datasets.pipelines.formatting.
  warnings.warn('DeprecationWarning: mmdet.datasets.pipelines.formating will be '
[>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 200/200, 15.2 task/s, elapsed: 13s, ETA:     0s2022-03-24 18:25:14,215 - mmdet.ssod - INFO - Evaluating bbox...
2022-03-24 18:25:14,216 - mmdet.ssod - ERROR - The testing results of the whole dataset is empty.
Loading and preparing results...

2022-03-24 18:25:14,218 - mmdet.ssod - INFO - Exp name: soft_teacher_faster_rcnn_r50_caffe_fpn_coco_full_720k.py
2022-03-24 18:25:14,218 - mmdet.ssod - INFO - Iter(val) [12000]
2022-03-24 18:25:26,778 - mmdet.ssod - INFO - Iter [12050/720000]       lr: 1.000e-02, eta: 2 days, 22:13:09, time: 0.809, data_time: 0.571, memory: 2890, ema_momentum: 0.9990, sup_loss_rpn_cls: 0.0000, sup_loss_rpn_bbox: 0.0000, sup_loss_cls: 0.0000, sup_acc: 100.0000, sup_loss_bbox: 0.0000, unsup_loss_rpn_cls: 0.0000, unsup_loss_rpn_bbox: 0.0000, unsup_loss_cls: 0.0000, unsup_acc: 100.0000, unsup_loss_bbox: 0.0000, loss: 0.0000

In the logs, I'm seeing this:

{"mode": "train", "epoch": 1, "iter": 12000, "lr": 0.01, "memory": 2890, "ema_momentum": 0.999, "data_time": 0.01367, "sup_loss_rpn_cls": 0.0, "sup_loss_rpn_bbox": 0.0, "sup_loss_cls": 0.0, "sup_acc": 100.0, "sup_loss_bbox": 0.0, "unsup_loss_rpn_cls": 0.0, "unsup_loss_rpn_bbox": 0.0, "unsup_loss_cls": 0.0, "unsup_acc": 100.0, "unsup_loss_bbox": 0.0, "loss": 0.0, "time": 0.36499}
{"mode": "val", "epoch": 1, "iter": 12000, "lr": 0.01}
{"mode": "train", "epoch": 1, "iter": 12050, "lr": 0.01, "memory": 2890, "ema_momentum": 0.999, "data_time": 0.57121, "sup_loss_rpn_cls": 0.0, "sup_loss_rpn_bbox": 0.0, "sup_loss_cls": 0.0, "sup_acc": 100.0, "sup_loss_bbox": 0.0, "unsup_loss_rpn_cls": 0.0, "unsup_loss_rpn_bbox": 0.0, "unsup_loss_cls": 0.0, "unsup_acc": 100.0, "unsup_loss_bbox": 0.0, "loss": 0.0, "time": 0.80853}

I'm not sure why it doesn't seem to evaluate. I've changed the config in /vc_data/users/lashar/fhl_spring2022/reinstalled/SoftTeacher/configs/soft_teacher/soft_teacher_faster_rcnn_r50_caffe_fpn_coco_full_720k.py to have val as a dict:

_base_="base.py"

data = dict(
    samples_per_gpu=2,
    workers_per_gpu=2,
    train=dict(

        sup=dict(

            ann_file="data/coco/annotations/instances_train2017.json",
            img_prefix="data/coco/train2017/",

        ),
        unsup=dict(

            ann_file="data/coco/annotations/instances_unlabeled2017.json",
            img_prefix="data/coco/unlabeled2017/",

        ),
    ),
    val=dict(
        ann_file="data/coco/annotations/instances_val2017.json",
        img_prefix="data/coco/val2017/",
    ),
    sampler=dict(
        train=dict(
            sample_ratio=[1, 1],
        )
    ),
)

semi_wrapper = dict(
    train_cfg=dict(
        unsup_weight=2.0,
    )
)

lr_config = dict(step=[120000 * 4, 160000 * 4])
runner = dict(_delete_=True, type="IterBasedRunner", max_iters=180000 * 4)

I choose not to use wandb to visualize results, not sure if this causes any issues.

wandb: (1) Create a W&B account
wandb: (2) Use an existing W&B account
wandb: (3) Don't visualize my results
wandb: Enter your choice: 3

Appreicate any pointers, thanks!

sharmalakshay93 commented 2 years ago

One other detail: some of the images in my train and val set don't have the object of interest. For such images, there is no corresponding entry in "annotations" : but there is an entry in "images" ::

mjehanzaib999 commented 2 years ago

same here

mjehanzaib999 commented 2 years ago

@sharmalakshay93 ensure that you have specified the workflow variable in your config and have included (val,1) in that

sharmalakshay93 commented 2 years ago

@mjehanzaib999 thanks for your response! I'm not sure I understand what you're referring to. Could you pls point me to the line in the config I should change? Did this solve it for you?

mjehanzaib999 commented 2 years ago

@sharmalakshay93 workflow = [('train', 1), ('val',1)] add this to your config if its not already there

mjehanzaib999 commented 2 years ago

@sharmalakshay93 btw what bash command are you running for soft teacher training?

sharmalakshay93 commented 2 years ago

Thanks, I've been running bash tools/dist_train.sh configs/soft_teacher/soft_teacher_faster_rcnn_r50_caffe_fpn_coco_full_720k.py 2

mjehanzaib999 commented 2 years ago

@sharmalakshay93 were you able to run validation?