loss_box_reg increasing while training mask rcnn

[ArpanGyawali]

🐛 Describe the bug

I am trying to train maskRcnn model on my custom LVO deteset. My dataset is a single class dataset and some of the image have no annotation in it. The architecture need to learn negative examples as well for proper training as the test data contains both positive and negative lvo cases. I have segmentation annotation in coco format and have registered it using CocoRegistration. When I try to train the maskrcnn model the overall loss decreases but the loss_box_reg increases, and the prediction results bounding box have scores less then 0.1 for every cases (even positive cases). Why is this happening.

How to reproduce this error:

cfg = get_cfg()
# cfg.merge_from_file(model_zoo.get_config_file("COCO-Detection/retinanet_R_101_FPN_3x.yaml"))
cfg.merge_from_file(model_zoo.get_config_file("COCO-InstanceSegmentation/mask_rcnn_R_101_FPN_3x.yaml"))
cfg.DATASETS.TRAIN = ("train",)
cfg.DATASETS.TEST = ()   # no metrics implemented for this dataset
cfg.DATALOADER.NUM_WORKERS = 2
cfg.INPUT.MAX_SIZE_TRAIN = 512         # every training image have size 512
cfg.INPUT.MIN_SIZE_TRAIN = (512,)
cfg.INPUT.MAX_SIZE_TEST = 512
cfg.INPUT.MIN_SIZE_TEST = 512
cfg.INPUT.MASK_FORMAT = "bitmask"
# cfg.MODEL.WEIGHTS = model_zoo.get_checkpoint_url("COCO-Detection/retinanet_R_101_FPN_3x.yaml")  # initialize from model zoo
cfg.MODEL.WEIGHTS = model_zoo.get_checkpoint_url("COCO-InstanceSegmentation/mask_rcnn_R_101_FPN_3x.yaml")
cfg.SOLVER.IMS_PER_BATCH = 2
cfg.SOLVER.BASE_LR = 0.00025
cfg.SOLVER.MAX_ITER = 2000
cfg.SOLVER.CHECKPOINT_PERIOD = 200
cfg.SOLVER.STEPS = []        # do not decay learning rate
cfg.MODEL.ROI_HEADS.BATCH_SIZE_PER_IMAGE = 512   # faster, and good enough for this toy dataset
cfg.MODEL.ROI_HEADS.NUM_CLASSES = 1  # only has one class (ballon)
cfg.DATALOADER.FILTER_EMPTY_ANNOTATIONS = False
cfg.OUTPUT_DIR = out_dir
trainer = DefaultTrainer(cfg) 
trainer.resume_or_load(resume=False)
trainer.train()

My positive and negative dataset sample Annotation example:

 {
      "id": 80,
      "image_id": 180,
      "category_id": 1,
      "segmentation": {
        "counts": [
          large list
        ],
        "size": [512, 512]
      },
      "area": 247.0,
      "bbox": [302.0, 227.0, 24.0, 13.0],
      "iscrowd": 0,
      "attributes": {  "occluded": false
      }},

Issue: Total loss:

Loss_box_reg:

My prediction scoreexample for positive cases: scores: tensor([0.0901, 0.0862, 0.0737, 0.0697, 0.0679, 0.0670, 0.0668, 0.0665, 0.0664, ........])

Help me solve this problem

Versions

Versions: PyTorch version: 2.0.0+cu117 Is debug build: False CUDA used to build PyTorch: 11.7 ROCM used to build PyTorch: N/A

OS: Red Hat Enterprise Linux 9.4 (Plow) (x86_64) GCC version: (GCC) 11.3.0 Clang version: Could not collect CMake version: version 3.28.3 Libc version: glibc-2.34

Python version: 3.9.18 (main, May 16 2024, 00:00:00) [GCC 11.4.1 20231218 (Red Hat 11.4.1-3)] (64-bit runtime) Python platform: Linux-5.14.0-427.18.1.el9_4.x86_64-x86_64-with-glibc2.34

Versions of relevant libraries: [pip3] mypy-extensions==1.0.0 [pip3] numpy==1.26.4 [pip3] torch==2.0.0 [pip3] torchvision==0.15.1 [pip3] triton==2.0.0 [conda] Could not collect

[github-actions[bot]]

You've chosen to report an unexpected problem or bug. Unless you already know the root cause of it, please include details about it by filling the issue template. The following information is missing: "Your Environment";

[github-actions[bot]]

Requested information was not provided in 7 days, so we're closing this issue.

Please open new issue if information becomes available. Otherwise, use github discussions for free-form discussions.