KeyError: 'labels' raised when inference an image using my own trained model

[Kongsea]

❓ Questions and Help

I write a program to inference an image based on my own trained model as follows:

# -*- coding: utf-8 -*-
import argparse
import cv2

from maskrcnn_benchmark.config import cfg
from predictor import COCODemo

import time

def main():
    parser = argparse.ArgumentParser(description="PyTorch Object Detection Demo")
    parser.add_argument(
        "--config-file",
        default="./configs/coca/bottle_faster_rcnn_R_101_FPN.yaml",
        metavar="FILE",
        help="path to config file",
    )
    parser.add_argument(
        "--image-file",
        default="4e705193-b5ee-4d15-a5ed-8d0296aae1ed.jpg",
        metavar="FILE",
        help="path to image file",
    )
    parser.add_argument(
        "--confidence-threshold",
        type=float,
        default=0.7,
        help="Minimum score for the prediction to be shown",
    )
    parser.add_argument(
        "opts",
        help="Modify model config options using the command-line",
        default=None,
        nargs=argparse.REMAINDER,
    )

    args = parser.parse_args()

    # load config from file and command-line arguments
    cfg.merge_from_file(args.config_file)
    cfg.merge_from_list(args.opts)
    cfg.freeze()

    # prepare object that handles inference plus adds predictions on top of image
    coco_demo = COCODemo(
        cfg,
        confidence_threshold=args.confidence_threshold,
        show_mask_heatmaps=False,
        min_image_size=800,
    )

    start_time = time.time()
    image = cv2.imread(args.image_file)
    composite = coco_demo.run_on_opencv_image(image)
    print("Time: {:.2f} s / img".format(time.time() - start_time))
    cv2.imshow("COCO detections", composite)

if __name__ == "__main__":
    main()

The config file ./configs/coca/bottle_faster_rcnn_R_101_FPN.yaml is as follows:

MODEL:
  META_ARCHITECTURE: "GeneralizedRCNN"
  WEIGHT: "catalog://ImageNetPretrained/MSRA/R-101"
  BACKBONE:
    CONV_BODY: "R-101-FPN"
  RESNETS:
    BACKBONE_OUT_CHANNELS: 256
  RPN:
    USE_FPN: True
    ANCHOR_SIZES: (32, 64, 128, 256, 512)
    ANCHOR_STRIDE: (4, 8, 16, 32, 64)
    ASPECT_RATIOS: (0.5, 1.0, 2.0)
    PRE_NMS_TOP_N_TRAIN: 6000
    PRE_NMS_TOP_N_TEST: 3000
    POST_NMS_TOP_N_TRAIN: 2000
    POST_NMS_TOP_N_TEST: 1000
    FPN_POST_NMS_TOP_N_TRAIN: 1000
    FPN_POST_NMS_TOP_N_TEST: 1000
  ROI_HEADS:
    USE_FPN: True
    BATCH_SIZE_PER_IMAGE: 512
    POSITIVE_FRACTION: 0.25
    NMS: 0.4
    DETECTIONS_PER_IMG: 300
  ROI_BOX_HEAD:
    POOLER_RESOLUTION: 7
    POOLER_SCALES: (0.25, 0.125, 0.0625, 0.03125)
    POOLER_SAMPLING_RATIO: 2
    FEATURE_EXTRACTOR: "FPN2MLPFeatureExtractor"
    PREDICTOR: "FPNPredictor"
    NUM_CLASSES: 3
INPUT:
  MIN_SIZE_TRAIN: (600, 800, 1000)
  MAX_SIZE_TRAIN: 1666
  MIN_SIZE_TEST: 800
  MAX_SIZE_TEST: 1666
DATASETS:
  TRAIN: ("bottle_train_cocostyle",)
  TEST: ("bottle_val_cocostyle",)
DATALOADER:
  SIZE_DIVISIBILITY: 32
SOLVER:
  BASE_LR: 0.005
  WEIGHT_DECAY: 0.0001
  STEPS: (50000, 70000)
  MAX_ITER: 80000
  IMS_PER_BATCH: 4
  CHECKPOINT_PERIOD: 1000
TEST:
  BBOX_AUG:
    ENABLED: True
    H_FLIP: True
    SCALES: (600, 1000)
    MAX_SIZE: 1666
    SCALE_H_FLIP: False
  IMS_PER_BATCH: 4
OUTPUT_DIR: "output/bottle"

However, when I run it to inference an image, it raised the following error:

Traceback (most recent call last):
  File "demo/test_image.py", line 65, in <module>
    main()
  File "demo/test_image.py", line 59, in main
    composite = coco_demo.run_on_opencv_image(image)
  File "maskrcnn-benchmark/demo/predictor.py", line 210, in run_on_opencv_image
    result = self.overlay_boxes(result, top_predictions)
  File "maskrcnn-benchmark/demo/predictor.py", line 294, in overlay_boxes
    labels = predictions.get_field("labels")
  File "maskrcnn-benchmark/maskrcnn_benchmark/structures/bounding_box.py", line 43, in get_field
    return self.extra_fields[field]
KeyError: 'labels'

Could you help me to fix the problem? Any advice will be appreciated. Thank you.

[Darshan2701]

Also what's in the predictor.py Line 52(may be) under the keyword "CATAGORIES= [ '' , '' ]. In any case, your class names must be mentioned under this field in predictor.py which is right below the lin e class COCODemo(object):

[Kongsea]

Thank you. I have checked that and there is no problem with them.

[Darshan2701]

@Kongsea Did you manage to solve the issue?

[jackarailo]

There seems to be a serious bug during inference for segmentation models (MaskPostProcessor) when test time augmentation is turned on. I get the same error even when trying to evaluate on the coco dataset. This is because in the box_head/inference.py PostProcessor module during forward the post processor skips filtering the results if bbox_aug_enabled is True and so there is no "labels" field in the MaskPostProcessor input:

if not self.bbox_aug_enabled:  # If bbox aug is enabled, we will do it later
    boxlist = self.filter_results(boxlist, num_classes)

Also because of skipping filtering there is risk that you can run out of memory if there are too many proposals. My config file is as follows (modified e2e_mask_rcnn_R_50_FPN_1x.yaml to include TTA, 1 image per batch and smaller image size):

MODEL:
  META_ARCHITECTURE: "GeneralizedRCNN"
  WEIGHT: "catalog://ImageNetPretrained/MSRA/R-50"
  BACKBONE:
    CONV_BODY: "R-50-FPN"
  RESNETS:
    BACKBONE_OUT_CHANNELS: 256
  RPN:
    USE_FPN: True
    ANCHOR_STRIDE: (4, 8, 16, 32, 64)
    PRE_NMS_TOP_N_TRAIN: 2000
    PRE_NMS_TOP_N_TEST: 1000
    POST_NMS_TOP_N_TEST: 1000
    FPN_POST_NMS_TOP_N_TEST: 1000
  ROI_HEADS:
    USE_FPN: True
  ROI_BOX_HEAD:
    POOLER_RESOLUTION: 7
    POOLER_SCALES: (0.25, 0.125, 0.0625, 0.03125)
    POOLER_SAMPLING_RATIO: 2
    FEATURE_EXTRACTOR: "FPN2MLPFeatureExtractor"
    PREDICTOR: "FPNPredictor"
  ROI_MASK_HEAD:
    POOLER_SCALES: (0.25, 0.125, 0.0625, 0.03125)
    FEATURE_EXTRACTOR: "MaskRCNNFPNFeatureExtractor"
    PREDICTOR: "MaskRCNNC4Predictor"
    POOLER_RESOLUTION: 14
    POOLER_SAMPLING_RATIO: 2
    RESOLUTION: 28
    SHARE_BOX_FEATURE_EXTRACTOR: False
  MASK_ON: True
DATASETS:
  TRAIN: ("coco_2014_train", "coco_2014_valminusminival")
  TEST: ("coco_2017_val",)
DATALOADER:
  SIZE_DIVISIBILITY: 32
  NUM_WORKERS: 0
SOLVER:
  BASE_LR: 0.02
  WEIGHT_DECAY: 0.0001
  STEPS: (60000, 80000)
  MAX_ITER: 90000
TEST:
   BBOX_AUG:
     ENABLED: True
     H_FLIP: True
     SCALE_H_FLIP: True
   IMS_PER_BATCH: 1
INPUT:
  MIN_SIZE_TEST: 10
  MAX_SIZE_TEST: 20

[thangnx183]

have anyone solve this issue ? i face exactly same problem as above