Get the confusion matrix

[fereshteramez]

Hi. I was wondering how to get the confusion matrix during the training and saving the results (without using test.py). Here is my code to train DetectoRS on a custom dataset:

# Applying DetectoRS (Cascade + ResNet-50)

# Read the data 
with zipfile.ZipFile('/content/sample_data/Modified_Classes Dataset_ObjectDetection.v2i.coco.zip', 'r') as zip_ref:
# with zipfile.ZipFile('/content/sample_data/Modified_Classes.v3i.coco-segmentation.zip', 'r') as zip_ref:
    zip_ref.extractall('/content/sample_data/data/')

from mmcv import Config
from mmdet.apis import set_random_seed

# Make the checkpoints
!mkdir checkpoints
!wget -c https://download.openmmlab.com/mmdetection/v2.0/detectors/detectors_cascade_rcnn_r50_1x_coco/detectors_cascade_rcnn_r50_1x_coco-32a10ba0.pth \
      -O checkpoints/detectors_cascade_rcnn_r50_1x_coco-32a10ba0.pth
config_path='/content/mmdetection/configs/detectors/detectors_cascade_rcnn_r50_1x_coco.py'
load_path='/content/mmdetection/checkpoints/detectors_cascade_rcnn_r50_1x_coco-32a10ba0.pth'

cfg = Config.fromfile(config_path)  
cfg.load_from = load_path

cfg.dataset_type = 'CocoDataset'
cfg.classes = ('Less_Thick','Thick','Thin')
cfg.data_root = '/content/sample_data/data'

cfg.model.roi_head.bbox_head[0]['num_classes']=3
cfg.model.roi_head.bbox_head[1]['num_classes']=3
cfg.model.roi_head.bbox_head[2]['num_classes']=3

cfg.data.test.classes = ('Less_Thick_BN','Thick_BN','Thin_BN')
cfg.data.test.ann_file = '/content/sample_data/data/test/_annotations.coco.json'
cfg.data.test.img_prefix = '/content/sample_data/data/test/'

cfg.data.train.ann_file = '/content/sample_data/data/train/_annotations.coco.json'
cfg.data.train.img_prefix = '/content/sample_data/data/train/'
cfg.data.train.classes = ('Less_Thick','Thick','Thin')

cfg.data.val.ann_file = '/content/sample_data/data/valid/_annotations.coco.json'
cfg.data.val.img_prefix = '/content/sample_data/data/valid/'
cfg.data.val.classes = ('Less_Thick','Thick','Thin')

cfg.work_dir = './tutorial_exps3'

# The original learning rate (LR) is set for 8-GPU training.
# You divide it by 8 since you only use one GPU with Kaggle.
cfg.optimizer.lr = 0.02 / 8
cfg.lr_config.warmup = None
cfg.log_config.interval = 10

# We can set the evaluation interval to reduce the evaluation times
cfg.evaluation.interval = 1
# We can set the checkpoint saving interval to reduce the storage cost
cfg.checkpoint_config.interval = 1
cfg.runner.max_epochs = 5

# Set seed thus the results are more reproducible
cfg.seed = 0
set_random_seed(0, deterministic=False)
cfg.gpu_ids = range(1)

cfg.device='cuda' # added myself since working on cpu

# We can also use tensorboard to log the training process
cfg.log_config.hooks = [
    dict(type='TextLoggerHook'),
    dict(type='TensorboardLoggerHook')]

# We can initialize the logger for training and have a look
# at the final config used for training
print(f'Config:\n{cfg.pretty_text}')

# Train our model (DetectoRS)
import mmcv
import os.path as osp

import os
from mmdet.datasets import build_dataset
from mmdet.models import build_detector
from mmdet.apis import train_detector

# Build dataset
datasets = [build_dataset(cfg.data.train)]

# Build the detector
model = build_detector(cfg.model)

# Add an attribute for visualization convenience
model.CLASSES = datasets[0].CLASSES

# Create work_dir
mmcv.mkdir_or_exist(osp.abspath(cfg.work_dir))
train_detector(model, datasets, cfg, distributed=False, validate=True)

[jbwang1997]

For now, we support a script to calculate the confusion matrix offline. The online confusion matrix calculation is not supported.

[fereshteramez]

I tried the "confusion_matrix.py" to calculate the confusion matrix, however it gives me error concerning the results input which should be in pkl format. Therefore, I tried "test.py" to generate the results.pkl on my own test set as below:

!python tools/test.py /content/mmdetection/configs/detectors/detectors_cascade_rcnn_r50_1x_coco.py /content/mmdetection/tutorial_exps3/latest.pth --out results.pkl

content/mmdetection/mmdet/utils/setup_env.py:39: UserWarning: Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
  f'Setting OMP_NUM_THREADS environment variable for each process '
/content/mmdetection/mmdet/utils/setup_env.py:49: UserWarning: Setting MKL_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
  f'Setting MKL_NUM_THREADS environment variable for each process '
loading annotations into memory...
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/mmcv/utils/registry.py", line 66, in build_from_cfg
    return obj_cls(**args)
  File "/content/mmdetection/mmdet/datasets/custom.py", line 95, in __init__
    self.data_infos = self.load_annotations(local_path)
  File "/content/mmdetection/mmdet/datasets/coco.py", line 72, in load_annotations
    self.coco = COCO(ann_file)
  File "/content/mmdetection/mmdet/datasets/api_wrappers/coco_api.py", line 23, in __init__
    super().__init__(annotation_file=annotation_file)
  File "/usr/local/lib/python3.7/dist-packages/pycocotools/coco.py", line 81, in __init__
    with open(annotation_file, 'r') as f:
FileNotFoundError: [Errno 2] No such file or directory: 'data/coco/annotations/instances_val2017.json'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "tools/test.py", line 275, in <module>
    main()
  File "tools/test.py", line 217, in main
    dataset = build_dataset(cfg.data.test)
  File "/content/mmdetection/mmdet/datasets/builder.py", line 82, in build_dataset
    dataset = build_from_cfg(cfg, DATASETS, default_args)
  File "/usr/local/lib/python3.7/dist-packages/mmcv/utils/registry.py", line 69, in build_from_cfg
    raise type(e)(f'{obj_cls.__name__}: {e}')
FileNotFoundError: CocoDataset: [Errno 2] No such file or directory: 'data/coco/annotations/instances_val2017.json'

Since the code above did not work, I was wondering how I can get my results in the code below in pkl format (I trained the algorithm on my own dataset and I apply it on the test set here):

from mmdet.apis import init_detector, inference_detector, show_result_pyplot
from google.colab.patches import cv2_imshow
from skimage.io import imread_collection

model.cfg = cfg

# path to the images
col_dir = '/content/sample_data/data/test/*.jpg'
#creating a collection with the available images
col = imread_collection(col_dir)

for i in range(len(col)):
    # image=col[i]
    name=col.files[i]
    # name=name.split("test/") # reading the name of the image
    # output_name=name[1]
    image=mmcv.imread(name)
    cv2_imshow(image)
    result = inference_detector(model, image)
    show_result_pyplot(model, image, result)

[wanghonglie]

It seems that you have not set the correct path of the dataset. You can modify this line to the correct path: https://github.com/open-mmlab/mmdetection/blob/e71b499608e9c3ccd4211e7c815fa20eeedf18a2/configs/_base_/datasets/coco_detection.py#L41