search

IDEA-Research / detrex

detrex is a research platform for DETR-based object detection, segmentation, pose estimation and other visual recognition tasks.
https://detrex.readthedocs.io/en/latest/
Apache License 2.0
2.01k stars 209 forks source link

How can i get TP, FP, FN and loss graph? and I wonder the meaning of learning result metrics.json #166

Closed KappyDays closed 1 year ago

KappyDays commented 1 year ago

Hi, I conducted the learning with custom dataset in COCO-format using "dino_swin_large_224_4scale_12ep.py".

1. I want to get "TP, FP, FN" values in the learning process.

2. I would like to check the performance graph of loss for each of Train and Validation Dataset that results after learning.

The picture below is a performance graph that results when learning is conducted in YOLOv5. I want to check a similar graph when I run detrex. image

3. As a result of learning, loss_bbox (0-4), loss_class (0-4), lossgiou (0-4), and total_loss are output, and I want to know what each means.

image

SlongLiu commented 1 year ago

Thanks for the questions.

  1. I have no idea if you want "TP, FP, FN" for predicted classes or objects. You need to set a score threshold to obtain a "TP..". You need to select a matching strategy if you need object's "TP". As the "TP.." are influenced by some thresholds(class scores, IOU scores, ranking ways, etc.), these metrics are not commonly used in our paper.

  2. The detrex code supports tensorboard for loss comparison by default. Please refer to https://github.com/IDEA-Research/detrex/blob/e000f09bd0917229b539272d517eadc54c5bcb26/docs/source/tutorials/Tools.md#tensorboard-log-analysis for more details.

  3. DINO(or other DETR-like models) decoder has 6 decoder layers. we calculate losses after each layer. "loss_0" is the losses after the 1st layer. "_1" for the 2nd layer.. "_4" for the 5th layer. no postfix for the last layer.