Accuracy was not improved the custom dataset

[twmht]

I tried training in two phases using your method. In the first phase, I trained the custom dataset from scratch. In the second phase, I finetuned the custom dataset, but the mAP dropped by nearly 15% in the second phase. the validation accuracy is unstable at each epoch in the second stage, Do you have any insights on this?

[ouyanghaodong]

Could you provide more details about the training and datasets?

[twmht]

@ouyanghaodong

the datasets are private.

I can provide the validation curve for reference.

the yellow curve was the first stage and blue curve was the second stage.

Here is my first stage for training

from ultralytics import YOLO
import torch
# Train from Scratch
model = YOLO("cfg/models/v10/yolov10n.yaml")
model = torch.nn.SyncBatchNorm.convert_sync_batchnorm(model)
# Use the model
model.train(data = "crocs.yaml", epochs = 500, scale = 0.5, mixup = 0, copy_paste = 0, device=[0,1,2,3], seed=133, batch=64)

and the second stage we load the checkpoint from first stage

from ultralytics import DEYO
model = DEYO("cfg/models/deyo/deyov1.5n.yaml")
model.load("best.pt")

# Use the model
model.train(data = "crocs.yaml", epochs = 144, lr0 = 0.0001, lrf = 0.0001, weight_decay = 0.0001, optimizer = 'AdamW', warmup_epochs = 0, mosaic = 0, scale = 0.5, mixup = 0, copy_paste = 0, freeze = 23)

I have no idea why i can't use DEYOV1.5 on my dataset, and i want to know why if my outputs are poor.

[ouyanghaodong]

Could you please tell me how many images are in your dataset? During the second phase of training, the AP of the first epoch seems too low, which is not normal.

[twmht]

@ouyanghaodong

just about 3000 images for trainnig, the validation images are about 800 images.

[ouyanghaodong]

Thank you very much. We will attempt to reproduce this result on a dataset of similar size and release a new model to address this issue.

[ouyanghaodong]

We have reproduced this issue on our custom dataset, and it seems that the YOLO evaluation program may not be suitable for DEYO. We found a significant discrepancy between the evaluation results and those obtained using pycocotools. You may choose the best model based on val_cls_loss and use pycocotools to assess the model's performance. We found that our method can align the accuracy of one-to-many branches very well on small datasets.

You can use the following code to evaluate the model's performance:

from pycocotools.coco import COCO
from pycocotools.cocoeval import COCOeval

anno = COCO("gt.json")
pred = anno.loadRes("pred.json")
cocoEval = COCOeval(anno, pred, "bbox")

cocoEval.evaluate()
cocoEval.accumulate()
cocoEval.summarize()

[ouyanghaodong]

We have updated our post-processing code, and the issues have been successfully resolved. Now, you can directly utilize the evaluation tool from ultralytics to measure the performance of your model.

[chenzx2]

常规数据集上没有达到相应的精度，分析了下原因，发现是第一阶段的召回率很低，第二阶段没有发挥作用