Fix PETR transforms and box handling to work with old checkpoints

carlinds commented 3 months ago

Motivation

Current version of the PETR-project does not produce the reported metrics (see https://github.com/open-mmlab/mmdetection3d/issues/2510 for more details). The current state of the project has the following issues:

Incorrect rotation and scaling of the camera pose when doing 3D transforms.
Incorrect normalizing (and denormalizing) of bounding boxes. Current normalizing is not compatible with the checkpoint in the repo, effectively switching length and width, causing large scale errors.

Modification

This PR fixes:

The 3D transforms to work correctly with versions >v1.0.0 (i.e after the coordinate system refactoring).
The normalizing (and denormalizing) of boxes to work with the old checkpoint.

Results from testing the PR on nuScenes validation set (with the old checkpoint):

mAP: 0.3829
mATE: 0.7376
mASE: 0.2702
mAOE: 0.4803
mAVE: 0.8703
mAAE: 0.2040
NDS: 0.4352

Log from the test: 20240305_103838.log

CLAassistant commented 3 months ago

All committers have signed the CLA.

bradyz commented 3 months ago

Wow great catch. Have you tried training with these changes?

TillBeemelmanns commented 2 months ago

I trained PETR with the current code and with the suggestions in this PR. Both versions end up at the same metrics with only marginal differences:

NuScenes metric/pred_instances_3d_NuScenes/NDS: 0.4321
NuScenes metric/pred_instances_3d_NuScenes/mAP: 0.3766

vs.

NuScenes metric/pred_instances_3d_NuScenes/NDS: 0.4353
NuScenes metric/pred_instances_3d_NuScenes/mAP: 0.3771

I would not merge this PR, but rather provide new weights for the current setup.

bradyz commented 2 months ago

here's a little weight conversion script - let me know if i missed something, i only did a quick qualitative check

import torch

ckpt = torch.load('weights/petr.pth')

keys = [
    'pts_bbox_head.reg_branches.0.4.weight',
    'pts_bbox_head.reg_branches.1.4.weight',
    'pts_bbox_head.reg_branches.2.4.weight',
    'pts_bbox_head.reg_branches.3.4.weight',
    'pts_bbox_head.reg_branches.4.4.weight',
    'pts_bbox_head.reg_branches.5.4.weight',
]

for k in keys:
    orig = ckpt['state_dict'][k]

    # weights are (10, 256)
    modified = orig.clone()
    modified[2:3] = orig[3:4]
    modified[3:4] = orig[2:3]

    ckpt['state_dict'][k] = modified

torch.save(ckpt, 'weights/petr_fixed.pth')

carlinds commented 2 months ago

I trained PETR with the current code and with the suggestions in this PR. Both versions end up at the same metrics with only marginal differences:
NuScenes metric/pred_instances_3d_NuScenes/NDS: 0.4321
NuScenes metric/pred_instances_3d_NuScenes/mAP: 0.3766 
vs.
NuScenes metric/pred_instances_3d_NuScenes/NDS: 0.4353
NuScenes metric/pred_instances_3d_NuScenes/mAP: 0.3771
I would not merge this PR, but rather provide new weights for the current setup.

I trained PETR with the current code and with the suggestions in this PR. Both versions end up at the same metrics with only marginal differences:
NuScenes metric/pred_instances_3d_NuScenes/NDS: 0.4321
NuScenes metric/pred_instances_3d_NuScenes/mAP: 0.3766 
vs.
NuScenes metric/pred_instances_3d_NuScenes/NDS: 0.4353
NuScenes metric/pred_instances_3d_NuScenes/mAP: 0.3771
I would not merge this PR, but rather provide new weights for the current setup.

Providing new weights would also suffice. However, I would still suggest keeping some of the fixes in this request (such as the incorrect rotation, misleading naming and unnecessary conversion between torch and numpy).

open-mmlab / mmdetection3d

Fix PETR transforms and box handling to work with old checkpoints #2919

Motivation

Modification