[Bug] Use --amp for mixed precision accelerated training

LZ-CH commented 3 months ago

Prerequisite

[X] I have searched Issues and Discussions but cannot get the expected help.
[X] I have read the FAQ documentation but cannot get the expected help.
[X] The bug has not been fixed in the latest version (dev-1.x) or latest version (dev-1.0).

Task

I'm using the official example scripts/configs for the officially supported tasks/models/datasets.

Branch

main branch https://github.com/open-mmlab/mmdetection3d

Environment

python 3.9 torch 1.11.0 torchaudio 0.11.0 torchvision 0.12.0

Reproduces the problem - code sample

From EmbodiedScan/embodiedscan/structures/bbox_3d/euler_box3d.py:line 108:


 _, iou3d = box3d_overlap(corners1, corners2, eps=eps)

Reproduces the problem - command or script

CUDA_VISIBLE_DEVICES=0,1,2,3 python -m torch.distributed.launch --nproc_per_node=4 tools/train.py configs/grounding/mv-grounding_8xb12_embodiedscan-vg-9dof.py --work-dir=work_dirs/mv-grounding --launcher="pytorch" --amp

Reproduces the problem - error message

  File "/data1/luojingzhou/projects/EmbodyAI/EmbodiedScan/embodiedscan/models/losses/match_cost.py", line 108, in __ca
ll__
    cost = match_cost(pred_instances=pred_instances_3d,
  File "/data1/luojingzhou/projects/EmbodyAI/EmbodiedScan/embodiedscan/models/losses/match_cost.py", line 108, in __ca
ll__
    overlaps = pred_bboxes.overlaps(pred_bboxes, gt_bboxes)
  File "/data1/luojingzhou/projects/EmbodyAI/EmbodiedScan/embodiedscan/structures/bbox_3d/euler_box3d.py", line 134, i
n overlaps
    overlaps = pred_bboxes.overlaps(pred_bboxes, gt_bboxes)
  File "/data1/luojingzhou/projects/EmbodyAI/EmbodiedScan/embodiedscan/structures/bbox_3d/euler_box3d.py", line 134, i
n overlaps
    _, iou3d = box3d_overlap(corners1, corners2, eps=eps)
  File "/data1/luojingzhou/anaconda3/envs/3DVQA/lib/python3.9/site-packages/pytorch3d/ops/iou_box3d.py", line 164, in 
box3d_overlap
        overlaps = pred_bboxes.overlaps(pred_bboxes, gt_bboxes)
_, iou3d = box3d_overlap(corners1, corners2, eps=eps)
  File "/data1/luojingzhou/projects/EmbodyAI/EmbodiedScan/embodiedscan/structures/bbox_3d/euler_box3d.py", line 134, i
n overlaps
  File "/data1/luojingzhou/anaconda3/envs/3DVQA/lib/python3.9/site-packages/pytorch3d/ops/iou_box3d.py", line 164, in 
box3d_overlap
    _, iou3d = box3d_overlap(corners1, corners2, eps=eps)
  File "/data1/luojingzhou/anaconda3/envs/3DVQA/lib/python3.9/site-packages/pytorch3d/ops/iou_box3d.py", line 164, in 
box3d_overlap
    vol, iou = _box3d_overlap.apply(boxes1, boxes2)    vol, iou = _box3d_overlap.apply(boxes1, boxes2)

  File "/data1/luojingzhou/anaconda3/envs/3DVQA/lib/python3.9/site-packages/pytorch3d/ops/iou_box3d.py", line 103, in 
forward
  File "/data1/luojingzhou/anaconda3/envs/3DVQA/lib/python3.9/site-packages/pytorch3d/ops/iou_box3d.py", line 103, in 
forward
    vol, iou = _C.iou_box3d(boxes1, boxes2)    vol, iou = _C.iou_box3d(boxes1, boxes2)

RuntimeError: RuntimeError: expected scalar type Float but found Half
    vol, iou = _box3d_overlap.apply(boxes1, boxes2)
  File "/data1/luojingzhou/anaconda3/envs/3DVQA/lib/python3.9/site-packages/pytorch3d/ops/iou_box3d.py", line 103, in 
forward
    vol, iou = _C.iou_box3d(boxes1, boxes2)
RuntimeError: expected scalar type Float but found Half
expected scalar type Float but found Half

Additional information

When I try to use the following method to solve it, it can run normally, but during training, the loss will appear as a nan value.

_, iou3d = box3d_overlap(corners1.float(), corners2.float(), eps=eps)
if (corners1.dtype == torch.float16):
     iou3d = iou3d.half()

log:

2024/03/30 17:08:51 - mmengine - INFO - Epoch(train)  [1][1000/1501]  base_lr: 5.0000e-04 lr: 5.0000e-04  eta: 1 day, 16:41:29  time: 8.5409  data_time: 0.6598  memory: 16470  grad_norm: nan  loss: nan  loss_cls: nan  loss_bbox: nan  d0.loss_cls: nan  d0.loss_bbox: nan  d1.loss_cls: nan  d1.loss_bbox: nan  d2.loss_cls: nan  d2.loss_bbox: nan  d3.loss_cls: nan  d3.loss_bbox: nan  d4.loss_cls: nan  d4.loss_bbox: nan

Tai-Wang commented 3 months ago

We do not try mixed precision training with our code. Typically, such training can make the training unstable, and fine-tuning the hyper-parameters is required. Maybe you can try to reduce the learning rate and optimize other related codes to make the training more stable.

Tai-Wang commented 3 months ago

Close due to inactivity. Please feel free to reopen this issue if you have any further questions.

OpenRobotLab / EmbodiedScan