FP16 Training

[michaelschleiss]

When I try to train with fp16 precision I get the following errors:

Traceback (most recent call last): File "tools/train.py", line 153, in main() File "tools/train.py", line 149, in main meta=meta) File "/home/mmdetection/mmdet/apis/train.py", line 128, in train_detector runner.run(data_loaders, cfg.workflow, cfg.total_epochs) File "/home/schlmi/miniconda2/envs/open-mmlab/lib/python3.7/site-packages/mmcv/runner/epoch_based_runner.py", line 122, in run epoch_runner(data_loaders[i], kwargs) File "/home/schlmi/miniconda2/envs/open-mmlab/lib/python3.7/site-packages/mmcv/runner/epoch_based_runner.py", line 32, in train kwargs) File "/home/schlmi/miniconda2/envs/open-mmlab/lib/python3.7/site-packages/mmcv/parallel/data_parallel.py", line 31, in train_step return self.module.train_step(inputs[0], kwargs[0]) File "/home/schlmi/mmdetection/mmdet/models/detectors/base.py", line 236, in train_step losses = self(data) File "/home/schlmi/miniconda2/envs/open-mmlab/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in call result = self.forward(input, *kwargs) File "/home/schlmi/mmdetection/mmdet/core/fp16/decorators.py", line 77, in new_func output = old_func(new_args, new_kwargs) File "/home/schlmi/mmdetection/mmdet/models/detectors/base.py", line 171, in forward return self.forward_train(img, img_metas, kwargs) File "/home/schlmi/research/dudoof/mmdetection/mmdet/models/detectors/single_stage.py", line 93, in forward_train gt_labels, gt_bboxes_ignore) File "/home/schlmi/mmdetection/mmdet/models/dense_heads/base_dense_head.py", line 54, in forward_train losses = self.loss(loss_inputs, gt_bboxes_ignore=gt_bboxes_ignore) File "/home/schlmi/mmdetection/mmdet/core/fp16/decorators.py", line 156, in new_func output = old_func(new_args, new_kwargs) File "/home/schlmi/mmdetection/mmdet/models/dense_heads/gfl_head.py", line 369, in loss num_total_samples=num_total_samples) File "/home/schlmi/mmdetection/mmdet/core/utils/misc.py", line 54, in multi_apply return tuple(map(list, zip(map_results))) File "/home/schlmi/mmdetection/mmdet/models/dense_heads/gfl_head.py", line 266, in loss_single pos_bbox_pred_corners = self.integral(pos_bbox_pred) File "/home/schlmi/miniconda2/envs/open-mmlab/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in call result = self.forward(input, kwargs) File "/home/schlmi/mmdetection/mmdet/models/dense_heads/gfl_head.py", line 55, in forward x = F.linear(x, self.project).reshape(-1, 4) File "/home/schlmi/miniconda2/envs/open-mmlab/lib/python3.7/site-packages/torch/nn/functional.py", line 1372, in linear output = input.matmul(weight.t()) RuntimeError: Expected object of scalar type Float but got scalar type Half for argument #2 'vec' in call to _th_mv

Is there a way to fix this?

[implus]

You can modify self.project to self.project.float(). It will work. Thanks~

[implus]

More specifically, x = F.linear(x, self.project).reshape(-1, 4) -> x = F.linear(x, self.project.float()).reshape(-1, 4)

[michaelschleiss]

Thanks, it worked.