search

open-mmlab / mmdetection

OpenMMLab Detection Toolbox and Benchmark
https://mmdetection.readthedocs.io
Apache License 2.0
29.09k stars 9.38k forks source link

Train Error with AMP #11852

Open caiduoduo12138 opened 2 months ago

caiduoduo12138 commented 2 months ago

Thanks for your error report and we appreciate it a lot.

Checklist

  1. I have searched related issues but cannot get the expected help.
  2. I have read the FAQ documentation but cannot get the expected help.
  3. The bug has not been fixed in the latest version.

Describe the bug I train the dectors with amp. If I use the large image size such as [2560, 1000], the error occurs. If I use [2048, 1000], it seems to be normal.

My log(it contains my eny and the config) 20240713_220159.log

Error Detail 1720880376536

ArthurHartAB commented 1 month ago

Having the same issue