search

facebookresearch / unbiased-teacher

PyTorch code for ICLR 2021 paper Unbiased Teacher for Semi-Supervised Object Detection
https://arxiv.org/abs/2102.09480
MIT License
409 stars 84 forks source link

RuntimeError: CUDA error: the launch timed out and was terminated #80

Closed fedyhajali closed 1 year ago

fedyhajali commented 1 year ago

Hi, I am trying to run an experiment with COCO-additional using this yaml configuration file: https://github.com/facebookresearch/unbiased-teacher/blob/main/configs/coco_additional/faster_rcnn_R_50_FPN_focal_cross_coco_unsupW2.yaml

Due to device limitations, I use only 2 GPUs (Tesla K80), each 12GB with 4 Batch-Size.

After the first round of train and evaluation, it stops with this error.

RuntimeError: CUDA error: the launch timed out and was terminated
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
fedyhajali commented 1 year ago

By using a GPU other than the Tesla K80 everything works properly