Open letdivedeep opened 3 weeks ago
same issue , could you solve that ?
@Yerma-bit No i was not able to solve it yet. @WongKinYiu @ws6125 any feedback you can provide on this
@letdivedeep still have the same issue. tried yolov9 repo and also Ultralytics yolov9 too, the problem is similiar
@Yerma-bit have you raise this issue with Ultralytics, If so can you point me to the thread ?
@letdivedeep not yet, but i found out that the problem could be with cuda. I have failed with cuda 12.2 but then tried on other machine with Cuda 11.6 and the multi gpu training worked well.
@Yerma-bit That great to know, can you share your environment details
@letdivedeep Sure : NVIDIA-SMI 510.108.03 Driver Version: 510.108.03 CUDA Version: 11.6 and torch 1.13cu11.6 Also, do not forget to clean your dataset cache
Getting an RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one. when running the yolov9 gelan segmentation model on aws p4d.24xlarge instance
below is the error logs :
error_log.txt
I even tried with using 'find_unused_parameters=True' with that the model get stuck