RuntimeError: CUDA error: throwing an instance of 'c10::Error'

[alamayreh]

Hi all,

I'm getting this error while training vid2vid;

RuntimeError: CUDA error: the launch timed out and was terminated
terminate called after throwing an instance of 'c10::Error' 

The error pops up when the script starts the validation phase; here is the entire log; log.txt

I'm training vid2vid on the Kitti dataset using a server with 8 X 12GB Quadro M6000 GPUs. I'm also attaching the config file; ampO1.txt

Any suggestions?

Thanks :)

[o98k-ok]

I have met this error before, maybe resource is not enough. Try to reduce dataset.