Use case pipelines based on EDDL and ECVL libraries. Different tasks (e.g. classification and segmentation) and datasets (e.g. MNIST, ISIC, and PNEUMOTHORAX) are taken into account.
The pipeline in a SLURM environment with two GPUs and EDDL compiled with CUDNN, crashes every time with a different error, which is always one of these:
[CUDA ERROR]: an illegal memory access was encountered (700) raised in set | (check_cuda)
[CUDNN ERROR]: CUDNN_STATUS_EXECUTION_FAILED (8) raised in cudnnConvolutionForward
It seems it was a problem with an imported onnx. From 56cfe97680b6e1f0430ec32fbb67880612e6a710 we changed the onnx and the problem did not appear again.
The pipeline in a SLURM environment with two GPUs and EDDL compiled with CUDNN, crashes every time with a different error, which is always one of these:
[CUDA ERROR]: an illegal memory access was encountered (700) raised in set | (check_cuda)
[CUDNN ERROR]: CUDNN_STATUS_EXECUTION_FAILED (8) raised in cudnnConvolutionForward
EDDL: develop branch (commit 99aa78f9bd30eb6a8fcb27fee38528175b899f5d) ECVL: 0.4.1 CUDA: 11.0 CUDNN: 8.0.4
In a local machine everything works fine. We don't think it's related to the latest ECVL release because it occurs also with the LoadBatch function.