Cupy: cudaErrorIllegalAddress: an illegal memory access was encountered

[janbechtold]

Hi, thanks for your great work. I run into the above mentioned cuda error when running the cupy script on the following normal map. I was wondering whether you can reproduce the issue and have an idea where to start looking. Potentially somewhere where the masks are computed or the diagonal_data_term is updated.

It works with the numpy script, but I want to use the cupy script as it is faster for many of my normal_maps.

Looking forward to hearing from you. Best, Jan

K.txt

[xucao-42]

Hi, I have tried running the cupy script on your data, but I did not encounter any issue. I tested using cuda 11.3 with cupy-cuda11x.

I guess this cuda error are possibly due to the conflict with other threads using cuda (e.g., pytorch) or parallelization running. Have you tried running the script along?

Hope it helps.

[janbechtold]

Hi, thanks for your quick response and testing it on my data. For me the error is reproducible for a number of shapes. I could not figure out yet what they have in common. Independent of the time or machine that I run on. Also there are no other GPU processes running when I start the script.

These are my cuda, cupy versions:

• cuda/10.2.89
• cudnn/10.2_v7.6
• cupy-cuda102 (name) 11.3.0 (version)

So I'll try with a newer cuda version. Thanks for the hint.

[xucao-42]

How about commenting out the two lines? https://github.com/hoshino042/bilateral_normal_integration/blob/a224fb3897ac7c3a59f7e2c97d497962febd6604/bilateral_normal_integration_cupy.py#L16

The two lines are used to avoid out-of-memory for huge normal maps (like >2k). But in your case, the normal maps are several hundreds pixels wide. It is then safe to ignore such explicit memory management.

[janbechtold]

I tried it with the two lines commented out but the error remains. I also tried with rather large images with 5000x3000 pixels but that worked. Even with the cuda memory allocator in place.

[janbechtold]

I updated my environment to use cuda/11.3 and respectively cupy-cuda11x (name) 11.4.0 (version)

Additional remark: I had to update pyvista to 0.37.0 to resolve another error (ImportError: cannot import name 'vtkExtractEdges' from 'vtkmodules.vtkFiltersExtraction')

This resolves my issue. Thanks again for the hint.