search

VolkerH / Lattice_Lightsheet_Deskew_Deconv

Open-source, GPU accelerated code for deskewing and deconvolving lattice light sheet data
Other
22 stars 3 forks source link

Fail gracefully when out of GPU memory #36

Closed VolkerH closed 5 years ago

VolkerH commented 5 years ago

If there is not enough GPU memory to process the data, fail gracefully and proceed to the next data set when running in batch mode

Depending on how large the stack is, GPU memory errors occur

  1. when allocating the FFT plan
  2. during building the tensorflow graph
  3. in pyopencl during the affine transforms

Example for case 1:

019-04-16 10:31:35.730608: F tensorflow/stream_executor/cuda/cuda_fft.cc:444] failed to initialize batched cufft plan with customized allocator: 
Aborted (core dumped)

Example for case 2:

fill in

Example for case 3:

ME Stack5

pyopencl._cl.MemoryError: clEnqueueNDRangeKernel failed: MEM_OBJECT_ALLOCATION_FAILURE
VolkerH commented 5 years ago

While not super-graceful (catches all errors), the changes in this pull request will enable batch processing to continue https://github.com/VolkerH/Lattice_Lightsheet_Deskew_Deconv/pull/39