search

clEsperanto / pyclesperanto_prototype

GPU-accelerated bio-image analysis focusing on 3D+t microscopy image data
http://clesperanto.net
BSD 3-Clause "New" or "Revised" License
210 stars 46 forks source link

Meet `MEM_OBJECT_ALLOCATION_FAILURE` with a large medical image #326

Closed xiachenrui closed 11 months ago

xiachenrui commented 11 months ago

Hi, all

I run voronoi_labeling function with a large medical image (20,000 x 30,000) with tthousands of cells, and raise MEM_OBJECT_ALLOCATION_FAILURE error. I am using 2080ti with 11GB memory. Is there any solutions? Should I tile the whole image into smaller parts?

pyopencl._cl.MemoryError: clEnqueueNDRangeKernel failed: MEM_OBJECT_ALLOCATION_FAILURE

Best, Rui

xiachenrui commented 11 months ago

By the way, my server has 8x 2080ti, could I use them to parallel.

StRigaud commented 11 months ago

Hi @xiachenrui

currently, in the prototype, your data are stored in the GPU as float, that means that in memory your image take ~20000*30000*4 b = 2.4Gb.

voronoi_labeling requires x4 or x5 times the memory size of the image you are trying to process. This brings you easily to 12Gb of memory occupancy which hit your max GPU memory.

I would indeed suggest you to tile your image if the problematic does not require you to analyses the entire image in one shot, ... or to use a bigger GPU ....

By the way, my server has 8x 2080ti, could I use them to parallel.

Not directly with pyclesperanto as we do not manage multi-device distribution. However you can still distribute your processing across GPU or HPC by relying on dask and by looking at the demo example

Hope that's help

xiachenrui commented 11 months ago

Thanks for your kind help! It is useful for me.