search

clij / clij2

GPU-accelerated image processing for everyone
https://clij.github.io/clij2
Other
48 stars 14 forks source link

Geforce GT 730 Win10 CL_INVALID_COMMAND_QUEUE and CL_OUT_OF_RESOURCES errors #36

Closed chalkie666 closed 3 years ago

chalkie666 commented 3 years ago

Hi Robert,

I'm getting error messages and no output running CLIJ2: Gaussian 3D (sometimes) and Convolve (always) from an imageJ macro:

I'm writing an IJ macro for the reference implementation of the accelerated deconvolution algorithm I mentioned earlier, and trying to do it in CLIJ2 as much as possible. See it here: https://github.com/chalkie666/imagejMacros/blob/master/DeconvolutionDemos/AgardAccellVanCittertGoldReferenceImplementation.ijm

As of now, the script should run fine (testing image data is in the same repository) but, i'm getting very glitchy action on my new computer at home, with windows 10 64, nvidia geforce GT 730 (2GB), nvidia driver installed today, 27.21.14.6589 (driver version from windows task manager), from nvidia download file 465.89-desktop-win10-64bit-international-dch-whql.exe

Maybe its just me, maybe a bad card.... its been around a while.... I will try to test on my little 2 in 1 with integrated intel HD graphics and see if I get the same CL errors.

But maybe you have a clue what the problem might be? Is this card too old?

One last thing.... is the Convolve 2 images together gadget working in real spac3 (slow) or via FFT (fast)? Just out of interest.... I will want to run stuff using the CL-FFT stuff eventually.

cheers

Dan

CLIJ Error: For support please contact the CLIJ2 developers via the forum on https://image.sc or create an issue on https://github.com/clij/clij2/issues . Therefore, please report the complete error message, the code snippet or workflow you were running, an example image if possible and details about your graphics hardware.

net.haesleinhuepf.clij.clearcl.exceptions.OpenCLException: OpenCL error: -36 -> CL_INVALID_COMMAND_QUEUE at net.haesleinhuepf.clij.clearcl.backend.BackendUtils.checkOpenCLError(BackendUtils.java:346) at net.haesleinhuepf.clij.clearcl.backend.jocl.ClearCLBackendJOCL.lambda$waitQueueToFinish$46(ClearCLBackendJOCL.java:1208) at net.haesleinhuepf.clij.clearcl.backend.BackendUtils.checkExceptions(BackendUtils.java:171) at net.haesleinhuepf.clij.clearcl.backend.jocl.ClearCLBackendJOCL.waitQueueToFinish(ClearCLBackendJOCL.java:1207) at net.haesleinhuepf.clij.clearcl.ClearCLQueue.waitToFinish(ClearCLQueue.java:56) at net.haesleinhuepf.clij.clearcl.ClearCLKernel.lambda$run$0(ClearCLKernel.java:497) at net.haesleinhuepf.clij.clearcl.util.ElapsedTime.measure(ElapsedTime.java:97) at net.haesleinhuepf.clij.clearcl.util.ElapsedTime.measure(ElapsedTime.java:64) at net.haesleinhuepf.clij.clearcl.ClearCLKernel.run(ClearCLKernel.java:477) at net.haesleinhuepf.clij.clearcl.ClearCLKernel.run(ClearCLKernel.java:459) at net.haesleinhuepf.clij.clearcl.util.CLKernelExecutor.lambda$enqueue$1(CLKernelExecutor.java:267) at net.haesleinhuepf.clij.clearcl.util.ElapsedTime.measure(ElapsedTime.java:97) at net.haesleinhuepf.clij.clearcl.util.ElapsedTime.measure(ElapsedTime.java:28) at net.haesleinhuepf.clij.clearcl.util.CLKernelExecutor.enqueue(CLKernelExecutor.java:266) at net.haesleinhuepf.clij2.CLIJ2.lambda$executeSubsequently$1(CLIJ2.java:502) at net.haesleinhuepf.clij.clearcl.util.ElapsedTime.measure(ElapsedTime.java:97) at net.haesleinhuepf.clij.clearcl.util.ElapsedTime.measure(ElapsedTime.java:28) at net.haesleinhuepf.clij2.CLIJ2.executeSubsequently(CLIJ2.java:492) at net.haesleinhuepf.clij2.CLIJ2.executeSubsequently(CLIJ2.java:479) at net.haesleinhuepf.clij2.CLIJ2.executeSubsequently(CLIJ2.java:474) at net.haesleinhuepf.clij2.CLIJ2.execute(CLIJ2.java:459) at net.haesleinhuepf.clij2.plugins.Convolve.convolve(Convolve.java:53) at net.haesleinhuepf.clij2.CLIJ2Ops.convolve(CLIJ2Ops.java:7271) at net.haesleinhuepf.clij2.plugins.Convolve.executeCL(Convolve.java:43) at net.haesleinhuepf.clij.macro.CLIJHandler.lambda$handleExtension$0(CLIJHandler.java:163) at net.haesleinhuepf.clij.clearcl.util.ElapsedTime.measure(ElapsedTime.java:97) at net.haesleinhuepf.clij.clearcl.util.ElapsedTime.measure(ElapsedTime.java:28) at net.haesleinhuepf.clij.macro.CLIJHandler.handleExtension(CLIJHandler.java:53) at ij.macro.ExtensionDescriptor.dispatch(ExtensionDescriptor.java:288) at ij.macro.Functions.doExt(Functions.java:4967) at ij.macro.Functions.getStringFunction(Functions.java:278) at ij.macro.Interpreter.getStringTerm(Interpreter.java:1475) at ij.macro.Interpreter.getString(Interpreter.java:1453) at ij.macro.Interpreter.doStatement(Interpreter.java:333) at ij.macro.Interpreter.doBlock(Interpreter.java:712) at ij.macro.Interpreter.doStatement(Interpreter.java:323) at ij.macro.Interpreter.doFor(Interpreter.java:634) at ij.macro.Interpreter.doStatement(Interpreter.java:305) at ij.macro.Interpreter.doStatements(Interpreter.java:264) at ij.macro.Interpreter.run(Interpreter.java:160) at ij.macro.Interpreter.run(Interpreter.java:93) at ij.macro.Interpreter.run(Interpreter.java:104) at ij.plugin.Macro_Runner.runMacro(Macro_Runner.java:161) at ij.IJ.runMacro(IJ.java:153) at ij.IJ.runMacro(IJ.java:142) at net.imagej.legacy.IJ1Helper$3.call(IJ1Helper.java:1148) at net.imagej.legacy.IJ1Helper$3.call(IJ1Helper.java:1144) at net.imagej.legacy.IJ1Helper.runMacroFriendly(IJ1Helper.java:1095) at net.imagej.legacy.IJ1Helper.runMacro(IJ1Helper.java:1144) at net.imagej.legacy.plugin.IJ1MacroEngine.eval(IJ1MacroEngine.java:145) at org.scijava.script.ScriptModule.run(ScriptModule.java:157) at org.scijava.module.ModuleRunner.run(ModuleRunner.java:165) at org.scijava.module.ModuleRunner.call(ModuleRunner.java:124) at org.scijava.module.ModuleRunner.call(ModuleRunner.java:63) at org.scijava.thread.DefaultThreadService.lambda$wrap$2(DefaultThreadService.java:225) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) net.haesleinhuepf.clij.clearcl.exceptions.OpenCLException: OpenCL error: -5 -> CL_OUT_OF_RESOURCES at net.haesleinhuepf.clij.clearcl.backend.BackendUtils.checkOpenCLError(BackendUtils.java:346) at net.haesleinhuepf.clij.clearcl.backend.jocl.ClearCLBackendJOCL.lambda$enqueueReadFromBuffer$24(ClearCLBackendJOCL.java:791) at net.haesleinhuepf.clij.clearcl.backend.BackendUtils.checkExceptions(BackendUtils.java:171) at net.haesleinhuepf.clij.clearcl.backend.jocl.ClearCLBackendJOCL.enqueueReadFromBuffer(ClearCLBackendJOCL.java:790) at net.haesleinhuepf.clij.clearcl.ClearCLBuffer.writeTo(ClearCLBuffer.java:400) at net.haesleinhuepf.clij.clearcl.ClearCLBuffer.writeTo(ClearCLBuffer.java:366) at net.haesleinhuepf.clij.converters.implementations.ClearCLBufferToImagePlusConverter.convert(ClearCLBufferToImagePlusConverter.java:73) at net.haesleinhuepf.clij.converters.implementations.ClearCLBufferToImagePlusConverter.convert(ClearCLBufferToImagePlusConverter.java:26) at net.haesleinhuepf.clij.CLIJ.convert(CLIJ.java:475) at net.haesleinhuepf.clij.CLIJ.show(CLIJ.java:358) at net.haesleinhuepf.clij.macro.CLIJHandler.pullFromGPU(CLIJHandler.java:253) at net.haesleinhuepf.clij2.plugins.Pull.executeCL(Pull.java:24) at net.haesleinhuepf.clij.macro.CLIJHandler.lambda$handleExtension$0(CLIJHandler.java:163) at net.haesleinhuepf.clij.clearcl.util.ElapsedTime.measure(ElapsedTime.java:97) at net.haesleinhuepf.clij.clearcl.util.ElapsedTime.measure(ElapsedTime.java:28) at net.haesleinhuepf.clij.macro.CLIJHandler.handleExtension(CLIJHandler.java:53) at ij.macro.ExtensionDescriptor.dispatch(ExtensionDescriptor.java:288) at ij.macro.Functions.doExt(Functions.java:4967) at ij.macro.Functions.getStringFunction(Functions.java:278) at ij.macro.Interpreter.getStringTerm(Interpreter.java:1475) at ij.macro.Interpreter.getString(Interpreter.java:1453) at ij.macro.Interpreter.doStatement(Interpreter.java:333) at ij.macro.Interpreter.doBlock(Interpreter.java:712) at ij.macro.Interpreter.doStatement(Interpreter.java:323) at ij.macro.Interpreter.doFor(Interpreter.java:634) at ij.macro.Interpreter.doStatement(Interpreter.java:305) at ij.macro.Interpreter.doStatements(Interpreter.java:264) at ij.macro.Interpreter.run(Interpreter.java:160) at ij.macro.Interpreter.run(Interpreter.java:93) at ij.macro.Interpreter.run(Interpreter.java:104) at ij.plugin.Macro_Runner.runMacro(Macro_Runner.java:161) at ij.IJ.runMacro(IJ.java:153) at ij.IJ.runMacro(IJ.java:142) at net.imagej.legacy.IJ1Helper$3.call(IJ1Helper.java:1148) at net.imagej.legacy.IJ1Helper$3.call(IJ1Helper.java:1144) at net.imagej.legacy.IJ1Helper.runMacroFriendly(IJ1Helper.java:1095) at net.imagej.legacy.IJ1Helper.runMacro(IJ1Helper.java:1144) at net.imagej.legacy.plugin.IJ1MacroEngine.eval(IJ1MacroEngine.java:145) at org.scijava.script.ScriptModule.run(ScriptModule.java:157) at org.scijava.module.ModuleRunner.run(ModuleRunner.java:165) at org.scijava.module.ModuleRunner.call(ModuleRunner.java:124) at org.scijava.module.ModuleRunner.call(ModuleRunner.java:63) at org.scijava.thread.DefaultThreadService.lambda$wrap$2(DefaultThreadService.java:225) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

haesleinhuepf commented 3 years ago

Hey @chalkie666 ,

thanks for reporting this issue. I think a reviewer had a similar one...

Maybe its just me, maybe a bad card.... its been around a while....

Yes, I've had similar reports with the same card. It is 7 years old and has a bandwith lower than current integrated GPUs. https://www.techpowerup.com/gpu-specs/geforce-gt-730.c1988

Furthermore, your card has 2 GB of memory limiting the image size in type float (32 bit) < 500 MB. How large is your image data?

Could you try running your code on another GPU? If the issue persists, I'll take a closer look :-)

One last thing.... is the Convolve 2 images together gadget working in real spac3 (slow) or via FFT (fast)? Just out of interest.... I will want to run stuff using the CL-FFT stuff eventually.

CLIJ2_convolve runs in real space. CLIJx_convolveFFT runs in FFT space :-) https://github.com/clij/clij2-fft/blob/master/src/main/java/net/haesleinhuepf/clijx/plugins/ConvolveFFT.java

Cheers, Robert

chalkie666 commented 3 years ago

@haesleinhuepf OK, maybe i need to replace that old gpu.... it was cheap second hand.

i tested on a tiny old 2 in 1. levono miix, 4 GB RAM, with intel atom x5 and intel HD graphics, so totally weak and ancient. Good news, the two clij2 functions i used, 3d gauss blur and convolve both worked, albeit with small kernels. i guess this is a weaker gpu than the geforce gt 730... but maybe not....

i see a silent fail with no error message when using a large convolution kernel on this intel hd graphics machine. sometimes i got an out of resources error, as i expected, but not always. just so you know.

in the same repos as my script are 2 psf images, a large and smaller one for testing.

thanks for the tips so far. if its not worth trouble shooting the old geforce gt 730 card i will get my hands on something newer.... i though i might be able to use it for small tests and algorithm dev .... maybe not.

i dont see the CLIJx_convolveFFT in my menus.... only the clfft RL. but i do see simpleITK things including Wiener Deconv.... this might be exactly what i need for inverse filtering residuals in the algorithm... is this real space or FFT? i see other deconv stuff in simpleITK too... i will have a play with it cheers!

chalkie666 commented 3 years ago

@haesleinhuepf Yikes! The great GPU shortage has really made even cheaper newer GPUs super expensive... no problem, i will fall back to CPU (i have 12 threads at 2.8 GHx) and just use the small test data while i get things working in the code for the algorithm im working on.

I just tried Ext.CLIJx_convolveFFT(gaussGuessGPU, psfGPU, convGuessGPU) and it seems to work (on my CPU openCL), and is about as fast as 3D Gaussian blur with 1,1,1 so that's awesome! I was confused that i couldn't see it in the menus, but its sure there and apparently working! Next i will make use of whatever openCL image Math and intensity normalisation you have implemented, to make my 2 flavours of deconv iterations work... and figure out the regularized Wiener filter from simpleITK... i need that as the magic ingredient for this algorithm, in order for it to converge faster than RL.

thanks R!