search

CSBDeep / CSBDeep_fiji

BSD 2-Clause "Simplified" License
11 stars 4 forks source link

GPU usage #33

Closed LauLauThom closed 4 years ago

LauLauThom commented 4 years ago

Hi guys, We are really impressed by the results, however we tried running both CARE and StarDist plugins on GPU in fiji on windows machine, but we observe that the load is exclusively on the CPU and basically nothing on GPU. It seems that the tensorflow configuration is fine (CUDA 9.0, cuDNN 7.6.5) and the console log also states using the CUDA version of tensorflow during execution.

Is there a command to list the devices visible by Tensorflow within Fiji like in python the list_local_device ?

Any other suggestions ?

frauzufall commented 4 years ago

Jepp, the log is misleading there.. Can you please add the TensorFlow dll in Fiji.app/libs/win64 to your PATH? That helped others. I have to add this to the docs.

LauLauThom commented 4 years ago

Hi, thanks for the reply, Adding this directory to the PATH didn't fix it unfortunately.. I tried updating the GPU driver (442.5 now). I also tried updating to TF 1.14.0 with CUDA 10.0 and cuDNN 7.6.5.

And now I have cudaGetDevice() failed. Status: CUDA driver version is insufficient for CUDA runtime version 😢

I dont get it, this configuration works for the python version of CARE on this same machine ><

frauzufall commented 4 years ago

Good to know that it works in python.. Can you try not just adding the directory to PATH but the whole /PATH/TO/Fiji.app/lib/win64/tensorflow_jni.dll?

Does echo %CUDA_PATH% point to CUDA 10.0 and you copied the cuDNN files in there as described here?

LauLauThom commented 4 years ago

Yes I followed the usual CUDA installation, nvcc --version and echo %CUDA_PATH% report the correct version, I dont think the issue is there since the python implementation is working with that.

I have the feeling that the Java implementation is more senstiive to the driver version. Maybe I should give a try the compilation from source, as it is a windows Server OS, but might be tedious.

Otherwise on other windows systems where it runs, we actually noticed that the GPU load stay low but the GPU clock frequency is increasing. So the gain compared to CPU for prediction was not super obvious. Maybe it would for training... Thanks for the help !

uschmidt83 commented 4 years ago

Hi Laurent,

Maybe I should give a try the compilation from source, as it is a windows Server OS, but might be tedious.

From what I've heard (in the past), compiling TensorFlow is not that easy.

Otherwise on other windows systems where it runs, we actually noticed that the GPU load stay low but the GPU clock frequency is increasing. So the gain compared to CPU for prediction was not super obvious. Maybe it would for training...

Yes, you'll mostly notice it when predicting on large (3D) images and especially training.

Best, Uwe