IBBM / Cascaded-FCN

Source code for the MICCAI 2016 Paper "Automatic Liver and Lesion Segmentation in CT Using Cascaded Fully Convolutional NeuralNetworks and 3D Conditional Random Fields"
Other
304 stars 127 forks source link

Example Docker does not work (crashes) #31

Closed michaelschwier closed 6 years ago

michaelschwier commented 6 years ago

After I couldn't get the notebook to work on my own caffe installation (see #29) I tried running your prepared docker. However this line

# Load network
net1 = caffe.Net(STEP1_DEPLOY_PROTOTXT, STEP1_MODEL_WEIGHTS, caffe.TEST)

throws the following error

...
F0509 19:58:49.805807    24 cudnn_relu_layer.cpp:15] Check failed: status == CUDNN_STATUS_SUCCESS (1 vs. 0)  CUDNN_STATUS_NOT_INITIALIZED

My host system is ubuntu 18.04, NVIDIA Driver Version 390.48, CUDA Version 9.1.85 (if that is any help) caffe is set to CPU mode in your example jupyter notebook

Any help would be appreciated.

michaelschwier commented 6 years ago

However, tha main issue is actually that the model does not seem to work on the current official Caffe version (see #29).

mohamed-ezz commented 6 years ago

Hi @michaelschwier , I think we used a cropping layer which is different from what original Caffe currently has, and it has a slightly different behavior.

The error you get when using the Docker image is most probably some Nvidia Driver and CUDA compatibility issue. The docker image uses an older CUDA, like 6.5 or 7.

Have you tried to install an older Nvidia driver ? for example Nvidia driver 367.57, it is compatible with Titan X, Titan X Pascal and Titan Xp and various other Nvidia GPU models but you have to check if your GPU is supported.

mohamed-ezz commented 6 years ago

@michaelschwier this comment highlights the difference between Jonlong's caffe version (the one we used) and the original caffe version : https://github.com/IBBM/Cascaded-FCN/issues/3#issuecomment-272343778

michaelschwier commented 6 years ago

Hi @mohamed-ezz, thank you for your answer. I gave up on the Nvidia Docker, but managed to get a CPU-only version running buy building Jonlong's caffe version. Thank you for the hint to #3.

mohamed-ezz commented 6 years ago

@michaelschwier So is the problem solved yet ?

michaelschwier commented 6 years ago

@mohamed-ezz I'd consider it solved for me, the CPU version is enough for my purposes. Thanks!