search

faustomilletari / VNet

GNU General Public License v3.0
284 stars 123 forks source link

Out of memory #14

Closed Utena closed 7 years ago

Utena commented 7 years ago

Dear Fausto,

As I commented you some days ago, I am very interested in your V-NET article. For this reason, I successfully installed your 3D-Caffe in a cluster with a TITAN X GPU (12 GB of memory). Fortunately, after some errors... all was right! :)

I am currently running your original V-NET code, without any modification. I downloaded the PROMISE12 database, and I selected 2 images for training and only 1 for testing in order to fastly check if all code works. However, it seems that I do not have enough memory to run all the network as the following error appears:

I0129 12:04:15.040880 54018 net.cpp:248] Memory required for data: 4021813252 I0129 12:04:15.041596 54018 solver.cpp:42] Solver scaffolding done. F0129 12:04:17.743491 54018 syncedmem.cpp:51] Check failed: error == cudaSuccess (2 vs. 0) out of memory

I have reduced the batchsize to 1, but the same error appears another time... It is very strange for me, since I have a lot of memory in this GPU, and I only use your V-NET with 3 images... Do you know why this happens?? :(

I will appreciate your help so much.

Thank you!

Jordina

faustomilletari commented 7 years ago

Hello,

Thanks for contacting me.

This error normally is due to the fact that you are not using cudnn for the conv operations.

You just need to recompile caffe with cudnn support. Be sure cmake sees the latest cudnn version.

Regards,

Fausto Milletarì Sent from my iPhone

On 29 Jan 2017, at 06:25, Jordina Torrents Barrena notifications@github.com wrote:

Dear Fausto,

As I commented you some days ago, I am very interested in your V-NET article. For this reason, I successfully installed your 3D-Caffe in a cluster with a TITAN X GPU (12 GB of memory). Fortunately, after some errors... all was right! :)

I am currently running your original V-NET code, without any modification. I downloaded the PROMISE12 database, and I selected 2 images for training and only 1 for testing in order to fastly check if all code works. However, it seems that I do not have enough memory to run all the network as the following error appears:

I0129 12:04:15.040880 54018 net.cpp:248] Memory required for data: 4021813252 I0129 12:04:15.041596 54018 solver.cpp:42] Solver scaffolding done. F0129 12:04:17.743491 54018 syncedmem.cpp:51] Check failed: error == cudaSuccess (2 vs. 0) out of memory

I have reduced the batchsize to 1, but the same error appears another time... It is very strange for me, since I have a lot of memory in this GPU, and I only use your V-NET with 3 images... Do you know why this happens?? :(

I will appreciate your help so much.

Thank you!

Jordina

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

Utena commented 7 years ago

Hello Fausto!

Perfect, it seems that now your V-NET code works on my server! :D

Thank you very much!

Jordina

faustomilletari commented 7 years ago

I’m happy to hear that.

Fausto

On Jan 30, 2017, at 10:30 AM, Jordina Torrents Barrena notifications@github.com wrote:

Hello Fausto!

Perfect, it seems that now your V-NET code works on my server! :D

Thank you very much!

Jordina

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/faustomilletari/VNet/issues/14#issuecomment-276092965, or mute the thread https://github.com/notifications/unsubscribe-auth/AMtsvqex_zuLEh48tgQ9V1QcxReCPHQcks5rXgICgaJpZM4Lww_F.