search

kr-colab / locator

deep learning prediction of geographic location from individual genome sequences
Other
46 stars 18 forks source link

GPU Out of memory #27

Open gaushi opened 2 years ago

gaushi commented 2 years ago

Hi CJ and Andy, I ran into out of memory issue with GPU while running bootstraps. Here is a portion of the log: 2021-11-26 16:42:28.228503: W tensorflow/core/framework/op_kernel.cc:1745] OP_REQUIRES failed at matmul_op_impl.h:681 : RESOURCE_EXHAUSTED: OOM when allocating tensor with shape[387840,256] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc

Could you please suggest a solution? When I run with different seeds (and no bootstraps) Locator works just fine.

Thanks, Best, Gautam

andrewkern commented 2 years ago

Hi @gaushi -- huh that's strange. it seems to me like a potential memory leak in either our code or tensorflow itself. do you have a minimal working example that you could share so that we can debug this?

andrewkern commented 2 years ago

also it would be helpful to know the versions of everything you have installed-- pip list should do it