search

FalkonML / falkon

Large-scale, multi-GPU capable, kernel solver
https://falkonml.github.io/falkon/
MIT License
181 stars 22 forks source link

Running out of memory when using KeOps #35

Closed MrHuff closed 3 years ago

MrHuff commented 3 years ago

Hi again,

Thanks for the help these days. Running into:

cudaSafeCall() failed at /data/greyostrich/not-backed-up/nvme00/rhu/miniconda3/envs/new_nnenv/lib/python3.8/site-packages/pykeops/cmake_scripts/script_keops_formula/../../keops/core/mapreduce/GpuConv1D.cu:432 : out of memory

when running FALKON.

Setup: X: 10^9x3 Y: 10^9x1 GaussianKernel with ls=3 penalty=1e-5 GPU: V100 32GB CPU RAM: 180GB 8 CPU processors

Thank you for the help!

Best regards, Robert

Giodiro commented 3 years ago

Hi Robert!

You're using a large dataset :)

It could be a problem with how I estimate the amount of memory KeOps uses (I couldn't figure out the correct way to do this). I added an option named keops_memory_slack in order to control this: what happens is that if data matrices don't fit in memory I split them along the N axis. The size of the split depends on the amount of memory available * some slack. If you decrease keops_memory_slack from 0.7 (default) to 0.5 (for example), it could avoid going OOM!

I hope this works for you, let me know if it doesn't.

p.s. in order to get the new option you need to clone and re-install Falkon!

MrHuff commented 3 years ago

Hi!

Sorry for my delayed response, had to fix many bugs in my code prior to trying this out. :crying_cat_face: Ok great, gonna try it out and see if it can run without any hacking.

Will report back!

MrHuff commented 3 years ago

Tried it now, it works flawlessly! Setting it to 0.25 did the trick!

Thank you so much for the help!