Open aaroncaffrey opened 5 years ago
Thanks!
It is possible to restrict GPU memory consumption in your environment. However, constraining the host memory is very challenging in your case. This is because the host memory consumption is mainly due to storing the training data. Partially storing the training data on host memory would be extremely inefficient.
Restricting the memory consumption has no negative impact on the quality of the trained model. So, feel free to restrict memory consumption if longer training time is acceptable :)
I am running several instances of thundersvm GPU based cross-validation and training in parallel and sequentially. However, on occasion, I get a fatal out of memory error. In which case, I have a fail-safe to delay and then rerun the task, which is fine. However, please, if possible, I would like to better understand the issues and also suggest improvements.
Please would it be possible to specify whether this error message refers to the GPU memory, system memory, or which type of memory exactly? Either here and/or within the error messages if it can be both GPU and system memory.
Does the command line '-m memory size' parameter refer to GPU memory or system memory? Is it possible to restrict the usage of both types of memory? If so, could this be a parameter? Does restricting the memory size have any negative consequences apart from longer training times (i.e. can it affect classification metric performance)?
Thank you.