Multi-GPU crash in Caffe Library

[vinaykumarngitub]

Important - read before submitting

Please read the guidelines for contributing before submitting this issue!

Please do not post installation, build, usage, or modeling questions, or other requests for help to Issues. Use the caffe-users list instead. This helps developers maintain a clear, uncluttered, and efficient view of the state of Caffe.

Issue summary

The below is the function used to run training jobs in multiple GPU. void Run(const vector& gpus); But, the above function call don't support passing of the following parameters:

1. Iteration Per Epoch
2. Resume file
3. Minimum epoch
4. Patience To overcome the above said issues, we created one more function call "RunMultipleGPU()" which includes all the said parameters. void RunMultipleGpu(const vector& gpus, std::string Path, int g_IterPerEpoch, const char* resume_file, int iMinEpoch, int iPatience); But the issue is the following When we interrupt while training, it is causing crashing. But, when we complete training: NO ISSUES

   Steps to reproduce

Tried solutions

System configuration

• Operating system: Windows 10
• Compiler: C++
• CUDA version (if applicable): 8
• CUDNN version (if applicable): 7.6.4
• BLAS:
• Python version (if using pycaffe):
• MATLAB version (if using matcaffe):

Issue checklist

• [ ] read the guidelines and removed the first paragraph
• [ ] written a short summary and detailed steps to reproduce
• [ ] explained how solutions to related problems failed (tick if found none)
• [ ] filled system configuration
• [ ] attached relevant logs/config files (tick if not applicable)