I'm training the model yolov5n-0.5 on WIDERFACE dataset. Whenever the train loop hits the validation step, it will just crash. When I use htop to check the memory usage, it showed me that whenever I hit the validation step, it will consume all of my RAM and swap memory (which is 16GB of RAM and 16GB of swap, resulting in 32GB of total memory) and cause memory overflow. Does anyone encountered this problem and what is the suggested fix?
I'm training the model yolov5n-0.5 on WIDERFACE dataset. Whenever the train loop hits the validation step, it will just crash. When I use htop to check the memory usage, it showed me that whenever I hit the validation step, it will consume all of my RAM and swap memory (which is 16GB of RAM and 16GB of swap, resulting in 32GB of total memory) and cause memory overflow. Does anyone encountered this problem and what is the suggested fix?