about physical memory and virtual memory

[Songpenglei123]

Search before asking

• [X] I have searched the YOLOv5 issues and discussions and found no similar questions.

Question

Why are all the images loaded into the virtual memory during training? I set the virtual memory to about 300g and the physical memory to 128g

Additional

No response

[glenn-jocher]

@Songpenglei123 hello! Thank you for reaching out with your question. 🌟

During training, YOLOv5 utilizes virtual memory to handle large datasets that might not fit entirely into physical RAM. This approach allows the system to swap parts of the data between physical memory and disk space (virtual memory), enabling the processing of larger datasets without running out of physical memory.

The usage of virtual memory as you described seems to be a part of this mechanism, especially when dealing with extensive data or high-resolution images. If you find that the virtual memory usage is excessively high, you might consider:

• Reducing the batch size.
• Using a dataset with smaller image sizes.
• Increasing physical memory, if possible.

These adjustments can help manage memory usage more effectively during training. If you have further questions or need more specific advice, feel free to ask!