AcademySoftwareFoundation / OpenCue

A render management system you can deploy for visual effects and animation productions.
https://www.opencue.io
Apache License 2.0
832 stars 202 forks source link

[cuebot/rqd] Prevent running frames on Swap memory #1497

Closed DiegoTavares closed 2 months ago

DiegoTavares commented 2 months ago

Improve logic previously implemented to handle Out-of-memory conditions to consider swap usage.

When a host is using more than dispatcher.oom_max_safe_used_physical_memory_threshold if its physical memory and more than dispatcher.oom_max_safe_used_swap_memory_threshold of its swap memory, a logic that kills frames that are relying heavily on swap memory is triggered. This logic will automatically mark killed frames to be retried and possibly increase its parent layer memory requirements if it had been using more memory than initially reserved.