Ray Workers Failing Due to Memory Pressure (OOM) in sim_plancnn.sh Script

[JuanchobananoAalto]

I am encountering errors when running the sim_plancnn.sh script. The errors indicate that Ray workers are dying unexpectedly due to a system error, most likely caused by memory pressure (OOM).

Error Messages:

(raylet) A worker died or was killed while executing a task by an unexpected system error...

These messages are followed by details about the specific worker IDs, PIDs, and exit types, which suggest a system error. Additionally, the log mentions:

(raylet) 14 Workers (tasks / actors) killed due to memory pressure (OOM)...

This confirms that the worker failures are related to memory limitations.

Potential Causes:

• The tasks being executed in the script might be exceeding the available memory on the worker nodes.
• The script configuration might be requesting too much parallelism for the available resources.

Steps Taken:

• I have reviewed the Ray documentation on OOM prevention: https://docs.ray.io/en/latest/ray-core/scheduling/ray-oom-prevention.html

Request:

• I would appreciate guidance on how to troubleshoot this issue further.
• Can you suggest ways to optimize the sim_plancnn.sh script to reduce memory usage or adjust Ray worker configurations to prevent OOM errors?
• Are there specific logs from the dead workers that I should check for more detailed information?

[mh0797]

Hi @JuanchobananoAalto , Thanks for the great description of your issue. If you run the simulation with limited memory and face OOM issues, you can reduce the number of workers. This can be done by adding the following line to the script: worker.threads_per_node=X where X is the number of workers that are run in parallel. By default, ray will set this to the number of CPUs on your system. You can set it to a lower value to reduce the memory demand.

Ray does indeed create logs for each worker. You can find them in the experiment folder, e.g. ~/nuplan/exp/exp/sim_plancnn/2024.07.03.08.02.00/logs. You need to check the output of the main job to find the worker that died so that you can check the corresponding log - However, if it just went out of memory, there might be limited information.

I am closing this for now. Let me know if you have any further questions.