search

skypilot-org / skypilot

SkyPilot: Run AI and batch jobs on any infra (Kubernetes or 12+ clouds). Get unified execution, cost savings, and high GPU availability via a simple interface.
https://skypilot.readthedocs.io
Apache License 2.0
6.82k stars 512 forks source link

[k8s] Prevent mounting of /dev/shm in pods #4233

Open roclark opened 3 weeks ago

roclark commented 3 weeks ago

I'm attempting to launch jobs on my kubernetes cluster with Skypilot and my cluster auto-mounts memory to /dev/shm to all pods which causes a conflict as Skypilot also auto-mounts data to /dev/shm. I understand with release 0.6.0 of Skypilot we can now override the amount of memory that is allocated to /dev/shm in the Sky config file, but it doesn't appear that there is a way to completely disable mounting this directory with Skypilot. If there isn't already, it would be helpful if there was a way to prevent mounting memory in /dev/shm in pods with Skypilot.

romilbhardwaj commented 3 weeks ago

Welcome to SkyPilot @roclark! Curious, what mechanism do you use to auto-mount /dev/shm? Wondering if there's a way we can auto-detect that and exclude our own /dev/shm mount.

Currently to remove /dev/shm, users need to edit the template we use and remove the volume mount:

https://github.com/skypilot-org/skypilot/blob/1a9e90d55d174cef9166b759e4eafdb944aab172/sky/templates/kubernetes-ray.yml.j2#L389-L390