search

skypilot-org / skypilot

SkyPilot: Run AI and batch jobs on any infra (Kubernetes or 12+ clouds). Get unified execution, cost savings, and high GPU availability via a simple interface.
https://skypilot.readthedocs.io
Apache License 2.0
6.75k stars 501 forks source link

[k8s] Fix /dev/fuse access on Kubernetes #4108

Open romilbhardwaj opened 2 weeks ago

romilbhardwaj commented 2 weeks ago

Background

When users specify file_mounts using cloud buckets in MOUNT mode, we use a device plugin (smarter-device-manager) to provide the SkyPilot pod access to /dev/fuse without running the container in privileged mode.

This device plugin creates and advertises an extended resource smarter-devices/fuse on each node. Any pod requiring FUSE mounting requests smarter-devices/fuse: 1 resource, and the kubelet running the device plugin mounts /dev/fuse directly in the container.

Problems

  1. Interaction with autoscalers: because we request smarter-devices/fuse: 1 resource in the pod, cluster autoscalers (e.g., GKE) get tripped up by this new resource request.

    • They fail to provision new nodes because none of their configured nodes offers smarter-devices/fuse resource.
    • This resource is a “virtual” resource created by our smarter-device-manager daemonset, but GKE does not have any way to configure the autoscaler to ignore this resource request when making autoscaling decisions.

  2. Reliability: Users have reported issues where the smarter-devices pods would fail silently and mounting stops working.

Current recommendation

Continue using smarter-devices device plugin. For users with autoscaling issues on GKE, recommend using GCSFuse CSI Driver with pod_config. Eventually we should build our own FUSE proxy or find better solutions.

vicaya commented 2 weeks ago

Take a look at https://github.com/pfnet-research/meta-fuse-csi-plugin which is a even better approach without requiring CAP_SYS_ADMIN.