I am trying to launch experiments locally with mounted data stored on GCS. Eventually I would like to run the experiments remotely on Kubernetes/VertexAI so I'd like the container to mount the data itself instead of mounting the data on the host machine manually.
I am using prebuilt containers by passing in the image path to the Container executable type. My entrypoint script mounts the GS data buckets with gcsfuse and runs the python training script. However, when running my containers through the xmanager launch script the gcsfuse operation exits with code 1. Running the containers manually with the --privileged option seemed to resolve the issue. However, there currently doesn't exist a xmanager DockerOption to specify --privileged mode.
Is it possible to add such an option? Or is there a different recommended way to mount the data?
I am trying to launch experiments locally with mounted data stored on GCS. Eventually I would like to run the experiments remotely on Kubernetes/VertexAI so I'd like the container to mount the data itself instead of mounting the data on the host machine manually. I am using prebuilt containers by passing in the image path to the Container executable type. My entrypoint script mounts the GS data buckets with gcsfuse and runs the python training script. However, when running my containers through the xmanager launch script the gcsfuse operation exits with code 1. Running the containers manually with the --privileged option seemed to resolve the issue. However, there currently doesn't exist a xmanager DockerOption to specify
--privilegedmode. Is it possible to add such an option? Or is there a different recommended way to mount the data?