subPath does not work when Anthos Service Mesh is enabled

[songjiaxun]

Symptom

The subPath field does not work when Anthos Service Mesh is enabled.

Root Cause

The root cause is described in https://github.com/GoogleCloudPlatform/gcs-fuse-csi-driver/issues/46.

When the kubelet checks the subpath, it does not timeout in 2 minutes, thus the workload container start up will hang forever.

Workaround

Instead of using subPath, you can use only-dir flag to mount the bucket. It allows you to only mount a sub-folder in the bucket to the mount path.

See the documentation https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/cloud-storage-fuse-csi-driver#mounting-flags for details.

Solution

Similar to https://github.com/GoogleCloudPlatform/gcs-fuse-csi-driver/issues/46, The sidecar feature KEP will be the solution.

The sidecar container issue is tracked on GitHub, and we are waiting for the Kubernetes sidecar container feature to ultimately solve this issue.

[songjiaxun]

This is fixed in GKE 1.29 and above clusters.

To make subPath work with ASM or istio, you need to:

1. Make sure all the nodes are upgraded to 1.29+.
2. Add pod annotationproxy.istio.io/config: '{ "holdApplicationUntilProxyStarts": true }'.

For example:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    gke-gcsfuse/volumes: "true"
    proxy.istio.io/config: '{ "holdApplicationUntilProxyStarts": true }'
  name: gcsfuse-istio-test
  namespace: default
spec:
  containers:
  - command:
    - /bin/sh
    - -c
    - sleep infinite
    image: busybox
    name: busybox
    volumeMounts:
    - mountPath: /data
      name: gcp-gcs-csi-ephemeral
      subPath: test-folder
  serviceAccount: default
  volumes:
  - name: gcp-gcs-csi-ephemeral
     csi:
      driver: gcsfuse.csi.storage.gke.io
      volumeAttributes:
        bucketName: xxx