Multi-Attach io2 - can't mount device with `Invalid argument` error

[obeyda]

I am trying to mount an io2 volume using static provisioning:

pv.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: data-aws-pv
spec:
  capacity:
    storage: 3Ti
  csi:
    driver: ebs.csi.aws.com
    volumeHandle: vol-xxxxxxxxx
    fsType: ext4
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  volumeMode: Block
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
            - key: topology.ebs.csi.aws.com/zone
              operator: In
              values:
                - us-east-2a

pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data-aws-pvc
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 3Ti
  volumeName: data-aws-pv
  volumeMode: Block

pod.yaml

<...>
spec:
  initContainers:
    - command:
        - sh
        - '-c'
        - |
          mkdir -p /app/data
          mount /dev/xvdf /app/data
          echo $(whoami)
          # check files
          echo "Listing files in /app/data ..."
          ls -al /app/data
      image: alpine
      name: mount-block-device
      securityContext:
        capabilities:
          add:
            - SYS_ADMIN
      volumeDevices:
        - devicePath: /dev/xvdf
          name: data-volume
      volumeMounts:
        - mountPath: /app/data
          name: data-mount

  containers:
    - image: alpine
      imagePullPolicy: IfNotPresent
      name: app
      securityContext: {}
      volumeMounts:
        - mountPath: /app/data
          name: data-mount

  restartPolicy: Never
  volumes:
    - name: data-volume
      persistentVolumeClaim:
        claimName: data-aws-pvc
    - emptyDir: {}
      name: data-mount

• I have the io2 volume created and already populated with data.
• Multi-Attach is enabled in us-east-2a

The init container's output says that the mount command is failing with

mount: mounting /dev/xvdf on /app/data failed: Invalid argument

checking with dmesg shows this:

[ 1063.545150] zonefs (nvme2n1) ERROR: Not a zoned block device

My volume:

k8s version: 1.30 aws-ebs-csi-driver: v1.32.0-eksbuild.1 (advanced config only have default storage class enabled)

[ConnorJC3]

/close not-planned

Hi - the EBS CSI Driver appears to be functioning correctly - the volume is created as a multi-attach volume, and is attached successfully to your instance and provided to the pod as a block volume.

The problem as your logs indicate come from the zonefs filesystem - which is not a component we support. You will need to acquire support from the developers, a vendor, or other party willing to provide support for zonefs. The EBS CSI Driver team does not have the resources or expertise to support filesystems or applications running on top of multi-attach volumes. Thus, for multi-attach volumes the EBS CSI team can only support up to the volume being attached and made available as a block device to pods (which happened successfully in this case).

If you have any issues with the EBS CSI Driver (such as issues with volume creation or attach), please re-open this issue or open a new one.

[k8s-ci-robot]

@ConnorJC3: Closing this issue, marking it as "Not Planned".

In response to [this](https://github.com/kubernetes-sigs/aws-ebs-csi-driver/issues/2173#issuecomment-2400482547): >/close not-planned > >Hi - the EBS CSI Driver appears to be functioning correctly - the volume is created as a multi-attach volume, and is attached successfully to your instance and provided to the pod as a block volume. > >The problem as your logs indicate come from the `zonefs` filesystem - which is not a component we support. You will need to acquire support from the developers, a vendor, or other party willing to provide support for `zonefs`. The EBS CSI Driver team does not have the resources or expertise to support filesystems or applications running on top of multi-attach volumes. Thus, for multi-attach volumes the EBS CSI team can only support up to the volume being attached and made available as a block device to pods (which happened successfully in this case). > >If you have any issues with the EBS CSI Driver (such as issues with volume creation or attach), please re-open this issue or open a new one. Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.