search

SynologyOpenSource / synology-csi

Apache License 2.0
499 stars 107 forks source link

"volume" is apparently in use by the system; will not make a filesystem here! #54

Closed salzig closed 11 months ago

salzig commented 1 year ago

Synology-CSI is install using Helm, current 1.1.2 release.

I'm trying to deploy the Prometheus-Community/Prometheus Chart with the following configuration in regards to storage:

server:
      persistentVolume:
        size: 320Gi
        storageClass: synology-csi-retain

Sadly the container never comes to life, cause the volume mount fails:

MountVolume.MountDevice failed for volume "pvc-30b4c5c6-c7e8-4841-9da2-0164ff16107c" : rpc error: code = Internal desc = format of disk "/dev/disk/by-path/ip-<target>:3260-iscsi-iqn.2000-01.com.synology:storage.pvc-30b4c5c6-c7e8-4841-9da2-0164ff16107c-lun-1" failed: type:("ext4") target:("/var/lib/kubelet/plugins/kubernetes.io/csi/csi.san.synology.com/29266ca212fe6bef686188c7d8825cf302d27c7cb96c09f019cef5c9fc84cedb/globalmount") options:("rw,defaults") errcode:(exit status 1) output:(mke2fs 1.46.5 (30-Dec-2021)
/dev/disk/by-path/ip-<target>:3260-iscsi-iqn.2000-01.com.synology:storage.pvc-30b4c5c6-c7e8-4841-9da2-0164ff16107c-lun-1 is apparently in use by the system; will not make a filesystem here!
)

on the host machine I can see that the presented path is a link to /dev/disk/sda

ls -la /dev/disk/by-path/ip-<target>:3260-iscsi-iqn.2000-01.com.synology:storage.pvc-30b4c5c6-c7e8-4841-9da2-0164ff16107c-lun-1
lrwxrwxrwx 1 root root 9 Jun  7 22:17 /dev/disk/by-path/ip-<target:3260-iscsi-iqn.2000-01.com.synology:storage.pvc-30b4c5c6-c7e8-4841-9da2-0164ff16107c-lun-1 -> ../../sda

and the iscsi disk seems to be attached correctly as /dev/sda. 

Disk /dev/sda: 320 GiB, 343597383680 bytes, 671088640 sectors
Disk model: Storage         
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
chihyuwu commented 1 year ago

Hi @salzig, Could you provide more information for us to investigate? Thank you.

  1. output of mount or mount | grep sda on your machine
  2. Is this LUN new created by CSI?
  3. The cofig of the StorageClass synology-csi-retain
salzig commented 1 year ago

@chihyuwu

  1. Mount doesn't show any mounted volumes, but fdisk -l shows the availability.
  2. Yes.
  3. https://github.com/SynologyOpenSource/synology-csi/blob/main/deploy/helm/templates/storageclass.yaml

Edit: 1) any other beside nvme0n1 and tmpfs/overlay/etc for k8s workloads.

salzig commented 1 year ago

@chihyuwu Looks like a Volume size of 320GB ends up in a timeout. Using a 30GB it worked just fine. Monitoring the Synology SAN Manager I can see that the 320GB thin provisioned volume gets utilised ("used") gigabyte by gigabyte. At around ~80GB k8s fails with a timeout. Looks like the ext4 (default) fs creation takes to long.

chihyuwu commented 1 year ago

Hi @salzig, The timeout issue of formatting filesystem was resolved in PR #49. Please pull the latest CSI: synology/synology-csi:latest (note that version 1.1.2 doesn't include this fix), and configure your StorageClass with additional parameter formatOptions as shown below to skip discarding blocks:

For ext4 fs:

parameters:
  fsType: 'ext4'
  formatOptions: "-E nodiscard"
  protocol: 'iscsi'

For btrfs fs:

parameters:
  fsType: 'btrfs'
  formatOptions: "--nodiscard"
  protocol: 'iscsi'
salzig commented 11 months ago

Sorry for the late response: Yes, this works. Meanwhile I migrated away from iSCSI, as this prevents auto updates on symbology.