PVC with type RWM write error: Filename too long

[HristoA]

What happened: On pod shell with mounted RWM PVC i execute:

bash-4.4# echo "test" > text.txt
bash: echo: write error: Filename too long
bash-4.4# ls -la
total 4
drwxr-xr-x    1 root     root             1 Oct 10 08:52 .
drwxr-xr-x    1 root     root          4096 Oct 10 08:16 ..
-rw-r--r--    1 root     root             0 Oct 10 08:52 text.txt

File is created but empty. File is replicated to all pods that use it.

Pod use node:10.16.3-alpine image.
Host nodes are with type ext4

I think that is related in some way to CephFS but for now not found the reason for this error. Thanks

What you expected to happen: To not have limits on files and all to work correctly How to reproduce it (as minimally and precisely as possible): Add PVC with type RWM to some deployment and enter to pod shell.

Anything else we need to know?:

Environment:

• Pharos version (use pharos --version): 2.4.8
• Cloud provider or hardware configuration: Cloud VPS provider, 3 master/ 3 worker nodes, 40GB SSD each node
• OS (e.g. from /etc/os-release; both for the client and for the used nodes): 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
• Others:

cluster.yml:

kontena-storage:
    enabled: true
    data_dir: /home/kontena-storage
    storage:
      use_all_nodes: true
      # It's highly recommended to set directories and/or device_filter,
      # otherwise expanding storage does not work properly (see https://github.com/rook/rook/issues/1957)
      directories:
      - path: /home/ceph
    pool:
      replicated:
        size: 3
    dashboard:
      enabled: true
    filesystem:  # Shared filesystem - ReadWriteMany persistent volumes
      enabled: true
      pool:
        replicated:
          size: 3

[jakolehm]

Does it happen if host nodes are xfs or if storage is configured to use raw devices?

[HristoA]

Hi @jakolehm I did a new fresh installation of Kontena Pharos on the same OS,kernel and Pharos version but point

 directories:
      - path: /volume_xfs

Path /volume_xfs is mount point of XFS volume. After Pharos installation complete I just create PVC with accessMode: ReadWriteMany and new pod (image:alpine) that just month this PVC. In general creating/provisioning of PVC and attaching to container work BUT again the same error popup when try to create a file.

[jakolehm]

It seems that ceph filesystem does work only if raw disks are used as storage in ceph (bluestore backend). This is most likely issue with ceph upstream and might be hard to workaround in kontena-storage.

[HristoA]

I can verify that with real RAW disk (No filesystem and not mounted) CephFS working correct and file is created and writing to files is possible. Take note that parted -l must show your (50GB in this case) disk as:

Error: /dev/sdb: unrecognised disk label Model: HC Volume (scsi)
Disk /dev/sdb: 53.7GB Sector size (logical/physical): 512B/512B Partition Table: unknown Disk Flags:

By default my cloud provider create volumes with default filesystem even if i not mounted automatically to VPS and this lead me to case with:

Model: HC Volume (scsi) Disk /dev/sdb: 53.7GB Sector size (logical/physical): 512B/512B Partition Table: loop Disk Flags:

Number Start End Size File system Flags 1 0.00B 53.7GB 53.7GB xfs

This fact lead me to lose some time before figure out what is the case. I hope with this comment to save some time for people after me. Thanks