Unable to mount GlusterFS PVC with custom user configuration

[thulasi3]

Hi,

I am trying to add additional volume which comes from GlusterFS with default configuration. It works.

But when I try to add custom configuration even with sample file without modifying also, it throws the error - RunContainerError.

Error msg is below Warning Failed 25s kubelet, ip-xx-xx-xx-xx.xx-south-1.compute.internal Error: failed to start container "sftp": Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:402: container init caused \"rootfs_linux.go:58: mounting \\"/var/lib/kubelet/pods/41572663-5c97-11ea-9ad4-067a0ef083a0/volume-subpaths/sftp-json/sftp/0\\" to rootfs \\"/var/lib/docker/overlay2/9dc5cc60e7ffee5bd6df43b291b4ee1c4867ff98aac6ae897cd2029ea39776b4/merged\\" at \\"/var/lib/docker/overlay2/9dc5cc60e7ffee5bd6df43b291b4ee1c4867ff98aac6ae897cd2029ea39776b4/merged/app/config\\" caused \\"not a directory\\"\"": unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type Warning Failed 24s kubelet, ip-x-x-x-x.xx-south-1.compute.internal Error: failed to start container "sftp": Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:402: container init caused \"rootfs_linux.go:58: mounting \\"/var/lib/kubelet/pods/41572663-5c97-11ea-9ad4-067a0ef083a0/volume-subpaths/sftp-json/sftp/0\\" to rootfs \\"/var/lib/docker/overlay2/686ee2e095418bac1d040fc4d069628cd8b705514c1946266acdc9d3e62e10ca/merged\\" at \\"/var/lib/docker/overlay2/686ee2e095418bac1d040fc4d069628cd8b705514c1946266acdc9d3e62e10ca/merged/app/config\\" caused \\"not a directory\\"\"": unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type Normal Pulled 11s (x3 over 25s) kubelet, ip-xx-xx-xx-xx.xx-south-1.compute.internal Container image "emberstack/sftp:2.0.4" already present on machine Normal Created 11s (x3 over 25s) kubelet, ip-xx-xx-xx-xx.xx-south-1.compute.internal Created container Warning Failed 11s kubelet, ip-xx-xx-xx-xx.xx-south-1.compute.internal Error: failed to start container "sftp": Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:402: container init caused \"rootfs_linux.go:58: mounting \\"/var/lib/kubelet/pods/41572663-5c97-11ea-9ad4-067a0ef083a0/volume-subpaths/sftp-json/sftp/0\\" to rootfs \\"/var/lib/docker/overlay2/6eb97b75022fa6e565b40bbef27f82f69d6f4cb167a7b2894d4e944d90abfd60/merged\\" at \\"/var/lib/docker/overlay2/6eb97b75022fa6e565b40bbef27f82f69d6f4cb167a7b2894d4e944d90abfd60/merged/app/config\\" caused \\"not a directory\\"\"": unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type Warning BackOff 3s (x2 over 4s) kubelet, ip-xx-xx-xx.xx.xx-south-1.compute.internal Back-off restarting failed container

[winromulus]

@thulasi3 can you share you values file (but REMOVE ANY CONFIDENTIAL VALUES like passwords, IPs and usernames)? Need to see how you configured the mount

[thulasi3]

Thanks for the quick reply.. Below is the configuration and storage blocks. Issue appears only when I add configuration block. If I remove the complete configuration block, it works without any issue.

configuration: 
  Users:
  - name: user1
    Username: xxxxx
    Password: xxxxx
    Chroot: /file-storage
  Global:
    Directories: ["sftp"]
    Chroot:
      Directory: "/file-storage"
      StartPath: "/file-storage"

storage:
  volumeMounts:
    - name: gluster-volumes
      mountPath: /file-storage

  volumes:
    - name: gluster-volumes
      persistentVolumeClaim:
        claimName: file-storage-pvc

[winromulus]

Hi, Not sure about that configuration (especially in terms of formatting) but here's a snippet from one of my tests:

  configuration:
    {
      "Global": {
        "Chroot": {
          "Directory": "%h",
          "StartPath": "data"
        },
        "Directories": ["data"],
        "Logging": {
          "IgnoreNoIdentificationString": true
        }
      },
      "Users": [
        { "Username": "username",              "Password": "JnpsU7LzWdPDjR9q"  }
      ]      
    }

Please note that the configuration is JSON in the above example (but put in the values file). You can use YAML as well. Also I noticed you have a Users tag with a name under it. That element doesn't exist in configuration. Try using the above and updating your configuration to your needs. After that we can further debug.

[thulasi3]

I tried the below things in values.yaml file. I am getting the same error. Whenever I am trying to add configuration in the values.yaml file, I am getting ContainerRunTime Error. Error msg is same as initial thread. But when I remove configuration from the values.yaml ie default configuration it works with same default username and password.

configuration:
  Global:
    Chroot:
      Directory: "%h"
      StartPath: "data"
    Directories: ["data"]
    Logging: 
      IgnoreNoIdentificationString: true
  Users:
    Username: username
    Password: JnpsU7LzWdPDjR9q

[winromulus]

What version are you on (for the helm chart)?

[thulasi3]

Oops, I was using old chart. With the latest version it works. Thanks a lot for the response.