Cannot `chmod` a read only filesystem

[BabisK]

I get a chmod error: read-only file system when using the CSI driver. This error was not there 3 months ago. The image hash that I see the problem is 71845a27f96b. The image that worked fine before was 15fb01aae1da. Both are tagged the same v0.1.0-alpha.1.

I have tried on k8s 1.16.7 and 1.17.7. Cert-Manager 0.13.1, 0.15 and today 0.16. The only constant is the CSI driver so I guess the error is here.

This is the pod that I'm using, pretty simple:

apiVersion: v1
kind: Pod
metadata:
  name: my-csi-app
  namespace: default
  labels:
    app: my-csi-app
spec:
  containers:
    - name: my-frontend
      image: busybox
      volumeMounts:
      - mountPath: "/tls"
        name: tls
      command: [ "sleep", "1000000" ]
  volumes:
    - name: tls
      csi:
        driver: csi.cert-manager.io
        volumeAttributes:
              csi.cert-manager.io/issuer-name: ca-issuer
              csi.cert-manager.io/issuer-kind: ClusterIssuer
              csi.cert-manager.io/dns-names: my-service.sandbox.svc.cluster.local

Here 's the log from the cert-manager-csi container. I tried to trace the error down to mount.go but I cannot understand who calls chmod. I am not familiar with Go language :(

I0731 17:16:54.923292       1 server.go:129] server: call: /csi.v1.Node/NodePublishVolume
I0731 17:16:54.923332       1 server.go:130] server: request: {"target_path":"/var/lib/kubelet/pods/c192b6d3-ea53-4956-b624-7c2697b10c9a/volumes/kubernetes.io~csi/tls/mount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":1}},"volume_context":{"csi.cert-manager.io/dns-names":"my-service.sandbox.svc.cluster.local","csi.cert-manager.io/issuer-kind":"ClusterIssuer","csi.cert-manager.io/issuer-name":"ca-issuer","csi.storage.k8s.io/ephemeral":"true","csi.storage.k8s.io/pod.name":"my-csi-app","csi.storage.k8s.io/pod.namespace":"default","csi.storage.k8s.io/pod.uid":"c192b6d3-ea53-4956-b624-7c2697b10c9a","csi.storage.k8s.io/serviceAccount.name":"default"},"volume_id":"csi-f2084c47363e5076b4aa1039f57947a57e3520c681faed7f25743b971bba22da"}
I0731 17:16:54.925704       1 nodeserver.go:100] node: created volume: /csi-data-dir/csi-f2084c47363e5076b4aa1039f57947a57e3520c681faed7f25743b971bba22da
I0731 17:16:54.925736       1 nodeserver.go:102] node: creating key/cert pair with cert-manager: /csi-data-dir/csi-f2084c47363e5076b4aa1039f57947a57e3520c681faed7f25743b971bba22da
I0731 17:16:55.454180       1 certmanager.go:80] cert-manager: waiting for CertificateRequest to become ready csi-f2084c47363e5076b4aa1039f57947a57e3520c681faed7f25743b971bba22da
I0731 17:16:55.454321       1 certmanager.go:293] cert-manager: polling CertificateRequest csi-f2084c47363e5076b4aa1039f57947a57e3520c681faed7f25743b971bba22da/default for ready status
I0731 17:16:55.457314       1 certmanager.go:90] cert-manager: metadata written to file /csi-data-dir/csi-f2084c47363e5076b4aa1039f57947a57e3520c681faed7f25743b971bba22da/metadata.json
I0731 17:16:55.457584       1 certmanager.go:105] cert-manager: CA certificate written to file /csi-data-dir/csi-f2084c47363e5076b4aa1039f57947a57e3520c681faed7f25743b971bba22da/data/ca.pem
I0731 17:16:55.459810       1 certmanager.go:113] cert-manager: certificate written to file /csi-data-dir/csi-f2084c47363e5076b4aa1039f57947a57e3520c681faed7f25743b971bba22da/data/crt.pem
I0731 17:16:55.459960       1 certmanager.go:120] cert-manager: private key written to file: /csi-data-dir/csi-f2084c47363e5076b4aa1039f57947a57e3520c681faed7f25743b971bba22da/data/key.pem
E0731 17:16:55.459973       1 renew.go:181] volume already being watched, aborting second watcher: csi-f2084c47363e5076b4aa1039f57947a57e3520c681faed7f25743b971bba22da
I0731 17:16:55.460178       1 nodeserver.go:147] node: publish volume request ~ target:/var/lib/kubelet/pods/c192b6d3-ea53-4956-b624-7c2697b10c9a/volumes/kubernetes.io~csi/tls/mount volumeId:csi-f2084c47363e5076b4aa1039f57947a57e3520c681faed7f25743b971bba22da attributes:map[csi.cert-manager.io/ca-file:ca.pem csi.cert-manager.io/certificate-file:crt.pem csi.cert-manager.io/dns-names:my-service.sandbox.svc.cluster.local csi.cert-manager.io/duration:2160h0m0s csi.cert-manager.io/is-ca:false csi.cert-manager.io/issuer-group:cert-manager.io csi.cert-manager.io/issuer-kind:ClusterIssuer csi.cert-manager.io/issuer-name:ca-issuer csi.cert-manager.io/privatekey-file:key.pem csi.cert-manager.io/renew-before:720h0m0s csi.storage.k8s.io/ephemeral:true csi.storage.k8s.io/pod.name:my-csi-app csi.storage.k8s.io/pod.namespace:default csi.storage.k8s.io/pod.uid:c192b6d3-ea53-4956-b624-7c2697b10c9a csi.storage.k8s.io/serviceAccount.name:default]
I0731 17:16:55.460205       1 mount.go:84] Mounting cmd (mount) with arguments ([-o bind,ro /csi-data-dir/csi-f2084c47363e5076b4aa1039f57947a57e3520c681faed7f25743b971bba22da/data /var/lib/kubelet/pods/c192b6d3-ea53-4956-b624-7c2697b10c9a/volumes/kubernetes.io~csi/tls/mount])
E0731 17:16:55.474124       1 server.go:133] server: error: chmod /var/lib/kubelet/pods/c192b6d3-ea53-4956-b624-7c2697b10c9a/volumes/kubernetes.io~csi/tls/mount: read-only file system

[aeugenio]

yeah the code from the examples just doesnt work

I0929 05:25:05.082499       1 certmanager.go:80] cert-manager: waiting for CertificateRequest to become ready csi-c677c0a826b2787d1e99e749ac58b0e62ffb1acdedca70dbe3201ec44a5b9b80
I0929 05:25:05.087437       1 certmanager.go:105] cert-manager: CA certificate written to file /csi-data-dir/csi-c677c0a826b2787d1e99e749ac58b0e62ffb1acdedca70dbe3201ec44a5b9b80/data/ca.pem
I0929 05:25:05.087542       1 certmanager.go:113] cert-manager: certificate written to file /csi-data-dir/csi-c677c0a826b2787d1e99e749ac58b0e62ffb1acdedca70dbe3201ec44a5b9b80/data/crt.pem
I0929 05:25:05.087617       1 certmanager.go:120] cert-manager: private key written to file: /csi-data-dir/csi-c677c0a826b2787d1e99e749ac58b0e62ffb1acdedca70dbe3201ec44a5b9b80/data/key.pem
E0929 05:25:05.087630       1 renew.go:181] volume already being watched, aborting second watcher: csi-c677c0a826b2787d1e99e749ac58b0e62ffb1acdedca70dbe3201ec44a5b9b80
E0929 05:25:05.088631       1 server.go:133] server: error: chmod /var/lib/kubelet/pods/35434cc0-caac-4513-b016-a284170381d8/volumes/kubernetes.io~csi/tls/mount: read-only file system

using the eks 1.17 and the latest of cert-manager and cert-manager-csi

helm list
NAME                            REVISION    UPDATED                     STATUS      CHART                               APP VERSION NAMESPACE
cm                              1           Mon Sep 28 21:29:49 2020    DEPLOYED    cert-manager-v1.0.2                 v1.0.2      cert-manager```

[robinbraemer]

Same problem here. I deployed the given example.

pod events

Warning  FailedMount  28s                  kubelet, XXXXXXX  Unable to attach or mount volumes: unmounted volumes=[tls], unattached volumes=[default-token-s5nk7 tls]: timed out waiting for the condition
  Warning  FailedMount  16s (x9 over 2m26s)  kubelet, XXXXXXX  MountVolume.SetUp failed for volume "tls" : rpc error: code = Unknown desc = chmod /var/lib/kubelet/pods/c7bb4ae4-9e00-4ccd-8769-e186f28f696f/volumes/kubernetes.io~csi/tls/mount: read-only file system

k version

Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.2", GitCommit:"52c56ce7a8272c798dbc29846288d7cd9fbae032", GitTreeState:"clean", BuildDate:"2020-04-16T11:56:40Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.8", GitCommit:"9f2892aab98fe339f3bd70e3c470144299398ace", GitTreeState:"clean", BuildDate:"2020-08-13T16:04:18Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

[pb6]

Given that it does readonly bind mount (bind,ro), chown/chmod should fail afterwards. I changed code - removed 'ro' from mount options, and cert issuance succeeded,

total 12
-rw-------    1 root     root          1789 Oct 28 14:20 ca.pem
-rw-------    1 root     root          1464 Oct 28 14:20 crt.pem
-rw-------    1 root     root          1675 Oct 28 14:20 key.pem```

But I suspect that chmod comes from volume handling in k8s, and maybe it would be possible to specify owner and umask for created files.

[bmagic]

@pb6 What do you change to have it working ?

[pb6]

pkg/util/mount.go:102: options = append(options, "bind", "ro")

remove "ro" and it will work, as you cannot chmod/chown read only filesystem. But I'm not sure if that is what authors intended.

[xmclark]

I rebuilt the project from scratch, and republished the dockerfile, and this issue seems resolved. My image that I build from source: https://github.com/users/xmclark/packages/container/package/cert-manager-csi