etcd 3.0.7 rkt container is not starting with permission denied for ssl certs

[sachincab]

Hi Team,

When trying to run etcd-3.0.7 as rkt container it complain with below error using etcd-wrapper. etcdmain: open /etc/ssl/etcd/kube-etcd-member.pem: permission denied****

I can see that volume seems to be mounted where certs are stored.

ls -lrt /etc/ssl/etcd
total 40
-rw-r--r--. 1 root root 1472 Sep  1 12:18 kube-etcd.pem
-rw-r--r--. 1 root root 1472 Sep  1 12:18 kube-etcd-member.pem
-rw-r--r--. 1 root root 1675 Sep  1 12:18 kube-etcd-member-key.pem
-rw-r--r--. 1 root root 1675 Sep  1 12:18 kube-etcd-key.pem
-rw-r--r--. 1 root root 1387 Sep  1 12:18 kube-ca.pem

run: adding {V:etcd-certs,kind=host,source=/etc/ssl/etcd M:{Volume:etcd-certs Path:/etc/ssl/etcd}} etcd-wrapper[3036]: run: cannot run as unprivileged user

Here is the etcd3.service content:

- name: etcd3.service
  enable: true
  command: start
  content: |
    [Unit]
    Conflicts=etcd.service etcd2.service
    Requires=mnt-data.mount resize-data.service
    After=mnt-data.mount resize-data.service

    [Service]
    Environment="ETCD_DATA_DIR=/mnt/data"
    Environment="ETCD_TAG=v3.0.7"
    Environment="ETCD_IMG_USER=etcd"
    Environment="RKT_RUN_ARGS=--mount volume=etcd-certs,target=/etc/ssl/etcd --volume etcd-certs,kind=host,source=/etc/ssl/etcd,readOnly=true"
    ExecStartPre=/usr/bin/chmod 0755 /opt/etcd3/etcd-wrapper
    ExecStart=/opt/etcd3/etcd-wrapper \
    --election-timeout=1200 \
    --advertise-client-urls=https://$private_ipv4:2379 \
    --initial-advertise-peer-urls=https://$private_ipv4:2380 \
    --initial-cluster=etcd1=https://${host1}:2380,etcd2=https://${host2}:2380,etcd3=https://${host3}:2380 \
    --initial-cluster-state=new \
    --initial-cluster-token=k8s_etcd \
    --listen-client-urls=https://$private_ipv4:2379,https://127.0.0.1:2379 \
    --listen-peer-urls=https://$private_ipv4:2380 \
    --name=${etcd_name} \
    --peer-trusted-ca-file=/etc/ssl/etcd/kube-ca.pem \
    --peer-cert-file=/etc/ssl/etcd/kube-etcd-member.pem \
    --peer-key-file=/etc/ssl/etcd/kube-etcd-member-key.pem \
    --peer-client-cert-auth=true \
    --trusted-ca-file=/etc/ssl/etcd/kube-ca.pem \
    --cert-file=/etc/ssl/etcd/kube-etcd.pem \
    --key-file=/etc/ssl/etcd/kube-etcd-key.pem \
    --client-cert-auth=true
    Restart=always
    RestartSec=10s

    [Install]
    WantedBy=multi-user.target

I tried to run same rkt run command manually on server using root, gives same problem:

---

/usr/bin/rkt run --debug --mount volume=etcd-certs,target=/etc/ssl/etcd --volume etcd-certs,kind=host,source=/etc/ssl/etcd --trust-keys-from-https --mount volume=systemd-dir,target=/run/systemd/system --volume systemd-dir,kind=host,source=/run/systemd/system,readOnly=true --volume data-dir,kind=host,source=/mnt/data --mount volume=etc-ssl-certs,target=/etc/ssl/certs --volume etc-ssl-certs,kind=host,source=/etc/ssl/certs --stage1-from-dir=stage1-fly.aci coreos.com/etcd:v3.0.7 --user=232 --exec=/etcd -- --election-timeout=1200 --advertise-client-urls=https://172.23.1.10:2379 --initial-advertise-peer-urls=https://172.23.1.10:2380 --initial-cluster=etcd1=https://172.23.1.10:2380,etcd2=https://172.23.1.11:2380,etcd3=https://172.23.1.12:2380 --initial-cluster-state=new --initial-cluster-token=k8s_etcd --listen-client-urls=https://172.23.1.10:2379,https://127.0.0.1:2379 --listen-peer-urls=https://172.23.1.10:2380 --name=etcd1 --peer-trusted-ca-file=/etc/ssl/etcd/kube-ca.pem --peer-cert-file=/etc/ssl/etcd/kube-etcd-member.pem --peer-key-file=/etc/ssl/etcd/kube-etcd-member-key.pem --peer-client-cert-auth=true --trusted-ca-file=/etc/ssl/etcd/kube-ca.pem --cert-file=/etc/ssl/etcd/kube-etcd.pem --key-file=/etc/ssl/etcd/kube-etcd-key.pem --client-cert-auth=true image: using image from file /usr/lib64/rkt/stage1-images/stage1-fly.aci image: using image from local store for image name coreos.com/etcd:v3.0.7 stage0: Preparing stage1 stage0: Writing image manifest stage0: Loading image sha512-f7e8c8ac24b6b995987bc4362beeeaacd859c0aaa8b6258655d79d37605d0905 stage0: Writing image manifest stage0: Writing pod manifest stage0: Setting up stage1 stage0: Wrote filesystem to /var/lib/rkt/pods/run/d07430ed-62ca-4d2d-bd85-8d0783a54e1f stage0: Pivoting to filesystem /var/lib/rkt/pods/run/d07430ed-62ca-4d2d-bd85-8d0783a54e1f stage0: Execing /run run: adding {V:,kind= M:{Volume:etcd-certs Path:/etc/ssl/etcd}} run: adding {V:,kind= M:{Volume:systemd-dir Path:/run/systemd/system}} run: adding {V:,kind= M:{Volume:etc-ssl-certs Path:/etc/ssl/certs}} run: adding {V:,kind= M:{Volume:data-dir Path:/data-dir}} run: adding {V:etcd-certs,kind=host,source=/etc/ssl/etcd M:{Volume:etcd-certs Path:/etc/ssl/etcd}} run: adding {V:systemd-dir,kind=host,source=/run/systemd/system,readOnly=true M:{Volume:systemd-dir Path:/run/systemd/system}} run: adding {V:data-dir,kind=host,source=/mnt/data M:{Volume:data-dir Path:/data-dir}} run: adding {V:etc-ssl-certs,kind=host,source=/etc/ssl/certs M:{Volume:etc-ssl-certs Path:/etc/ssl/certs}} run: adding {V:data-dir,kind=host,source=/mnt/data,readOnly=false M:{Volume:data-dir Path:/data-dir}} run: chroot to "stage1/rootfs/opt/stage2/etcd/rootfs" run: setting uid 232 gid 0 run: execing ["/etcd" "--election-timeout=1200" "--advertise-client-urls=https://172.23.1.10:2379" "--initial-advertise-peer-urls=https://172.23.1.10:2380" "--initial-cluster=etcd1=https://172.23.1.10:2380,etcd2=https://172.23.1.11:2380,etcd3=https://172.23.1.12:2380" "--initial-cluster-state=new" "--initial-cluster-token=k8s_etcd" "--listen-client-urls=https://172.23.1.10:2379,https://127.0.0.1:2379" "--listen-peer-urls=https://172.23.1.10:2380" "--name=etcd1" "--peer-trusted-ca-file=/etc/ssl/etcd/kube-ca.pem" "--peer-cert-file=/etc/ssl/etcd/kube-etcd-member.pem" "--peer-key-file=/etc/ssl/etcd/kube-etcd-member-key.pem" "--peer-client-cert-auth=true" "--trusted-ca-file=/etc/ssl/etcd/kube-ca.pem" "--cert-file=/etc/ssl/etcd/kube-etcd.pem" "--key-file=/etc/ssl/etcd/kube-etcd-key.pem" "--client-cert-auth=true"] in "stage1/rootfs/opt/stage2/etcd/rootfs" 2016-09-01 12:22:14.146834 I | flags: recognized and used environment variable ETCD_DATA_DIR=/data-dir 2016-09-01 12:22:14.148874 I | etcdmain: etcd Version: 3.0.7 2016-09-01 12:22:14.148956 I | etcdmain: Git SHA: 5695120 2016-09-01 12:22:14.149045 I | etcdmain: Go Version: go1.6.3 2016-09-01 12:22:14.149123 I | etcdmain: Go OS/Arch: linux/amd64 2016-09-01 12:22:14.149206 I | etcdmain: setting maximum number of CPUs to 1, total number of available CPUs is 1 2016-09-01 12:22:14.149324 W | etcdmain: found invalid file/dir lost+found under data dir /data-dir (Ignore this if you are upgrading etcd) 2016-09-01 12:22:14.149410 I | etcdmain: peerTLS: cert = /etc/ssl/etcd/kube-etcd-member.pem, key = /etc/ssl/etcd/kube-etcd-member-key.pem, ca = , trusted-ca = /etc/ssl/etcd/kube-ca.pem, client-cert-auth = true

2016-09-01 12:22:14.149488 C | etcdmain: open /etc/ssl/etcd/kube-etcd-member.pem: permission denied

Please let me your inputs on why it complain with permission denied error for SSL certs.

Thanks!

[sachincab]

Hi again,

I did some more testing and seems more problem like this:

Scenario-1: When I tried to use coreos latest beta version (CoreOS 1122.1.0) which has etcd-wrapper script natively then etcd rkt container is starting properly without any permission denied error.

Scenario-2: When I tried to use CoreOS stable version: CoreOS 1068.10.0 and tried to copy and use same wrapper script then I am getting permission denied error ssl certificates which mounted as volume.

Hence I am not sure the exact problem here. In this case, I would like to use CoreOS stable version if possible.

Else I need to go with Beta version to get rid of permission denied errors.

Please let me know your thoughts on this.

[xiang90]

@crawford @steveeJ can you help to take a look at this etcd wrapper related question? thanks!

[xiang90]

@sachincab Please create an issue here: https://github.com/coreos/bugs. Thanks!

[ijajmulani]

I'm also facing same issue. Can't start ETCD in TLS mode. saying permission denied for certificate file.

{"level":"fatal","ts":"2021-05-08T11:51:19.907Z","caller":"etcdmain/etcd.go:290","msg":"listener failed","error":"open /etc/ssl/certs/etcd-ca.cert.pem: permission denied","stacktrace":"go.etcd.io/etcd/etcdmain.startEtcdOrProxyV2\n\t/tmp/etcd-release-3.4.15/etcd/release/etcd/etcdmain/etcd.go:290\ngo.etcd.io/etcd/etcdmain.Main\n\t/tmp/etcd-release-3.4.15/etcd/release/etcd/etcdmain/main.go:46\nmain.main\n\t/tmp/etcd-release-3.4.15/etcd/release/etcd/main.go:28\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:200"}

 volumes:
      - /mnt/data/certstore/etcd:/etc/ssl/certs
 environment:
- ETCD_CERT_FILE=/etc/ssl/certs/etcd-ca.cert.pem
- ETCD_KEY_FILE=/etc/ssl/certs/etcd-ca.key.nopass.pem
- ETCD_CLIENT_CERT_AUTH=true