jwickens commented 7 years ago

Hi after following the instructions here: https://coreos.com/kubernetes/docs/latest/kubernetes-on-aws-launch.html

I get the following error on trying to deploy my cluster to eu-west-1

➜  kube-aws up --s3-uri s3://my-bucket
WARN: the worker node pool "nodepool1" is associated to a k8s API endpoint behind the DNS name "my-bucket.s3.amazonaws.com" managed by YOU!
Please never point the DNS record for it to a different k8s cluster, especially when the name is a "stable" one which is shared among multiple k8s clusters for achieving blue-green deployments of k8s clusters!
kube-aws can't save users from mistakes like that
Creating AWS resources. Please wait. It may take a few minutes.
Error: Error creating cluster: Stack creation failed: CREATE_FAILED : The following resource(s) failed to create: [Controlplane]. 

Printing the most recent failed stack events:
CREATE_FAILED AWS::CloudFormation::Stack my-bucket The following resource(s) failed to create: [Controlplane].
CREATE_FAILED AWS::CloudFormation::Stack Controlplane Embedded stack arn:aws:cloudformation:eu-west-1:126855242091:stack/my-bucket-Controlplane-<ID> was not successfully created: The following resource(s) failed to create: [Etcd0].

I'm not really sure what I could have done wrong. Where should i look ?

jwickens commented 7 years ago

these seem related

198 -- issue with eu-west-2

679, #611 -- custom etcd config issues

jwickens commented 7 years ago

Inspecting further the logs in AWS: This account is currently blocked and not recognized as a valid account. Please contact aws-verification@amazon.com if you have questions. Launching EC2 instance failed.

It looks like the issue is on my end so will close for now.

TigerC10 commented 7 years ago

@jwickens what did you see?

I'm having an issue with Etcd0 right now,

21:36:14 UTC-0500   CREATE_FAILED   AWS::AutoScaling::AutoScalingGroup  Etcd0   Received 0 SUCCESS signal(s) out of 1. Unable to satisfy 100% MinSuccessfulInstancesPercent requirement
21:36:13 UTC-0500   UPDATE_IN_PROGRESS  AWS::AutoScaling::AutoScalingGroup  Etcd0   Failed to receive 1 resource signal(s) for the current batch. Each resource signal timeout is counted as a FAILURE.

Looking for anything I can about what could be going wrong here...

mumoshu commented 7 years ago

Hi @TigerC10, would you mind sharing us:

If you had enabled etcd automatic disaster-recovery feature?
If you had enabled etcd auto snapshotting feature?
EC2 Instance Type of your etcd nodes?
The result of journalctl in the newly-created-but-failed etcd node, if it is still there?

mumoshu commented 7 years ago

Also, what change(s) did you intend to apply on etcd nodes via kube-aws update?

TigerC10 commented 7 years ago

@mumoshu not doing an update, doing the initial kube-aws up. Following the instructions laid out here: https://coreos.com/kubernetes/docs/latest/kubernetes-on-aws.html

I haven't made any customizations to cluster.yaml. I didn't explicitly enable etcd disaster recovery or snapshot, and my cluster.yaml has the lines commented out.

I haven't specified anything for the instance type, so, I guess whatever the default is for us-east-1? I don't see any ec2 instances created.

redbaron commented 7 years ago

@TigetC10, quay.io had an outage yesterday, it is very much likely that you were affected by it.

To have more details ssh to etcd0 node and check journalctl output to see what went wrong

TigerC10 commented 7 years ago

@redbaron

Just a bunch of this on repeat (as it keeps trying to restart the service):

Jul 27 19:12:41 ip-10-0-0-29.ec2.internal systemd[1]: Starting etcdadm reconfigure runner...
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal systemctl[13027]: active
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal etcdadm[13033]: declare -x ETCDCTL_CACERT="/etc/ssl/certs/ca.pem"
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal etcdadm[13033]: declare -x ETCDCTL_CA_FILE="/etc/ssl/certs/ca.pem"
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal etcdadm[13033]: declare -x ETCDCTL_CERT="/etc/ssl/certs/etcd-client.pem"
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal etcdadm[13033]: declare -x ETCDCTL_CERT_FILE="/etc/ssl/certs/etcd-client.pem"
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal etcdadm[13033]: declare -x ETCDCTL_KEY="/etc/ssl/certs/etcd-client-key.pem"
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal etcdadm[13033]: declare -x ETCDCTL_KEY_FILE="/etc/ssl/certs/etcd-client-key.pem"
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal sudo[13039]:     root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/[ -w /var/run/coreos/etcdadm ]
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal sudo[13039]: pam_unix(sudo:session): session opened for user root by (uid=0)
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal systemd[1]: Created slice User Slice of root.
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal systemd[1]: Starting User Manager for UID 0...
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal systemd[13040]: pam_unix(systemd-user:session): session opened for user root by (uid=0)
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal systemd[1]: Started Session c115 of user root.
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal systemd[13040]: Reached target Paths.
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal systemd[13040]: Reached target Timers.
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal systemd[13040]: Reached target Sockets.
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal systemd[13040]: Reached target Basic System.
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal systemd[13040]: Reached target Default.
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal systemd[13040]: Startup finished in 12ms.
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal systemd[1]: Started User Manager for UID 0.
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal sudo[13039]: pam_unix(sudo:session): session closed for user root
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal systemd[1]: Stopping User Manager for UID 0...
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal systemd[13040]: Stopped target Default.
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal systemd[13040]: Stopped target Basic System.
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal systemd[13040]: Stopped target Sockets.
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal systemd[13040]: Stopped target Timers.
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal systemd[13040]: Stopped target Paths.
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal systemd[13040]: Reached target Shutdown.
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal systemd[13040]: Starting Exit the Session...
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal sudo[13053]:     root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/[ -w /var/run/coreos/etcdadm/snapshots ]
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal sudo[13053]: pam_unix(sudo:session): session opened for user root by (uid=0)
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal systemd[1]: Started Session c116 of user root.
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal systemd[13040]: Received SIGRTMIN+24 from PID 13054 (kill).
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal systemd[13041]: pam_unix(systemd-user:session): session closed for user root
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal systemd[1]: Starting User Manager for UID 0...
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal systemd[13056]: pam_unix(systemd-user:session): session opened for user root by (uid=0)
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal systemd[13056]: Reached target Sockets.
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal systemd[13056]: Reached target Timers.
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal systemd[13056]: Reached target Paths.
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal systemd[13056]: Reached target Basic System.
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal systemd[13056]: Reached target Default.
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal systemd[13056]: Startup finished in 12ms.
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal systemd[1]: Started User Manager for UID 0.
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal sudo[13053]: pam_unix(sudo:session): session closed for user root
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal systemd[1]: Stopping User Manager for UID 0...
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal sudo[13066]:     root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/[ -w /var/lib/etcd2 ]
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal systemd[13056]: Stopped target Default.
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal sudo[13066]: pam_unix(sudo:session): session opened for user root by (uid=0)
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal systemd[13056]: Stopped target Basic System.
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal systemd[13056]: Stopped target Sockets.
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal systemd[13056]: Stopped target Timers.
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal systemd[13056]: Stopped target Paths.
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal systemd[13056]: Reached target Shutdown.
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal systemd[13056]: Starting Exit the Session...
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal systemd[13056]: Received SIGRTMIN+24 from PID 13067 (kill).
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal systemd[1]: Started Session c117 of user root.
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal systemd[13057]: pam_unix(systemd-user:session): session closed for user root
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal systemd[1]: Starting User Manager for UID 0...
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal systemd[13069]: pam_unix(systemd-user:session): session opened for user root by (uid=0)
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal systemd[13069]: Reached target Sockets.
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal systemd[13069]: Reached target Timers.
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal systemd[13069]: Reached target Paths.
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal systemd[13069]: Reached target Basic System.
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal systemd[13069]: Reached target Default.
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal systemd[13069]: Startup finished in 12ms.
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal systemd[1]: Started User Manager for UID 0.
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal sudo[13066]: pam_unix(sudo:session): session closed for user root
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal systemd[1]: Stopping User Manager for UID 0...
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal systemd[13069]: Stopped target Default.
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal systemd[13069]: Stopped target Basic System.
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal systemd[13069]: Stopped target Sockets.
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal systemd[13069]: Stopped target Timers.
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal systemd[13069]: Stopped target Paths.
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal systemd[13069]: Reached target Shutdown.
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal systemd[13069]: Starting Exit the Session...
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal systemd[13069]: Received SIGRTMIN+24 from PID 13090 (kill).
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal systemd[1]: Stopped User Manager for UID 0.
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal systemd[1]: Removed slice User Slice of root.
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal kernel: SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal etcdadm[13033]: Error:  tls: failed to find any PEM data in certificate input
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal etcdadm[13033]: /opt/bin/etcdadm: info: member_reconfigure: observing cluster state: quorum=1 healthy=0
Jul 27 19:12:41 ip-10-0-0-29.ec2.internal kernel: SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
Jul 27 19:12:42 ip-10-0-0-29.ec2.internal kernel: SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
Jul 27 19:12:43 ip-10-0-0-29.ec2.internal etcdadm[13033]: /opt/bin/etcdadm: info: member_reconfigure: 1 more nodes are required until the quorum is met
Jul 27 19:12:43 ip-10-0-0-29.ec2.internal etcdadm[13033]: /opt/bin/etcdadm: info: member_set_unit_type: setting etcd unit type to "notify". `systemctl daemon-reload` required afterwards
Jul 27 19:12:43 ip-10-0-0-29.ec2.internal etcdadm[13033]: /opt/bin/etcdadm: info: member_reconfigure: all the nodes are present but cluster is still unhealthy, which means the initial bootstrap is still in progress. keep retrying a while
Jul 27 19:12:43 ip-10-0-0-29.ec2.internal etcdadm[13033]: /opt/bin/etcdadm: info: _systemctl_daemon_reload: running `systemctl daemon-reload` to reload etcd-member.service
Jul 27 19:12:43 ip-10-0-0-29.ec2.internal systemd[1]: Reloading.
Jul 27 19:12:43 ip-10-0-0-29.ec2.internal systemd[1]: Starting Garbage Collection for rkt...
Jul 27 19:12:43 ip-10-0-0-29.ec2.internal rkt[13275]: gc: moving pod "83de0f07-80cb-47b4-b982-b97774624158" to garbage
Jul 27 19:12:43 ip-10-0-0-29.ec2.internal systemd[1]: Started etcdadm reconfigure runner.
Jul 27 19:12:43 ip-10-0-0-29.ec2.internal systemd[1]: Starting etcd (System Application Container)...
Jul 27 19:12:43 ip-10-0-0-29.ec2.internal rkt[13275]: gc: pod "479fbd70-0a32-4a52-916d-50185c4463a0" not removed: still within grace period (24h0m0s)
Jul 27 19:12:43 ip-10-0-0-29.ec2.internal rkt[13275]: gc: pod "83de0f07-80cb-47b4-b982-b97774624158" not removed: still within grace period (24h0m0s)
Jul 27 19:12:43 ip-10-0-0-29.ec2.internal rkt[13275]: gc: pod "a96354d0-eaff-456f-b6ac-d4f88bad557e" not removed: still within grace period (24h0m0s)
Jul 27 19:12:43 ip-10-0-0-29.ec2.internal systemd[1]: Started Garbage Collection for rkt.
Jul 27 19:12:43 ip-10-0-0-29.ec2.internal rkt[13284]: "83de0f07-80cb-47b4-b982-b97774624158"
Jul 27 19:12:43 ip-10-0-0-29.ec2.internal systemctl[13300]: active
Jul 27 19:12:43 ip-10-0-0-29.ec2.internal systemctl[13303]: active
Jul 27 19:12:43 ip-10-0-0-29.ec2.internal etcd-wrapper[13310]: ++ id -u etcd
Jul 27 19:12:43 ip-10-0-0-29.ec2.internal etcd-wrapper[13310]: + exec /usr/bin/rkt run --uuid-file-save=/var/lib/coreos/etcd-member-wrapper.uuid --trust-keys-from-https --mount volume=systemd-dir,target=/run/systemd/system --volume systemd-dir,kind=host,source=/run/systemd/system,readOnly=true --mount volume=notify,target=/run/systemd/notify --volume notify,kind=host,source=/run/systemd/notify --set-env=NOTIFY_SOCKET=/run/systemd/notify --volume data-dir,kind=host,source=/var/lib/etcd2,readOnly=false --volume etc-ssl-certs,kind=host,source=/etc/ssl/certs,readOnly=true --volume usr-share-certs,kind=host,source=/usr/share/ca-certificates,readOnly=true --volume etc-hosts,kind=host,source=/etc/hosts,readOnly=true --volume etc-resolv,kind=host,source=/etc/resolv.conf,readOnly=true --mount volume=data-dir,target=/var/lib/etcd --mount volume=etc-ssl-certs,target=/etc/ssl/certs --mount volume=usr-share-certs,target=/usr/share/ca-certificates --mount volume=etc-hosts,target=/etc/hosts --mount volume=etc-resolv,target=/etc/resolv.conf --inherit-env --stage1-from-dir=stage1-fly.aci quay.io/coreos/etcd:v3.1.5 --user=232 --
Jul 27 19:12:44 ip-10-0-0-29.ec2.internal etcd-wrapper[13310]: 2017-07-27 19:12:44.227592 I | pkg/flags: recognized and used environment variable ETCD_ADVERTISE_CLIENT_URLS=https://ec2-XXX-XXX-XXX-XXX.compute-1.amazonaws.com:2379
Jul 27 19:12:44 ip-10-0-0-29.ec2.internal etcd-wrapper[13310]: 2017-07-27 19:12:44.227660 I | pkg/flags: recognized and used environment variable ETCD_CERT_FILE=/etc/ssl/certs/etcd.pem
Jul 27 19:12:44 ip-10-0-0-29.ec2.internal etcd-wrapper[13310]: 2017-07-27 19:12:44.227672 I | pkg/flags: recognized and used environment variable ETCD_CLIENT_CERT_AUTH=true
Jul 27 19:12:44 ip-10-0-0-29.ec2.internal etcd-wrapper[13310]: 2017-07-27 19:12:44.227684 I | pkg/flags: recognized and used environment variable ETCD_DATA_DIR=/var/lib/etcd
Jul 27 19:12:44 ip-10-0-0-29.ec2.internal etcd-wrapper[13310]: 2017-07-27 19:12:44.227707 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_ADVERTISE_PEER_URLS=https://ec2-XXX-XXX-XXX-XXX.compute-1.amazonaws.com:2380
Jul 27 19:12:44 ip-10-0-0-29.ec2.internal etcd-wrapper[13310]: 2017-07-27 19:12:44.227713 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER=etcd0=https://ec2-XXX-XXX-XXX-XXX.compute-1.amazonaws.com:2380
Jul 27 19:12:44 ip-10-0-0-29.ec2.internal etcd-wrapper[13310]: 2017-07-27 19:12:44.227719 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER_STATE=new
Jul 27 19:12:44 ip-10-0-0-29.ec2.internal etcd-wrapper[13310]: 2017-07-27 19:12:44.227725 I | pkg/flags: recognized and used environment variable ETCD_KEY_FILE=/etc/ssl/certs/etcd-key.pem
Jul 27 19:12:44 ip-10-0-0-29.ec2.internal etcd-wrapper[13310]: 2017-07-27 19:12:44.227734 I | pkg/flags: recognized and used environment variable ETCD_LISTEN_CLIENT_URLS=https://10.0.0.29:2379
Jul 27 19:12:44 ip-10-0-0-29.ec2.internal etcd-wrapper[13310]: 2017-07-27 19:12:44.227740 I | pkg/flags: recognized and used environment variable ETCD_LISTEN_PEER_URLS=https://10.0.0.29:2380
Jul 27 19:12:44 ip-10-0-0-29.ec2.internal etcd-wrapper[13310]: 2017-07-27 19:12:44.227752 I | pkg/flags: recognized and used environment variable ETCD_NAME=etcd0
Jul 27 19:12:44 ip-10-0-0-29.ec2.internal etcd-wrapper[13310]: 2017-07-27 19:12:44.227767 I | pkg/flags: recognized and used environment variable ETCD_PEER_CERT_FILE=/etc/ssl/certs/etcd.pem
Jul 27 19:12:44 ip-10-0-0-29.ec2.internal etcd-wrapper[13310]: 2017-07-27 19:12:44.227776 I | pkg/flags: recognized and used environment variable ETCD_PEER_KEY_FILE=/etc/ssl/certs/etcd-key.pem
Jul 27 19:12:44 ip-10-0-0-29.ec2.internal etcd-wrapper[13310]: 2017-07-27 19:12:44.227786 I | pkg/flags: recognized and used environment variable ETCD_PEER_TRUSTED_CA_FILE=/etc/ssl/certs/ca.pem
Jul 27 19:12:44 ip-10-0-0-29.ec2.internal etcd-wrapper[13310]: 2017-07-27 19:12:44.227801 I | pkg/flags: recognized and used environment variable ETCD_TRUSTED_CA_FILE=/etc/ssl/certs/ca.pem
Jul 27 19:12:44 ip-10-0-0-29.ec2.internal etcd-wrapper[13310]: 2017-07-27 19:12:44.227815 W | pkg/flags: unrecognized environment variable ETCD_USER=etcd
Jul 27 19:12:44 ip-10-0-0-29.ec2.internal etcd-wrapper[13310]: 2017-07-27 19:12:44.227819 W | pkg/flags: unrecognized environment variable ETCD_IMAGE_TAG=v3.1.5
Jul 27 19:12:44 ip-10-0-0-29.ec2.internal etcd-wrapper[13310]: 2017-07-27 19:12:44.227840 I | etcdmain: etcd Version: 3.1.5
Jul 27 19:12:44 ip-10-0-0-29.ec2.internal etcd-wrapper[13310]: 2017-07-27 19:12:44.227872 I | etcdmain: Git SHA: 20490ca
Jul 27 19:12:44 ip-10-0-0-29.ec2.internal etcd-wrapper[13310]: 2017-07-27 19:12:44.227879 I | etcdmain: Go Version: go1.7.5
Jul 27 19:12:44 ip-10-0-0-29.ec2.internal etcd-wrapper[13310]: 2017-07-27 19:12:44.227882 I | etcdmain: Go OS/Arch: linux/amd64
Jul 27 19:12:44 ip-10-0-0-29.ec2.internal etcd-wrapper[13310]: 2017-07-27 19:12:44.227887 I | etcdmain: setting maximum number of CPUs to 2, total number of available CPUs is 2
Jul 27 19:12:44 ip-10-0-0-29.ec2.internal etcd-wrapper[13310]: 2017-07-27 19:12:44.227926 W | etcdmain: found invalid file/dir lost+found under data dir /var/lib/etcd (Ignore this if you are upgrading etcd)
Jul 27 19:12:44 ip-10-0-0-29.ec2.internal etcd-wrapper[13310]: 2017-07-27 19:12:44.227945 I | embed: peerTLS: cert = /etc/ssl/certs/etcd.pem, key = /etc/ssl/certs/etcd-key.pem, ca = , trusted-ca = /etc/ssl/certs/ca.pem, client-cert-auth = false
Jul 27 19:12:44 ip-10-0-0-29.ec2.internal etcd-wrapper[13310]: 2017-07-27 19:12:44.227977 C | etcdmain: tls: failed to find any PEM data in certificate input
Jul 27 19:12:44 ip-10-0-0-29.ec2.internal systemd[1]: etcd-member.service: Main process exited, code=exited, status=1/FAILURE
Jul 27 19:12:44 ip-10-0-0-29.ec2.internal systemd[1]: Failed to start etcd (System Application Container).
Jul 27 19:12:44 ip-10-0-0-29.ec2.internal systemd[1]: Dependency failed for etcdadm update status.
Jul 27 19:12:44 ip-10-0-0-29.ec2.internal systemd[1]: etcdadm-update-status.service: Job etcdadm-update-status.service/start failed with result 'dependency'.
Jul 27 19:12:44 ip-10-0-0-29.ec2.internal systemd[1]: etcd-member.service: Unit entered failed state.
Jul 27 19:12:44 ip-10-0-0-29.ec2.internal systemd[1]: etcd-member.service: Failed with result 'exit-code'.
Jul 27 19:12:54 ip-10-0-0-29.ec2.internal systemd[1]: etcd-member.service: Service hold-off time over, scheduling restart.
Jul 27 19:12:54 ip-10-0-0-29.ec2.internal systemd[1]: Stopped etcd (System Application Container).
Jul 27 19:12:54 ip-10-0-0-29.ec2.internal systemd[1]: Stopped etcdadm reconfigure runner.
Jul 27 19:12:54 ip-10-0-0-29.ec2.internal systemd[1]: Stopping etcdadm reconfigure runner...

Something about one of the PEM certs not loading anything. Might have to do with the unrecognized ETCD_USER and ETCD_IMAGE_TAG environment variables?? All of the PEMs are in the credentials directory.

redbaron commented 7 years ago

Can you verify that /etc/ssl/certs/etcd.pem, /etc/ssl/certs/etcd-key.pem and /etc/ssl/certs/ca.pem on etcd0 node contain actual PEM data and not garbage or empty values?

TigerC10 commented 7 years ago

Wow. Literally nothing in those files. Seems all the other certs in the dir are symlinks.

core@ip-10-0-0-7 ~ $ cat /etc/ssl/certs/etcd.pem
core@ip-10-0-0-7 ~ $ cat /etc/ssl/certs/etcd-key.pem
core@ip-10-0-0-7 ~ $ cat /etc/ssl/certs/ca.pem
core@ip-10-0-0-7 ~ $ ls -la /etc/ssl/certs
total 1852
drwxr-xr-x. 2 root root 20480 Jul 27 20:40 .
drwxr-xr-x. 4 root root  4096 Jul 19 00:10 ..
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 02265526.0 -> ../../../usr/share/ca-certificates/02265526.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 024dc131.0 -> ../../../usr/share/ca-certificates/024dc131.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 03179a64.0 -> ../../../usr/share/ca-certificates/03179a64.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 034868d6.0 -> ../../../usr/share/ca-certificates/034868d6.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 062cdee6.0 -> ../../../usr/share/ca-certificates/062cdee6.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 064e0aa9.0 -> ../../../usr/share/ca-certificates/064e0aa9.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 080911ac.0 -> ../../../usr/share/ca-certificates/080911ac.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 0810ba98.0 -> ../../../usr/share/ca-certificates/0810ba98.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 09789157.0 -> ../../../usr/share/ca-certificates/09789157.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 0b1b94ef.0 -> ../../../usr/share/ca-certificates/0b1b94ef.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 0c4c9b6c.0 -> ../../../usr/share/ca-certificates/0c4c9b6c.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 106f3e4d.0 -> ../../../usr/share/ca-certificates/106f3e4d.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 116bf586.0 -> ../../../usr/share/ca-certificates/116bf586.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 128805a3.0 -> ../../../usr/share/ca-certificates/128805a3.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 157753a5.0 -> ../../../usr/share/ca-certificates/157753a5.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 1636090b.0 -> ../../../usr/share/ca-certificates/1636090b.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 1874d4aa.0 -> ../../../usr/share/ca-certificates/1874d4aa.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 18856ac4.0 -> ../../../usr/share/ca-certificates/18856ac4.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 19c1fa33.0 -> ../../../usr/share/ca-certificates/19c1fa33.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 1d3472b9.0 -> ../../../usr/share/ca-certificates/1d3472b9.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 1e08bfd1.0 -> ../../../usr/share/ca-certificates/1e08bfd1.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 1e09d511.0 -> ../../../usr/share/ca-certificates/1e09d511.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 244b5494.0 -> ../../../usr/share/ca-certificates/244b5494.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 26eaad2f.0 -> ../../../usr/share/ca-certificates/26eaad2f.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 2ae6433e.0 -> ../../../usr/share/ca-certificates/2ae6433e.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 2b349938.0 -> ../../../usr/share/ca-certificates/2b349938.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 2c11d503.0 -> ../../../usr/share/ca-certificates/2c11d503.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 2c543cd1.0 -> ../../../usr/share/ca-certificates/2c543cd1.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 2e4eed3c.0 -> ../../../usr/share/ca-certificates/2e4eed3c.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 2e5ac55d.0 -> ../../../usr/share/ca-certificates/2e5ac55d.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 32888f65.0 -> ../../../usr/share/ca-certificates/32888f65.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 349f2832.0 -> ../../../usr/share/ca-certificates/349f2832.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 3513523f.0 -> ../../../usr/share/ca-certificates/3513523f.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 381ce4dd.0 -> ../../../usr/share/ca-certificates/381ce4dd.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 3bde41ac.0 -> ../../../usr/share/ca-certificates/3bde41ac.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 3e45d192.0 -> ../../../usr/share/ca-certificates/3e45d192.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 3efd4dc0.0 -> ../../../usr/share/ca-certificates/3efd4dc0.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 40193066.0 -> ../../../usr/share/ca-certificates/40193066.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 4042bcee.0 -> ../../../usr/share/ca-certificates/4042bcee.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 40547a79.0 -> ../../../usr/share/ca-certificates/40547a79.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 4304c5e5.0 -> ../../../usr/share/ca-certificates/4304c5e5.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 442adcac.0 -> ../../../usr/share/ca-certificates/442adcac.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 451b5485.0 -> ../../../usr/share/ca-certificates/451b5485.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 480720ec.0 -> ../../../usr/share/ca-certificates/480720ec.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 48bec511.0 -> ../../../usr/share/ca-certificates/48bec511.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 4a6481c9.0 -> ../../../usr/share/ca-certificates/4a6481c9.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 4bfab552.0 -> ../../../usr/share/ca-certificates/4bfab552.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 4f316efb.0 -> ../../../usr/share/ca-certificates/4f316efb.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 5273a94c.0 -> ../../../usr/share/ca-certificates/5273a94c.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 5443e9e3.0 -> ../../../usr/share/ca-certificates/5443e9e3.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 54657681.0 -> ../../../usr/share/ca-certificates/54657681.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 5620c4aa.0 -> ../../../usr/share/ca-certificates/5620c4aa.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 56657bde.0 -> ../../../usr/share/ca-certificates/56657bde.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 57bbd831.0 -> ../../../usr/share/ca-certificates/57bbd831.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 57bcb2da.0 -> ../../../usr/share/ca-certificates/57bcb2da.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 592c0a9a.0 -> ../../../usr/share/ca-certificates/592c0a9a.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 5a4d6896.0 -> ../../../usr/share/ca-certificates/5a4d6896.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 5ad8a5d6.0 -> ../../../usr/share/ca-certificates/5ad8a5d6.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 5c44d531.0 -> ../../../usr/share/ca-certificates/5c44d531.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 5cd81ad7.0 -> ../../../usr/share/ca-certificates/5cd81ad7.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 5d63b0ae.0 -> ../../../usr/share/ca-certificates/5d63b0ae.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 5f15c80c.0 -> ../../../usr/share/ca-certificates/5f15c80c.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 607986c7.0 -> ../../../usr/share/ca-certificates/607986c7.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 608a55ad.0 -> ../../../usr/share/ca-certificates/608a55ad.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 6410666e.0 -> ../../../usr/share/ca-certificates/6410666e.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 653b494a.0 -> ../../../usr/share/ca-certificates/653b494a.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 65b876bd.0 -> ../../../usr/share/ca-certificates/65b876bd.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 667c66d4.0 -> ../../../usr/share/ca-certificates/667c66d4.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 6b99d060.0 -> ../../../usr/share/ca-certificates/6b99d060.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 6f2c1157.0 -> ../../../usr/share/ca-certificates/6f2c1157.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 706f604c.0 -> ../../../usr/share/ca-certificates/706f604c.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 749e9e03.0 -> ../../../usr/share/ca-certificates/749e9e03.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 75d1b2ed.0 -> ../../../usr/share/ca-certificates/75d1b2ed.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 76cb8f92.0 -> ../../../usr/share/ca-certificates/76cb8f92.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 76faf6c0.0 -> ../../../usr/share/ca-certificates/76faf6c0.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 7719f463.0 -> ../../../usr/share/ca-certificates/7719f463.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 790a7190.0 -> ../../../usr/share/ca-certificates/790a7190.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 7992b8bb.0 -> ../../../usr/share/ca-certificates/7992b8bb.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 7d0b38bd.0 -> ../../../usr/share/ca-certificates/7d0b38bd.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 7d5a75e4.0 -> ../../../usr/share/ca-certificates/7d5a75e4.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 7f3d5d1d.0 -> ../../../usr/share/ca-certificates/7f3d5d1d.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 8096d0a9.0 -> ../../../usr/share/ca-certificates/8096d0a9.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 812e17de.0 -> ../../../usr/share/ca-certificates/812e17de.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 8160b96c.0 -> ../../../usr/share/ca-certificates/8160b96c.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 861a399d.0 -> ../../../usr/share/ca-certificates/861a399d.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 87229d21.0 -> ../../../usr/share/ca-certificates/87229d21.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 876f1e28.0 -> ../../../usr/share/ca-certificates/876f1e28.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 8867006a.0 -> ../../../usr/share/ca-certificates/8867006a.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 8b59b1ad.0 -> ../../../usr/share/ca-certificates/8b59b1ad.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 8d86cdd1.0 -> ../../../usr/share/ca-certificates/8d86cdd1.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 9007ae68.0 -> ../../../usr/share/ca-certificates/9007ae68.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 9168f543.0 -> ../../../usr/share/ca-certificates/9168f543.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 930ac5d2.0 -> ../../../usr/share/ca-certificates/930ac5d2.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 93bc0acc.0 -> ../../../usr/share/ca-certificates/93bc0acc.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 988a38cb.0 -> ../../../usr/share/ca-certificates/988a38cb.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 9c2e7d30.0 -> ../../../usr/share/ca-certificates/9c2e7d30.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 9c8dfbd4.0 -> ../../../usr/share/ca-certificates/9c8dfbd4.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 9d04f354.0 -> ../../../usr/share/ca-certificates/9d04f354.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 9d520b32.0 -> ../../../usr/share/ca-certificates/9d520b32.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 9f0f5fd6.0 -> ../../../usr/share/ca-certificates/9f0f5fd6.0
lrwxrwxrwx. 1 root root    48 Jul 19 00:10 ACCVRAIZ1.pem -> ../../../usr/share/ca-certificates/ACCVRAIZ1.pem
lrwxrwxrwx. 1 root root    52 Jul 19 00:10 ACEDICOM_Root.pem -> ../../../usr/share/ca-certificates/ACEDICOM_Root.pem
lrwxrwxrwx. 1 root root    65 Jul 19 00:10 'AC_Ra'$'\303\255''z_Certic'$'\303\241''mara_S.A..pem' -> '../../../usr/share/ca-certificates/AC_Ra'$'\303\255''z_Certic'$'\303\241''mara_S.A..pem'
lrwxrwxrwx. 1 root root    69 Jul 19 00:10 Actalis_Authentication_Root_CA.pem -> ../../../usr/share/ca-certificates/Actalis_Authentication_Root_CA.pem
lrwxrwxrwx. 1 root root    61 Jul 19 00:10 AddTrust_External_Root.pem -> ../../../usr/share/ca-certificates/AddTrust_External_Root.pem
lrwxrwxrwx. 1 root root    71 Jul 19 00:10 AddTrust_Low-Value_Services_Root.pem -> ../../../usr/share/ca-certificates/AddTrust_Low-Value_Services_Root.pem
lrwxrwxrwx. 1 root root    68 Jul 19 00:10 AddTrust_Public_Services_Root.pem -> ../../../usr/share/ca-certificates/AddTrust_Public_Services_Root.pem
lrwxrwxrwx. 1 root root    75 Jul 19 00:10 AddTrust_Qualified_Certificates_Root.pem -> ../../../usr/share/ca-certificates/AddTrust_Qualified_Certificates_Root.pem
lrwxrwxrwx. 1 root root    61 Jul 19 00:10 AffirmTrust_Commercial.pem -> ../../../usr/share/ca-certificates/AffirmTrust_Commercial.pem
lrwxrwxrwx. 1 root root    61 Jul 19 00:10 AffirmTrust_Networking.pem -> ../../../usr/share/ca-certificates/AffirmTrust_Networking.pem
lrwxrwxrwx. 1 root root    58 Jul 19 00:10 AffirmTrust_Premium.pem -> ../../../usr/share/ca-certificates/AffirmTrust_Premium.pem
lrwxrwxrwx. 1 root root    62 Jul 19 00:10 AffirmTrust_Premium_ECC.pem -> ../../../usr/share/ca-certificates/AffirmTrust_Premium_ECC.pem
lrwxrwxrwx. 1 root root    74 Jul 19 00:10 ApplicationCA_-_Japanese_Government.pem -> ../../../usr/share/ca-certificates/ApplicationCA_-_Japanese_Government.pem
lrwxrwxrwx. 1 root root    60 Jul 19 00:10 Atos_TrustedRoot_2011.pem -> ../../../usr/share/ca-certificates/Atos_TrustedRoot_2011.pem
lrwxrwxrwx. 1 root root    96 Jul 19 00:10 Autoridad_de_Certificacion_Firmaprofesional_CIF_A62634068.pem -> ../../../usr/share/ca-certificates/Autoridad_de_Certificacion_Firmaprofesional_CIF_A62634068.pem
lrwxrwxrwx. 1 root root    64 Jul 19 00:10 Baltimore_CyberTrust_Root.pem -> ../../../usr/share/ca-certificates/Baltimore_CyberTrust_Root.pem
lrwxrwxrwx. 1 root root    59 Jul 19 00:10 Buypass_Class_2_CA_1.pem -> ../../../usr/share/ca-certificates/Buypass_Class_2_CA_1.pem
lrwxrwxrwx. 1 root root    62 Jul 19 00:10 Buypass_Class_2_Root_CA.pem -> ../../../usr/share/ca-certificates/Buypass_Class_2_Root_CA.pem
lrwxrwxrwx. 1 root root    62 Jul 19 00:10 Buypass_Class_3_Root_CA.pem -> ../../../usr/share/ca-certificates/Buypass_Class_3_Root_CA.pem
lrwxrwxrwx. 1 root root    55 Jul 19 00:10 CA_Disig_Root_R1.pem -> ../../../usr/share/ca-certificates/CA_Disig_Root_R1.pem
lrwxrwxrwx. 1 root root    55 Jul 19 00:10 CA_Disig_Root_R2.pem -> ../../../usr/share/ca-certificates/CA_Disig_Root_R2.pem
lrwxrwxrwx. 1 root root    57 Jul 19 00:10 CA_WoSign_ECC_Root.pem -> ../../../usr/share/ca-certificates/CA_WoSign_ECC_Root.pem
lrwxrwxrwx. 1 root root    51 Jul 19 00:10 CFCA_EV_ROOT.pem -> ../../../usr/share/ca-certificates/CFCA_EV_ROOT.pem
lrwxrwxrwx. 1 root root    49 Jul 19 00:10 CNNIC_ROOT.pem -> ../../../usr/share/ca-certificates/CNNIC_ROOT.pem
lrwxrwxrwx. 1 root root    69 Jul 19 00:10 COMODO_Certification_Authority.pem -> ../../../usr/share/ca-certificates/COMODO_Certification_Authority.pem
lrwxrwxrwx. 1 root root    73 Jul 19 00:10 COMODO_ECC_Certification_Authority.pem -> ../../../usr/share/ca-certificates/COMODO_ECC_Certification_Authority.pem
lrwxrwxrwx. 1 root root    73 Jul 19 00:10 COMODO_RSA_Certification_Authority.pem -> ../../../usr/share/ca-certificates/COMODO_RSA_Certification_Authority.pem
lrwxrwxrwx. 1 root root    75 Jul 19 00:10 Camerfirma_Chambers_of_Commerce_Root.pem -> ../../../usr/share/ca-certificates/Camerfirma_Chambers_of_Commerce_Root.pem
lrwxrwxrwx. 1 root root    73 Jul 19 00:10 Camerfirma_Global_Chambersign_Root.pem -> ../../../usr/share/ca-certificates/Camerfirma_Global_Chambersign_Root.pem
lrwxrwxrwx. 1 root root    75 Jul 19 00:10 Certification_Authority_of_WoSign_G2.pem -> ../../../usr/share/ca-certificates/Certification_Authority_of_WoSign_G2.pem
lrwxrwxrwx. 1 root root    47 Jul 19 00:10 Certigna.pem -> ../../../usr/share/ca-certificates/Certigna.pem
lrwxrwxrwx. 1 root root    68 Jul 19 00:10 'Certinomis_-_Autorit'$'\303\251''_Racine.pem' -> '../../../usr/share/ca-certificates/Certinomis_-_Autorit'$'\303\251''_Racine.pem'
lrwxrwxrwx. 1 root root    59 Jul 19 00:10 Certinomis_-_Root_CA.pem -> ../../../usr/share/ca-certificates/Certinomis_-_Root_CA.pem
lrwxrwxrwx. 1 root root    66 Jul 19 00:10 Certplus_Class_2_Primary_CA.pem -> ../../../usr/share/ca-certificates/Certplus_Class_2_Primary_CA.pem
lrwxrwxrwx. 1 root root    58 Jul 19 00:10 Certplus_Root_CA_G1.pem -> ../../../usr/share/ca-certificates/Certplus_Root_CA_G1.pem
lrwxrwxrwx. 1 root root    58 Jul 19 00:10 Certplus_Root_CA_G2.pem -> ../../../usr/share/ca-certificates/Certplus_Root_CA_G2.pem
lrwxrwxrwx. 1 root root    53 Jul 19 00:10 Certum_Root_CA.pem -> ../../../usr/share/ca-certificates/Certum_Root_CA.pem
lrwxrwxrwx. 1 root root    64 Jul 19 00:10 Certum_Trusted_Network_CA.pem -> ../../../usr/share/ca-certificates/Certum_Trusted_Network_CA.pem
lrwxrwxrwx. 1 root root    66 Jul 19 00:10 Certum_Trusted_Network_CA_2.pem -> ../../../usr/share/ca-certificates/Certum_Trusted_Network_CA_2.pem
lrwxrwxrwx. 1 root root    71 Jul 19 00:10 Chambers_of_Commerce_Root_-_2008.pem -> ../../../usr/share/ca-certificates/Chambers_of_Commerce_Root_-_2008.pem
lrwxrwxrwx. 1 root root   101 Jul 19 00:10 China_Internet_Network_Information_Center_EV_Certificates_Root.pem -> ../../../usr/share/ca-certificates/China_Internet_Network_Information_Center_EV_Certificates_Root.pem
lrwxrwxrwx. 1 root root    49 Jul 19 00:10 ComSign_CA.pem -> ../../../usr/share/ca-certificates/ComSign_CA.pem
lrwxrwxrwx. 1 root root    63 Jul 19 00:10 Comodo_AAA_Services_root.pem -> ../../../usr/share/ca-certificates/Comodo_AAA_Services_root.pem
lrwxrwxrwx. 1 root root    66 Jul 19 00:10 Comodo_Secure_Services_root.pem -> ../../../usr/share/ca-certificates/Comodo_Secure_Services_root.pem
lrwxrwxrwx. 1 root root    67 Jul 19 00:10 Comodo_Trusted_Services_root.pem -> ../../../usr/share/ca-certificates/Comodo_Trusted_Services_root.pem
lrwxrwxrwx. 1 root root    61 Jul 19 00:10 Cybertrust_Global_Root.pem -> ../../../usr/share/ca-certificates/Cybertrust_Global_Root.pem
lrwxrwxrwx. 1 root root    69 Jul 19 00:10 D-TRUST_Root_Class_3_CA_2_2009.pem -> ../../../usr/share/ca-certificates/D-TRUST_Root_Class_3_CA_2_2009.pem
lrwxrwxrwx. 1 root root    72 Jul 19 00:10 D-TRUST_Root_Class_3_CA_2_EV_2009.pem -> ../../../usr/share/ca-certificates/D-TRUST_Root_Class_3_CA_2_EV_2009.pem
lrwxrwxrwx. 1 root root    53 Jul 19 00:10 DST_ACES_CA_X6.pem -> ../../../usr/share/ca-certificates/DST_ACES_CA_X6.pem
lrwxrwxrwx. 1 root root    53 Jul 19 00:10 DST_Root_CA_X3.pem -> ../../../usr/share/ca-certificates/DST_Root_CA_X3.pem
lrwxrwxrwx. 1 root root    65 Jul 19 00:10 Deutsche_Telekom_Root_CA_2.pem -> ../../../usr/share/ca-certificates/Deutsche_Telekom_Root_CA_2.pem
lrwxrwxrwx. 1 root root    66 Jul 19 00:10 DigiCert_Assured_ID_Root_CA.pem -> ../../../usr/share/ca-certificates/DigiCert_Assured_ID_Root_CA.pem
lrwxrwxrwx. 1 root root    66 Jul 19 00:10 DigiCert_Assured_ID_Root_G2.pem -> ../../../usr/share/ca-certificates/DigiCert_Assured_ID_Root_G2.pem
lrwxrwxrwx. 1 root root    66 Jul 19 00:10 DigiCert_Assured_ID_Root_G3.pem -> ../../../usr/share/ca-certificates/DigiCert_Assured_ID_Root_G3.pem
lrwxrwxrwx. 1 root root    62 Jul 19 00:10 DigiCert_Global_Root_CA.pem -> ../../../usr/share/ca-certificates/DigiCert_Global_Root_CA.pem
lrwxrwxrwx. 1 root root    62 Jul 19 00:10 DigiCert_Global_Root_G2.pem -> ../../../usr/share/ca-certificates/DigiCert_Global_Root_G2.pem
lrwxrwxrwx. 1 root root    62 Jul 19 00:10 DigiCert_Global_Root_G3.pem -> ../../../usr/share/ca-certificates/DigiCert_Global_Root_G3.pem
lrwxrwxrwx. 1 root root    73 Jul 19 00:10 DigiCert_High_Assurance_EV_Root_CA.pem -> ../../../usr/share/ca-certificates/DigiCert_High_Assurance_EV_Root_CA.pem
lrwxrwxrwx. 1 root root    63 Jul 19 00:10 DigiCert_Trusted_Root_G4.pem -> ../../../usr/share/ca-certificates/DigiCert_Trusted_Root_G4.pem
lrwxrwxrwx. 1 root root    70 Jul 19 00:10 E-Tugra_Certification_Authority.pem -> ../../../usr/share/ca-certificates/E-Tugra_Certification_Authority.pem
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 EC-ACC.pem -> ../../../usr/share/ca-certificates/EC-ACC.pem
lrwxrwxrwx. 1 root root    70 Jul 19 00:10 EE_Certification_Centre_Root_CA.pem -> ../../../usr/share/ca-certificates/EE_Certification_Centre_Root_CA.pem
lrwxrwxrwx. 1 root root    80 Jul 19 00:10 Entrust.net_Premium_2048_Secure_Server_CA.pem -> ../../../usr/share/ca-certificates/Entrust.net_Premium_2048_Secure_Server_CA.pem
lrwxrwxrwx. 1 root root    75 Jul 19 00:10 Entrust_Root_Certification_Authority.pem -> ../../../usr/share/ca-certificates/Entrust_Root_Certification_Authority.pem
lrwxrwxrwx. 1 root root    81 Jul 19 00:10 Entrust_Root_Certification_Authority_-_EC1.pem -> ../../../usr/share/ca-certificates/Entrust_Root_Certification_Authority_-_EC1.pem
lrwxrwxrwx. 1 root root    80 Jul 19 00:10 Entrust_Root_Certification_Authority_-_G2.pem -> ../../../usr/share/ca-certificates/Entrust_Root_Certification_Authority_-_G2.pem
lrwxrwxrwx. 1 root root    57 Jul 19 00:10 GeoTrust_Global_CA.pem -> ../../../usr/share/ca-certificates/GeoTrust_Global_CA.pem
lrwxrwxrwx. 1 root root    59 Jul 19 00:10 GeoTrust_Global_CA_2.pem -> ../../../usr/share/ca-certificates/GeoTrust_Global_CA_2.pem
lrwxrwxrwx. 1 root root    79 Jul 19 00:10 GeoTrust_Primary_Certification_Authority.pem -> ../../../usr/share/ca-certificates/GeoTrust_Primary_Certification_Authority.pem
lrwxrwxrwx. 1 root root    84 Jul 19 00:10 GeoTrust_Primary_Certification_Authority_-_G2.pem -> ../../../usr/share/ca-certificates/GeoTrust_Primary_Certification_Authority_-_G2.pem
lrwxrwxrwx. 1 root root    84 Jul 19 00:10 GeoTrust_Primary_Certification_Authority_-_G3.pem -> ../../../usr/share/ca-certificates/GeoTrust_Primary_Certification_Authority_-_G3.pem
lrwxrwxrwx. 1 root root    60 Jul 19 00:10 GeoTrust_Universal_CA.pem -> ../../../usr/share/ca-certificates/GeoTrust_Universal_CA.pem
lrwxrwxrwx. 1 root root    62 Jul 19 00:10 GeoTrust_Universal_CA_2.pem -> ../../../usr/share/ca-certificates/GeoTrust_Universal_CA_2.pem
lrwxrwxrwx. 1 root root    66 Jul 19 00:10 GlobalSign_ECC_Root_CA_-_R4.pem -> ../../../usr/share/ca-certificates/GlobalSign_ECC_Root_CA_-_R4.pem
lrwxrwxrwx. 1 root root    66 Jul 19 00:10 GlobalSign_ECC_Root_CA_-_R5.pem -> ../../../usr/share/ca-certificates/GlobalSign_ECC_Root_CA_-_R5.pem
lrwxrwxrwx. 1 root root    57 Jul 19 00:10 GlobalSign_Root_CA.pem -> ../../../usr/share/ca-certificates/GlobalSign_Root_CA.pem
lrwxrwxrwx. 1 root root    62 Jul 19 00:10 GlobalSign_Root_CA_-_R2.pem -> ../../../usr/share/ca-certificates/GlobalSign_Root_CA_-_R2.pem
lrwxrwxrwx. 1 root root    62 Jul 19 00:10 GlobalSign_Root_CA_-_R3.pem -> ../../../usr/share/ca-certificates/GlobalSign_Root_CA_-_R3.pem
lrwxrwxrwx. 1 root root    69 Jul 19 00:10 Global_Chambersign_Root_-_2008.pem -> ../../../usr/share/ca-certificates/Global_Chambersign_Root_-_2008.pem
lrwxrwxrwx. 1 root root    58 Jul 19 00:10 Go_Daddy_Class_2_CA.pem -> ../../../usr/share/ca-certificates/Go_Daddy_Class_2_CA.pem
lrwxrwxrwx. 1 root root    79 Jul 19 00:10 Go_Daddy_Root_Certificate_Authority_-_G2.pem -> ../../../usr/share/ca-certificates/Go_Daddy_Root_Certificate_Authority_-_G2.pem
lrwxrwxrwx. 1 root root    98 Jul 19 00:10 Hellenic_Academic_and_Research_Institutions_ECC_RootCA_2015.pem -> ../../../usr/share/ca-certificates/Hellenic_Academic_and_Research_Institutions_ECC_RootCA_2015.pem
lrwxrwxrwx. 1 root root    94 Jul 19 00:10 Hellenic_Academic_and_Research_Institutions_RootCA_2011.pem -> ../../../usr/share/ca-certificates/Hellenic_Academic_and_Research_Institutions_RootCA_2011.pem
lrwxrwxrwx. 1 root root    94 Jul 19 00:10 Hellenic_Academic_and_Research_Institutions_RootCA_2015.pem -> ../../../usr/share/ca-certificates/Hellenic_Academic_and_Research_Institutions_RootCA_2015.pem
lrwxrwxrwx. 1 root root    62 Jul 19 00:10 Hongkong_Post_Root_CA_1.pem -> ../../../usr/share/ca-certificates/Hongkong_Post_Root_CA_1.pem
lrwxrwxrwx. 1 root root    51 Jul 19 00:10 ISRG_Root_X1.pem -> ../../../usr/share/ca-certificates/ISRG_Root_X1.pem
lrwxrwxrwx. 1 root root    69 Jul 19 00:10 IdenTrust_Commercial_Root_CA_1.pem -> ../../../usr/share/ca-certificates/IdenTrust_Commercial_Root_CA_1.pem
lrwxrwxrwx. 1 root root    72 Jul 19 00:10 IdenTrust_Public_Sector_Root_CA_1.pem -> ../../../usr/share/ca-certificates/IdenTrust_Public_Sector_Root_CA_1.pem
lrwxrwxrwx. 1 root root    49 Jul 19 00:10 Izenpe.com.pem -> ../../../usr/share/ca-certificates/Izenpe.com.pem
lrwxrwxrwx. 1 root root    64 Jul 19 00:10 Microsec_e-Szigno_Root_CA.pem -> ../../../usr/share/ca-certificates/Microsec_e-Szigno_Root_CA.pem
lrwxrwxrwx. 1 root root    69 Jul 19 00:10 Microsec_e-Szigno_Root_CA_2009.pem -> ../../../usr/share/ca-certificates/Microsec_e-Szigno_Root_CA_2009.pem
lrwxrwxrwx. 1 root root    83 Jul 19 00:10 'NetLock_Arany_=Class_Gold=_F'$'\305\221''tan'$'\303\272''s'$'\303\255''tv'$'\303\241''ny.pem' -> '../../../usr/share/ca-certificates/NetLock_Arany_=Class_Gold=_F'$'\305\221''tan'$'\303\272''s'$'\303\255''tv'$'\303\241''ny.pem'
lrwxrwxrwx. 1 root root    78 Jul 19 00:10 Network_Solutions_Certificate_Authority.pem -> ../../../usr/share/ca-certificates/Network_Solutions_Certificate_Authority.pem
lrwxrwxrwx. 1 root root    70 Jul 19 00:10 OISTE_WISeKey_Global_Root_GA_CA.pem -> ../../../usr/share/ca-certificates/OISTE_WISeKey_Global_Root_GA_CA.pem
lrwxrwxrwx. 1 root root    70 Jul 19 00:10 OISTE_WISeKey_Global_Root_GB_CA.pem -> ../../../usr/share/ca-certificates/OISTE_WISeKey_Global_Root_GB_CA.pem
lrwxrwxrwx. 1 root root    59 Jul 19 00:10 OpenTrust_Root_CA_G1.pem -> ../../../usr/share/ca-certificates/OpenTrust_Root_CA_G1.pem
lrwxrwxrwx. 1 root root    59 Jul 19 00:10 OpenTrust_Root_CA_G2.pem -> ../../../usr/share/ca-certificates/OpenTrust_Root_CA_G2.pem
lrwxrwxrwx. 1 root root    59 Jul 19 00:10 OpenTrust_Root_CA_G3.pem -> ../../../usr/share/ca-certificates/OpenTrust_Root_CA_G3.pem
lrwxrwxrwx. 1 root root    49 Jul 19 00:10 PSCProcert.pem -> ../../../usr/share/ca-certificates/PSCProcert.pem
lrwxrwxrwx. 1 root root    55 Jul 19 00:10 QuoVadis_Root_CA.pem -> ../../../usr/share/ca-certificates/QuoVadis_Root_CA.pem
lrwxrwxrwx. 1 root root    60 Jul 19 00:10 QuoVadis_Root_CA_1_G3.pem -> ../../../usr/share/ca-certificates/QuoVadis_Root_CA_1_G3.pem
lrwxrwxrwx. 1 root root    57 Jul 19 00:10 QuoVadis_Root_CA_2.pem -> ../../../usr/share/ca-certificates/QuoVadis_Root_CA_2.pem
lrwxrwxrwx. 1 root root    60 Jul 19 00:10 QuoVadis_Root_CA_2_G3.pem -> ../../../usr/share/ca-certificates/QuoVadis_Root_CA_2_G3.pem
lrwxrwxrwx. 1 root root    57 Jul 19 00:10 QuoVadis_Root_CA_3.pem -> ../../../usr/share/ca-certificates/QuoVadis_Root_CA_3.pem
lrwxrwxrwx. 1 root root    60 Jul 19 00:10 QuoVadis_Root_CA_3_G3.pem -> ../../../usr/share/ca-certificates/QuoVadis_Root_CA_3_G3.pem
lrwxrwxrwx. 1 root root    59 Jul 19 00:10 RSA_Security_2048_v3.pem -> ../../../usr/share/ca-certificates/RSA_Security_2048_v3.pem
lrwxrwxrwx. 1 root root    69 Jul 19 00:10 Root_CA_Generalitat_Valenciana.pem -> ../../../usr/share/ca-certificates/Root_CA_Generalitat_Valenciana.pem
lrwxrwxrwx. 1 root root    64 Jul 19 00:10 S-TRUST_Universal_Root_CA.pem -> ../../../usr/share/ca-certificates/S-TRUST_Universal_Root_CA.pem
lrwxrwxrwx. 1 root root    54 Jul 19 00:10 SZAFIR_ROOT_CA2.pem -> ../../../usr/share/ca-certificates/SZAFIR_ROOT_CA2.pem
lrwxrwxrwx. 1 root root    58 Jul 19 00:10 SecureSign_RootCA11.pem -> ../../../usr/share/ca-certificates/SecureSign_RootCA11.pem
lrwxrwxrwx. 1 root root    53 Jul 19 00:10 SecureTrust_CA.pem -> ../../../usr/share/ca-certificates/SecureTrust_CA.pem
lrwxrwxrwx. 1 root root    55 Jul 19 00:10 Secure_Global_CA.pem -> ../../../usr/share/ca-certificates/Secure_Global_CA.pem
lrwxrwxrwx. 1 root root    72 Jul 19 00:10 Security_Communication_EV_RootCA1.pem -> ../../../usr/share/ca-certificates/Security_Communication_EV_RootCA1.pem
lrwxrwxrwx. 1 root root    69 Jul 19 00:10 Security_Communication_RootCA2.pem -> ../../../usr/share/ca-certificates/Security_Communication_RootCA2.pem
lrwxrwxrwx. 1 root root    69 Jul 19 00:10 Security_Communication_Root_CA.pem -> ../../../usr/share/ca-certificates/Security_Communication_Root_CA.pem
lrwxrwxrwx. 1 root root    61 Jul 19 00:10 Sonera_Class_2_Root_CA.pem -> ../../../usr/share/ca-certificates/Sonera_Class_2_Root_CA.pem
lrwxrwxrwx. 1 root root    71 Jul 19 00:10 Staat_der_Nederlanden_EV_Root_CA.pem -> ../../../usr/share/ca-certificates/Staat_der_Nederlanden_EV_Root_CA.pem
lrwxrwxrwx. 1 root root    73 Jul 19 00:10 Staat_der_Nederlanden_Root_CA_-_G2.pem -> ../../../usr/share/ca-certificates/Staat_der_Nederlanden_Root_CA_-_G2.pem
lrwxrwxrwx. 1 root root    73 Jul 19 00:10 Staat_der_Nederlanden_Root_CA_-_G3.pem -> ../../../usr/share/ca-certificates/Staat_der_Nederlanden_Root_CA_-_G3.pem
lrwxrwxrwx. 1 root root    59 Jul 19 00:10 Starfield_Class_2_CA.pem -> ../../../usr/share/ca-certificates/Starfield_Class_2_CA.pem
lrwxrwxrwx. 1 root root    80 Jul 19 00:10 Starfield_Root_Certificate_Authority_-_G2.pem -> ../../../usr/share/ca-certificates/Starfield_Root_Certificate_Authority_-_G2.pem
lrwxrwxrwx. 1 root root    89 Jul 19 00:10 Starfield_Services_Root_Certificate_Authority_-_G2.pem -> ../../../usr/share/ca-certificates/Starfield_Services_Root_Certificate_Authority_-_G2.pem
lrwxrwxrwx. 1 root root    71 Jul 19 00:10 StartCom_Certification_Authority.pem -> ../../../usr/share/ca-certificates/StartCom_Certification_Authority.pem
lrwxrwxrwx. 1 root root    74 Jul 19 00:10 StartCom_Certification_Authority_G2.pem -> ../../../usr/share/ca-certificates/StartCom_Certification_Authority_G2.pem
lrwxrwxrwx. 1 root root    61 Jul 19 00:10 SwissSign_Gold_CA_-_G2.pem -> ../../../usr/share/ca-certificates/SwissSign_Gold_CA_-_G2.pem
lrwxrwxrwx. 1 root root    65 Jul 19 00:10 SwissSign_Platinum_CA_-_G2.pem -> ../../../usr/share/ca-certificates/SwissSign_Platinum_CA_-_G2.pem
lrwxrwxrwx. 1 root root    63 Jul 19 00:10 SwissSign_Silver_CA_-_G2.pem -> ../../../usr/share/ca-certificates/SwissSign_Silver_CA_-_G2.pem
lrwxrwxrwx. 1 root root    57 Jul 19 00:10 Swisscom_Root_CA_1.pem -> ../../../usr/share/ca-certificates/Swisscom_Root_CA_1.pem
lrwxrwxrwx. 1 root root    57 Jul 19 00:10 Swisscom_Root_CA_2.pem -> ../../../usr/share/ca-certificates/Swisscom_Root_CA_2.pem
lrwxrwxrwx. 1 root root    60 Jul 19 00:10 Swisscom_Root_EV_CA_2.pem -> ../../../usr/share/ca-certificates/Swisscom_Root_EV_CA_2.pem
lrwxrwxrwx. 1 root root    67 Jul 19 00:10 T-TeleSec_GlobalRoot_Class_2.pem -> ../../../usr/share/ca-certificates/T-TeleSec_GlobalRoot_Class_2.pem
lrwxrwxrwx. 1 root root    67 Jul 19 00:10 T-TeleSec_GlobalRoot_Class_3.pem -> ../../../usr/share/ca-certificates/T-TeleSec_GlobalRoot_Class_3.pem
lrwxrwxrwx. 1 root root    67 Jul 19 00:10 TC_TrustCenter_Class_3_CA_II.pem -> ../../../usr/share/ca-certificates/TC_TrustCenter_Class_3_CA_II.pem
lrwxrwxrwx. 1 root root    88 Jul 19 00:10 TURKTRUST_Certificate_Services_Provider_Root_2007.pem -> ../../../usr/share/ca-certificates/TURKTRUST_Certificate_Services_Provider_Root_2007.pem
lrwxrwxrwx. 1 root root    58 Jul 19 00:10 TWCA_Global_Root_CA.pem -> ../../../usr/share/ca-certificates/TWCA_Global_Root_CA.pem
lrwxrwxrwx. 1 root root    72 Jul 19 00:10 TWCA_Root_Certification_Authority.pem -> ../../../usr/share/ca-certificates/TWCA_Root_Certification_Authority.pem
lrwxrwxrwx. 1 root root    50 Jul 19 00:10 Taiwan_GRCA.pem -> ../../../usr/share/ca-certificates/Taiwan_GRCA.pem
lrwxrwxrwx. 1 root root    61 Jul 19 00:10 TeliaSonera_Root_CA_v1.pem -> ../../../usr/share/ca-certificates/TeliaSonera_Root_CA_v1.pem
lrwxrwxrwx. 1 root root    58 Jul 19 00:10 Trustis_FPS_Root_CA.pem -> ../../../usr/share/ca-certificates/Trustis_FPS_Root_CA.pem
lrwxrwxrwx. 1 root root   104 Jul 19 00:10 'T'$'\303\234''B'$'\304\260''TAK_UEKAE_K'$'\303\266''k_Sertifika_Hizmet_Sa'$'\304\237''lay'$'\304\261''c'$'\304\261''s'$'\304\261''_-_S'$'\303\274''r'$'\303\274''m_3.pem' -> '../../../usr/share/ca-certificates/T'$'\303\234''B'$'\304\260''TAK_UEKAE_K'$'\303\266''k_Sertifika_Hizmet_Sa'$'\304\237''lay'$'\304\261''c'$'\304\261''s'$'\304\261''_-_S'$'\303\274''r'$'\303\274''m_3.pem'
lrwxrwxrwx. 1 root root    96 Jul 19 00:10 'T'$'\303\234''RKTRUST_Elektronik_Sertifika_Hizmet_Sa'$'\304\237''lay'$'\304\261''c'$'\304\261''s'$'\304\261''_H5.pem' -> '../../../usr/share/ca-certificates/T'$'\303\234''RKTRUST_Elektronik_Sertifika_Hizmet_Sa'$'\304\237''lay'$'\304\261''c'$'\304\261''s'$'\304\261''_H5.pem'
lrwxrwxrwx. 1 root root    96 Jul 19 00:10 'T'$'\303\234''RKTRUST_Elektronik_Sertifika_Hizmet_Sa'$'\304\237''lay'$'\304\261''c'$'\304\261''s'$'\304\261''_H6.pem' -> '../../../usr/share/ca-certificates/T'$'\303\234''RKTRUST_Elektronik_Sertifika_Hizmet_Sa'$'\304\237''lay'$'\304\261''c'$'\304\261''s'$'\304\261''_H6.pem'
lrwxrwxrwx. 1 root root    76 Jul 19 00:10 USERTrust_ECC_Certification_Authority.pem -> ../../../usr/share/ca-certificates/USERTrust_ECC_Certification_Authority.pem
lrwxrwxrwx. 1 root root    76 Jul 19 00:10 USERTrust_RSA_Certification_Authority.pem -> ../../../usr/share/ca-certificates/USERTrust_RSA_Certification_Authority.pem
lrwxrwxrwx. 1 root root    66 Jul 19 00:10 UTN_USERFirst_Email_Root_CA.pem -> ../../../usr/share/ca-certificates/UTN_USERFirst_Email_Root_CA.pem
lrwxrwxrwx. 1 root root    69 Jul 19 00:10 UTN_USERFirst_Hardware_Root_CA.pem -> ../../../usr/share/ca-certificates/UTN_USERFirst_Hardware_Root_CA.pem
lrwxrwxrwx. 1 root root    99 Jul 19 00:10 VeriSign_Class_3_Public_Primary_Certification_Authority_-_G4.pem -> ../../../usr/share/ca-certificates/VeriSign_Class_3_Public_Primary_Certification_Authority_-_G4.pem
lrwxrwxrwx. 1 root root    99 Jul 19 00:10 VeriSign_Class_3_Public_Primary_Certification_Authority_-_G5.pem -> ../../../usr/share/ca-certificates/VeriSign_Class_3_Public_Primary_Certification_Authority_-_G5.pem
lrwxrwxrwx. 1 root root    86 Jul 19 00:10 VeriSign_Universal_Root_Certification_Authority.pem -> ../../../usr/share/ca-certificates/VeriSign_Universal_Root_Certification_Authority.pem
lrwxrwxrwx. 1 root root    99 Jul 19 00:10 Verisign_Class_1_Public_Primary_Certification_Authority_-_G3.pem -> ../../../usr/share/ca-certificates/Verisign_Class_1_Public_Primary_Certification_Authority_-_G3.pem
lrwxrwxrwx. 1 root root    99 Jul 19 00:10 Verisign_Class_2_Public_Primary_Certification_Authority_-_G3.pem -> ../../../usr/share/ca-certificates/Verisign_Class_2_Public_Primary_Certification_Authority_-_G3.pem
lrwxrwxrwx. 1 root root    99 Jul 19 00:10 Verisign_Class_3_Public_Primary_Certification_Authority_-_G3.pem -> ../../../usr/share/ca-certificates/Verisign_Class_3_Public_Primary_Certification_Authority_-_G3.pem
lrwxrwxrwx. 1 root root    58 Jul 19 00:10 Visa_eCommerce_Root.pem -> ../../../usr/share/ca-certificates/Visa_eCommerce_Root.pem
lrwxrwxrwx. 1 root root    84 Jul 19 00:10 WellsSecure_Public_Root_Certificate_Authority.pem -> ../../../usr/share/ca-certificates/WellsSecure_Public_Root_Certificate_Authority.pem
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 WoSign.pem -> ../../../usr/share/ca-certificates/WoSign.pem
lrwxrwxrwx. 1 root root    51 Jul 19 00:10 WoSign_China.pem -> ../../../usr/share/ca-certificates/WoSign_China.pem
lrwxrwxrwx. 1 root root    59 Jul 19 00:10 XRamp_Global_CA_Root.pem -> ../../../usr/share/ca-certificates/XRamp_Global_CA_Root.pem
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 a760e1bd.0 -> ../../../usr/share/ca-certificates/a760e1bd.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 a8dee976.0 -> ../../../usr/share/ca-certificates/a8dee976.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 a94d09e5.0 -> ../../../usr/share/ca-certificates/a94d09e5.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 ad088e1d.0 -> ../../../usr/share/ca-certificates/ad088e1d.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 ae8153b9.0 -> ../../../usr/share/ca-certificates/ae8153b9.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 aee5f10d.0 -> ../../../usr/share/ca-certificates/aee5f10d.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 b0e59380.0 -> ../../../usr/share/ca-certificates/b0e59380.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 b1159c4c.0 -> ../../../usr/share/ca-certificates/b1159c4c.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 b13cc6df.0 -> ../../../usr/share/ca-certificates/b13cc6df.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 b1b8a7f3.0 -> ../../../usr/share/ca-certificates/b1b8a7f3.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 b204d74a.0 -> ../../../usr/share/ca-certificates/b204d74a.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 b66938e9.0 -> ../../../usr/share/ca-certificates/b66938e9.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 b727005e.0 -> ../../../usr/share/ca-certificates/b727005e.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 b7a5b843.0 -> ../../../usr/share/ca-certificates/b7a5b843.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 ba89ed3b.0 -> ../../../usr/share/ca-certificates/ba89ed3b.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 bb2d49a0.0 -> ../../../usr/share/ca-certificates/bb2d49a0.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 bd1910d4.0 -> ../../../usr/share/ca-certificates/bd1910d4.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 c01cdfa2.0 -> ../../../usr/share/ca-certificates/c01cdfa2.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 c089bbbd.0 -> ../../../usr/share/ca-certificates/c089bbbd.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 c0ff1f52.0 -> ../../../usr/share/ca-certificates/c0ff1f52.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 c28a8a30.0 -> ../../../usr/share/ca-certificates/c28a8a30.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 c47d9980.0 -> ../../../usr/share/ca-certificates/c47d9980.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 c5d3212a.0 -> ../../../usr/share/ca-certificates/c5d3212a.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 c5e082db.0 -> ../../../usr/share/ca-certificates/c5e082db.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 c679bc3f.0 -> ../../../usr/share/ca-certificates/c679bc3f.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 c99398f3.0 -> ../../../usr/share/ca-certificates/c99398f3.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 c9f83a1c.0 -> ../../../usr/share/ca-certificates/c9f83a1c.0
lrwxrwxrwx. 1 root root    54 Jul 19 00:10 ca-certificates.crt -> ../../../usr/share/ca-certificates/ca-certificates.crt
-rw-r--r--. 1 root root     0 Jul 27 20:40 ca.pem
-rw-r--r--. 1 root root  1232 Jul 27 20:40 ca.pem.enc
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 ca6e4ad9.0 -> ../../../usr/share/ca-certificates/ca6e4ad9.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 cb59f961.0 -> ../../../usr/share/ca-certificates/cb59f961.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 cbeee9e2.0 -> ../../../usr/share/ca-certificates/cbeee9e2.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 cbf06781.0 -> ../../../usr/share/ca-certificates/cbf06781.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 cc450945.0 -> ../../../usr/share/ca-certificates/cc450945.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 cd58d51e.0 -> ../../../usr/share/ca-certificates/cd58d51e.0
lrwxrwxrwx. 1 root root    55 Jul 19 00:10 certSIGN_ROOT_CA.pem -> ../../../usr/share/ca-certificates/certSIGN_ROOT_CA.pem
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 cfa1c2ee.0 -> ../../../usr/share/ca-certificates/cfa1c2ee.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 d4dae3dd.0 -> ../../../usr/share/ca-certificates/d4dae3dd.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 d6325660.0 -> ../../../usr/share/ca-certificates/d6325660.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 d7e8dc79.0 -> ../../../usr/share/ca-certificates/d7e8dc79.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 d853d49e.0 -> ../../../usr/share/ca-certificates/d853d49e.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 d957f522.0 -> ../../../usr/share/ca-certificates/d957f522.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 dc45b0bd.0 -> ../../../usr/share/ca-certificates/dc45b0bd.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 dd8e9d41.0 -> ../../../usr/share/ca-certificates/dd8e9d41.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 e113c810.0 -> ../../../usr/share/ca-certificates/e113c810.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 e18bfb83.0 -> ../../../usr/share/ca-certificates/e18bfb83.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 e2799e36.0 -> ../../../usr/share/ca-certificates/e2799e36.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 e36a6752.0 -> ../../../usr/share/ca-certificates/e36a6752.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 e536d871.0 -> ../../../usr/share/ca-certificates/e536d871.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 e73d606e.0 -> ../../../usr/share/ca-certificates/e73d606e.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 e8de2f56.0 -> ../../../usr/share/ca-certificates/e8de2f56.0
lrwxrwxrwx. 1 root root    72 Jul 19 00:10 ePKI_Root_Certification_Authority.pem -> ../../../usr/share/ca-certificates/ePKI_Root_Certification_Authority.pem
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 ee1365c0.0 -> ../../../usr/share/ca-certificates/ee1365c0.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 ee64a828.0 -> ../../../usr/share/ca-certificates/ee64a828.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 eed8c118.0 -> ../../../usr/share/ca-certificates/eed8c118.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 ef954a4e.0 -> ../../../usr/share/ca-certificates/ef954a4e.0
-rw-r--r--. 1 root root     0 Jul 27 20:40 etcd-client-key.pem
-rw-r--r--. 1 root root  1841 Jul 27 20:40 etcd-client-key.pem.enc
-rw-r--r--. 1 root root     0 Jul 27 20:40 etcd-client.pem
-rw-r--r--. 1 root root  1256 Jul 27 20:40 etcd-client.pem.enc
-rw-r--r--. 1 root root     0 Jul 27 20:40 etcd-key.pem
-rw-r--r--. 1 root root  1841 Jul 27 20:40 etcd-key.pem.enc
-rw-r--r--. 1 root root     0 Jul 27 20:40 etcd.pem
-rw-r--r--. 1 root root  1301 Jul 27 20:40 etcd.pem.enc
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 f060240e.0 -> ../../../usr/share/ca-certificates/f060240e.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 f081611a.0 -> ../../../usr/share/ca-certificates/f081611a.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 f30dd6ad.0 -> ../../../usr/share/ca-certificates/f30dd6ad.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 f3377b1b.0 -> ../../../usr/share/ca-certificates/f3377b1b.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 f387163d.0 -> ../../../usr/share/ca-certificates/f387163d.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 f38a011e.0 -> ../../../usr/share/ca-certificates/f38a011e.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 f39fc864.0 -> ../../../usr/share/ca-certificates/f39fc864.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 f90208f7.0 -> ../../../usr/share/ca-certificates/f90208f7.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 fc5a8f99.0 -> ../../../usr/share/ca-certificates/fc5a8f99.0
lrwxrwxrwx. 1 root root    45 Jul 19 00:10 fe8a2cd8.0 -> ../../../usr/share/ca-certificates/fe8a2cd8.0
lrwxrwxrwx. 1 root root    61 Jul 19 00:10 thawte_Primary_Root_CA.pem -> ../../../usr/share/ca-certificates/thawte_Primary_Root_CA.pem
lrwxrwxrwx. 1 root root    66 Jul 19 00:10 thawte_Primary_Root_CA_-_G2.pem -> ../../../usr/share/ca-certificates/thawte_Primary_Root_CA_-_G2.pem
lrwxrwxrwx. 1 root root    66 Jul 19 00:10 thawte_Primary_Root_CA_-_G3.pem -> ../../../usr/share/ca-certificates/thawte_Primary_Root_CA_-_G3.pem

mumoshu commented 7 years ago

Would you mind sharing us a full output from journalctl -u decrypt-assets.service?

TigerC10 commented 7 years ago

Here it is:

Jul 27 21:48:53 ip-10-0-0-207.ec2.internal systemd[1]: Starting decrypt etcd2 tls assets using amazon kms...
Jul 27 21:48:54 ip-10-0-0-207.ec2.internal rkt[995]: [   70.220119] awscli[5]: decrypting tls assets
Jul 27 21:48:54 ip-10-0-0-207.ec2.internal rkt[995]: [   70.220931] awscli[5]: decrypting /etc/ssl/certs/ca.pem.enc
Jul 27 21:48:55 ip-10-0-0-207.ec2.internal rkt[995]: [   70.573543] awscli[5]: An error occurred (AccessDeniedException) when calling the Decrypt operation: The ciphertext refers to a customer master key that does not exist, does not exist in this region, or you are not allowed to access.
Jul 27 21:48:55 ip-10-0-0-207.ec2.internal rkt[995]: [   70.600507] awscli[5]: decrypting /etc/ssl/certs/etcd-client-key.pem.enc
Jul 27 21:48:55 ip-10-0-0-207.ec2.internal rkt[995]: [   70.927699] awscli[5]: An error occurred (AccessDeniedException) when calling the Decrypt operation: The ciphertext refers to a customer master key that does not exist, does not exist in this region, or you are not allowed to access.
Jul 27 21:48:55 ip-10-0-0-207.ec2.internal rkt[995]: [   70.953341] awscli[5]: decrypting /etc/ssl/certs/etcd-client.pem.enc
Jul 27 21:48:55 ip-10-0-0-207.ec2.internal rkt[995]: [   71.294399] awscli[5]: An error occurred (AccessDeniedException) when calling the Decrypt operation: The ciphertext refers to a customer master key that does not exist, does not exist in this region, or you are not allowed to access.
Jul 27 21:48:55 ip-10-0-0-207.ec2.internal rkt[995]: [   71.319202] awscli[5]: decrypting /etc/ssl/certs/etcd-key.pem.enc
Jul 27 21:48:56 ip-10-0-0-207.ec2.internal rkt[995]: [   71.654703] awscli[5]: An error occurred (AccessDeniedException) when calling the Decrypt operation: The ciphertext refers to a customer master key that does not exist, does not exist in this region, or you are not allowed to access.
Jul 27 21:48:56 ip-10-0-0-207.ec2.internal rkt[995]: [   71.686200] awscli[5]: decrypting /etc/ssl/certs/etcd.pem.enc
Jul 27 21:48:56 ip-10-0-0-207.ec2.internal rkt[995]: [   72.006589] awscli[5]: An error occurred (AccessDeniedException) when calling the Decrypt operation: The ciphertext refers to a customer master key that does not exist, does not exist in this region, or you are not allowed to access.
Jul 27 21:48:56 ip-10-0-0-207.ec2.internal rkt[995]: [   72.032993] awscli[5]: done.
Jul 27 21:48:56 ip-10-0-0-207.ec2.internal systemd[1]: Started decrypt etcd2 tls assets using amazon kms.
Jul 27 21:48:56 ip-10-0-0-207.ec2.internal rkt[1144]: "48e5fed4-f673-4d6b-971e-bef8065844f2"

redbaron commented 7 years ago

did you copy certs from another cluster in another region? or changed region in cluster.yaml after certs were generated? or deleted KMS key ?

TigerC10 commented 7 years ago

KMS key is there, but I did a list-grants on the KMS key and nothing gets returned. Not sure why that is.

Did not change the region after certs were generated, but I think I did have to change the availabilityZone because I typo'd the letter suffix. Should I regenerate the certs?

redbaron commented 7 years ago

your safest bet is to regenerate certs, delete cloudformation stack and do kube-aws up cleanly again

TigerC10 commented 7 years ago

Yep, apparently certs care about the AZ they were generated for too. After regenerating certs, everything came up just fine. I don't know how long it takes to download containers, but it's been about an hour and I can't seem to access the controller interface. kube-aws created an elastic IP, so I used that for my DNS a record since I'm not using Route53. The a record has propagated, but I still get:

$ kube-aws status
WARN: the worker node pool "nodepool1" is associated to a k8s API endpoint behind the DNS name "x.example.com" managed by YOU!
Please never point the DNS record for it to a different k8s cluster, especially when the name is a "stable" one which is shared among multiple k8s clusters for achieving blue-green deployments of k8s clusters!
kube-aws can't save users from mistakes like that
Cluster Name:       test-kube
Controller DNS Names:   test-kube-APIEndpo-XXXXXXXXXXXX-XXXXXXXXXX.us-east-1.elb.amazonaws.com

$ kubectl --kubeconfig=kubeconfig get nodes
Unable to connect to the server: dial tcp XXX.XXX.XXX.XXX:443: i/o timeout

$ ping -c 1 x.example.com                                                                                                                                                                                                                                
PING x.example.com (XXX.XXX.XXX.XXX): 56 data bytes

--- x.example.com ping statistics ---
1 packets transmitted, 0 packets received, 100.0% packet loss

mumoshu commented 7 years ago

@TigerC10 I'd suggest you to configure your DNS so that x.example.com is mapped to the DNS name of the controller ELB(not etcd node's EIP) via a CNAME record.

TigerC10 commented 7 years ago

Ugh, the instructions said to use an A record, switching it to the CNAME for the ELB gives me:

$ kubectl --kubeconfig=kubeconfig get nodes
NAME                         STATUS    AGE       VERSION
ip-10-0-0-176.ec2.internal   Ready     1h        v1.6.3+coreos.0
ip-10-0-0-235.ec2.internal   Ready     32s       v1.6.3+coreos.0

So, progress! Still not seeing the web interface when I try to visit the dashboard in the browser. Just a bland "Unauthorized" message.

mumoshu commented 7 years ago

@TigerC10 Sorry, our doc must be out-dated for that then 😢

The dashboard isn't publicly visible. Generally, you need to run kubectl proxy and then access it via localhost as explained in the dashboard doc https://github.com/kubernetes/dashboard#usage

TigerC10 commented 7 years ago

That's what I was thinking too, so I did try it and got this:

$ kubectl proxy

Starting to serve on 127.0.0.1:8001I0727 20:25:29.239684   51096 logs.go:41] http: proxy error: dial tcp [::1]:8080: getsockopt: connection refused
I0727 20:25:32.018430   51096 logs.go:41] http: proxy error: dial tcp [::1]:8080: getsockopt: connection refused

TigerC10 commented 7 years ago

Derp, just figured it out. Needed to do this instead:

$ kubectl --kubeconfig=kubeconfig proxy

kubernetes-retired / kube-aws

Kube-aws v0.9.6 Failed to create Etcd0 #709

198 -- issue with eu-west-2

679, #611 -- custom etcd config issues