CrashLoopBackOff on RHEL/CentOS Host with KOPS+Calico+crossSubnet

[tatobi]

The main problem is that the certificate path is different on CentOS / RHEL systems. On Debian based systems the root cert store is: '/etc/ssl/certs/ca-certificates.crt' while on CentOS/RHEL (way different but the root store is): '/etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt'. The POD does not find this path on them and crashing.

https://github.com/ottoyiu/k8s-ec2-srcdst/search?utf8=%E2%9C%93&q=ca-certificates.crt&type=

Maybe it is kops related issue, but there is no way to inform kops about different hostOS and this configuration options combination to use/find different certpath .

POD LOG:

ubuntu@ip-10-202-4-127:~$ kubectl describe pod k8s-ec2-srcdst-78f785ff98-gmvsx --namespace kube-system
Name:           k8s-ec2-srcdst-78f785ff98-gmvsx
Namespace:      kube-system
Node:           ip-10-202-41-237.eu-west-1.compute.internal/10.202.41.237
Start Time:     Wed, 24 Jan 2018 08:39:30 +0000
Labels:         k8s-app=k8s-ec2-srcdst
                pod-template-hash=3493419954
                role.kubernetes.io/networking=1
Annotations:    kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"kube-system","name":"k8s-ec2-srcdst-78f785ff98","uid":"f2b81a96-00e1-11e8-bd3d-02...
                scheduler.alpha.kubernetes.io/critical-pod=
Status:         Running
IP:             10.202.41.237
Controlled By:  ReplicaSet/k8s-ec2-srcdst-78f785ff98
Containers:
  k8s-ec2-srcdst:
    Container ID:  docker://4db42f2dc7081d3b99d040935f6443573de47cf936f6143373f90390aa716854
    Image:         ottoyiu/k8s-ec2-srcdst:v0.1.0
    Image ID:      docker-pullable://ottoyiu/k8s-ec2-srcdst@sha256:d156bd23fb1e584fabfded239fcdd3f9612ed16feb941856c21d94390afcc080
    Port:          <none>
    State:         Waiting
      Reason:      CrashLoopBackOff
    Last State:    Terminated
      Reason:      ContainerCannotRun
      Message:     oci runtime error: container_linux.go:247: starting container process caused "process_linux.go:359: container init caused \"rootfs_linux.go:54: mounting \\\"/etc/ssl/certs/ca-certificates.crt\\\" to rootfs \\\"/var/lib/docker/overlay/2123c332b0b9198cbcfc9d82f936a49674af607d8a7e388166b58d6a39616924/merged\\\" at \\\"/var/lib/docker/overlay/2123c332b0b9198cbcfc9d82f936a49674af607d8a7e388166b58d6a39616924/merged/etc/ssl/certs/ca-certificates.crt\\\" caused \\\"not a directory\\\"\""
: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type
      Exit Code:    127
      Started:      Wed, 24 Jan 2018 10:23:53 +0000
      Finished:     Wed, 24 Jan 2018 10:23:53 +0000
    Ready:          False
    Restart Count:  25
    Requests:
      cpu:     10m
      memory:  64Mi
    Environment:
      AWS_REGION:  eu-west-1
    Mounts:
      /etc/ssl/certs/ca-certificates.crt from ssl-certs (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from k8s-ec2-srcdst-token-gd9jn (ro)
Conditions:
  Type           Status
  Initialized    True 
  Ready          False 
  PodScheduled   True 
Volumes:
  ssl-certs:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/ssl/certs/ca-certificates.crt
    HostPathType:  
  k8s-ec2-srcdst-token-gd9jn:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  k8s-ec2-srcdst-token-gd9jn
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  node-role.kubernetes.io/master=
Tolerations:     CriticalAddonsOnly
                 node-role.kubernetes.io/master:NoSchedule
                 node.alpha.kubernetes.io/notReady:NoExecute for 300s
                 node.alpha.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason      Age                From                                                  Message
  ----     ------      ----               ----                                                  -------
  Warning  FailedSync  4m (x490 over 1h)  kubelet, ip-10-202-41-237.eu-west-1.compute.internal  Error syncing pod

---

[tatobi]

Submitted to KOPS: https://github.com/kubernetes/kops/issues/4331

[ottoyiu]

Will take a look at the Kops issue; seems like a problem with the manifest file defined there. Going to close this issue. Thanks for the report!