/etc/localtime mount error

[akaliazin]

Hi team!

When installing ks-installer as recommended -

$ kubectl apply -f https://github.com/kubesphere/ks-installer/releases/download/v3.1.1/kubesphere-installer.yaml

I get an error:

$ k8 get pod -n kubesphere-system --watch
NAME                            READY   STATUS             RESTARTS   AGE
ks-installer-54c6bcf76b-gcnhc   0/1     CrashLoopBackOff   1          80s
ks-installer-54c6bcf76b-gcnhc   0/1     RunContainerError   2          89s

It turns out that /etc/localtime is not mounted:

$ k8 describe pod ks-installer-54c6bcf76b-gcnhc -n kubesphere-system
Name:         ks-installer-54c6bcf76b-gcnhc
Namespace:    kubesphere-system
Priority:     0
Node:         k3d-devops-toolkit-server-0/172.18.0.2
...
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       StartError
      Message:      failed to create containerd task: OCI runtime create failed: container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: rootfs_linux.go:59: mounting "/etc/localtime" to rootfs at "/run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/cc0e1a8703a2d0398c16f3036fa492576960cef9e300e43d29096dd55a680531/rootfs/usr/share/zoneinfo/Asia/Shanghai" caused: not a directory: unknown
      Exit Code:    128

If I comment out the volume mount of /etc/localtime, then ks-installer works and the localtime symlink appears hardcoded:

<<K9s-Shell>> Pod: kubesphere-system/ks-installer-7776556d6b-55nv9 | Container: installer
bash-5.1$ ls -la /etc/localtime
lrwxrwxrwx    1 root     root            33 Jul  6 15:43 /etc/localtime -> /usr/share/zoneinfo/Asia/Shanghai

hope it helps Andrey

[pixiake]

https://github.com/kubesphere/ks-installer/blob/2f48af0ff0cdfa6bbde4e3c223dfdd4f50f665eb/deploy/kubesphere-installer.yaml#L280 In order to ensure the timezone in container as same as node, /etc/localtime of the node is mounted in ks-installer‘s yaml.

If this file doesn‘’t exist, you can link your timezone to /etc/localtime.

example: mine is Asia/Shanghai, so i did: ln -s /usr/share/zoneinfo/Asia/Shanghai /etc/localtime

I notice that the localtime symlink is hardcoded in the Dockerfile，I will delete it. Thanks !

[akaliazin]

I know how to manage localtime, thanks. The issue is in your hardcoded symlink. Because when mounted it points at the non-existent file on my host. there is no /usr/share/zoneinfo/Asia/Shanghai there better not to use symlink there at all

[pixiake]

OK, I got it, I will delete the localtime symlink from the Dockerfile, thanks.

[Santhosh-Radhakrishna]

how do we deal with this if we are deploying it on EKS ? this still would fail i guess. in my case i caught up with logging side car with this issue

│ Events:                                                                                                                                                                                                          │
│   Type     Reason     Age                 From               Message                                                                                                                                             │
│   ----     ------     ----                ----               -------                                                                                                                                             │
│   Normal   Scheduled  15m                 default-scheduler  Successfully assigned kubesphere-logging-system/logsidecar-injector-deploy-fff7577fd-j9c25 to ip-10-0-14-25.eu-west-1.compute.internal              │
│   Normal   Pulling    15m                 kubelet            Pulling image "kubesphere/log-sidecar-injector:v1.2.0"                                                                                              │
│   Normal   Pulled     15m                 kubelet            Successfully pulled image "kubesphere/log-sidecar-injector:v1.2.0" in 1.89s (1.89s including waiting). Image size: 10656572 bytes.                  │
│   Normal   Created    15m                 kubelet            Created container logsidecar-injector                                                                                                               │
│   Normal   Started    15m                 kubelet            Started container logsidecar-injector                                                                                                               │
│   Normal   Pulling    15m                 kubelet            Pulling image "jimmidyson/configmap-reload:v0.7.1"                                                                                                  │
│   Normal   Pulled     15m                 kubelet            Successfully pulled image "jimmidyson/configmap-reload:v0.7.1" in 1.181s (1.181s including waiting). Image size: 3935101 bytes.                     │
│   Normal   Created    14m (x4 over 15m)   kubelet            Created container config-reloader                                                                                                                   │
│   Normal   Pulled     14m (x3 over 15m)   kubelet            Container image "jimmidyson/configmap-reload:v0.7.1" already present on machine                                                                     │
│   Warning  Failed     14m (x4 over 15m)   kubelet            Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container proce │
│ ss: error during container init: error mounting "/etc/localtime" to rootfs at "/etc/localtime": mount /etc/localtime:/etc/localtime (via /proc/self/fd/6), flags: 0x5001, data: context="system_u:object_r:data_ │
│ t:s0:c362,c594": not a directory: unknown                                                                                                                                                                        │
│   Warning  BackOff    16s (x70 over 15m)  kubelet            Back-off restarting failed container config-reloader in pod logsidecar-injector-deploy-fff7577fd-j9c25_kubesphere-logging-system(5386d5b4-161e-4f2f │
│ -9d44-96e4cc47c953)

[redscholar]

It seems to be related to the timezone configuration on your EKS node (possibly due to the absence of the /etc/localtime file). I found a method to change the timezone on the EKS official website. https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-time.html#change-time-zone-of-instance Or seek help from the EKS vendor.

[Santhosh-Radhakrishna]

but why only this while the same is handled as part of installer pod. it does not throw error in this case and this also runs on the same eks cluster on same nodes. The problem is only with log side car container

    spec:
      serviceAccountName: ks-installer
      containers:
      - name: installer
        image: kubesphere/ks-installer:v3.4.1
        imagePullPolicy: "IfNotPresent"
        resources:
          limits:
            cpu: "2"
            memory: 2Gi
          requests:
            cpu: 20m
            memory: 100Mi
        volumeMounts:
        - mountPath: /etc/localtime
          name: host-time
          readOnly: true
      volumes:
      - hostPath:
          path: /etc/localtime
          type: ""
        name: host-time

[redscholar]

@Santhosh-Radhakrishna Could you provide the complete pod information for both the problematic pod (log sidecar) and the non-problematic pod (ks-installer)?

[pixiake]

@Santhosh-Radhakrishna If you can connect to the node where log sidecar is located, you can check if /etc/localtime exists.