[kubernetes] container does not start due to "failed: open /run/systemd/resolve/resolv.conf: no such file or directory"

[stepanblyschak]

Description

Container does not start due to "failed: open /run/systemd/resolve/resolv.conf: no such file or directory"

Steps to reproduce the issue:

1. Build master image cff37b00 with INCLUDE_KUBERNETES=y
2. Run this image on the switch
3. config kubernetes server ip <VIP>
4. Apply this manifest:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: snmp-sv2
spec:
  template:
    metadata:
      labels:
        name: snmp
    spec:
      hostname: sonic
      hostNetwork: true
      containers:
      - name: snmp
        image: stepanblischak/docker-snmp:master_0
        name: snmp
        tty: true
        env:
        - name: OWNER
          value: "kube"
        securityContext:
          privileged: true
        volumeMounts:
        - name: sonic
          mountPath: /etc/sonic
          readOnly: true
        - name: scripts
          mountPath: /usr/share/sonic/scripts
          readOnly: true
        - name: redis
          mountPath: /var/run/redis
          readOnly: false
        - name: redis-chassis
          mountPath: /var/run/redis-chassis
          readOnly: true
        - name: hwsku
          mountPath: /usr/share/sonic/hwsku/
          readOnly: true
        - name: platform
          mountPath: /usr/share/sonic/platform/
          readOnly: true
        imagePullPolicy: IfNotPresent
      volumes:
      - name: sonic
        hostPath:
          path: /etc/sonic
      - name: redis
        hostPath:
          path: /var/run/redis
      - name: redis-chassis
        hostPath:
          path: /var/run/redis-chassis
      - name: scripts
        hostPath:
          path: /usr/share/sonic/scripts
      - name: hwsku
        hostPath:
          path: /usr/share/sonic/device/x86_64-mlnx_msn4410-r0/ACS-MSN4410
      - name: platform
        hostPath:
          path: /usr/share/sonic/device/x86_64-mlnx_msn4410-r0/
      nodeSelector:
        snmp_enabled: "true"
        lldp_enabled: "true"
  selector:
    matchLabels:
      name: snmp
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1

Describe the results you received:

Jun  8 10:19:57.162893 r-bulldog-03 INFO kubelet[23191]: E0608 10:19:57.161271   23191 kuberuntime_sandbox.go:41] GeneratePodSandboxConfig for pod "snmp-sv2-zv6mw_default(20d56030-58f3-4cb1-8b50-ccd4dd2c91e4
)" failed: open /run/systemd/resolve/resolv.conf: no such file or directory
Jun  8 10:19:57.163095 r-bulldog-03 INFO kubelet[23191]: E0608 10:19:57.161345   23191 kuberuntime_manager.go:727] createPodSandbox for pod "snmp-sv2-zv6mw_default(20d56030-58f3-4cb1-8b50-ccd4dd2c91e4)" failed: open /run/systemd/resolve/resolv.conf: no such file or directory
Jun  8 10:19:57.163334 r-bulldog-03 INFO kubelet[23191]: E0608 10:19:57.161466   23191 pod_workers.go:191] Error syncing pod 20d56030-58f3-4cb1-8b50-ccd4dd2c91e4 ("snmp-sv2-zv6mw_default(20d56030-58f3-4cb1-8
b50-ccd4dd2c91e4)"), skipping: failed to "CreatePodSandbox" for "snmp-sv2-zv6mw_default(20d56030-58f3-4cb1-8b50-ccd4dd2c91e4)" with CreatePodSandboxError: "GeneratePodSandboxConfig for pod \"snmp-sv2-zv6mw_d
efault(20d56030-58f3-4cb1-8b50-ccd4dd2c91e4)\" failed: open /run/systemd/resolve/resolv.conf: no such file or directory"

Describe the results you expected:

Container is deployed successfully. Local container is stopped, kube managed container is starting.

Output of show version:

SONiC Software Version: SONiC.master.0-cff37b00
Distribution: Debian 10.9
Kernel: 4.19.0-12-2-amd64
Build commit: cff37b00
Build date: Wed Jun  2 08:43:11 UTC 2021
Built by: stepanb@r-build-sonic03

Platform: x86_64-mlnx_msn2100-r0
HwSKU: ACS-MSN2100
ASIC: mellanox
ASIC Count: 1
Serial Number: MT1752X06330
Uptime: 10:23:04 up  1:02,  1 user,  load average: 4.15, 3.75, 3.24

Docker images:
REPOSITORY                    TAG                 IMAGE ID            SIZE
docker-sonic-mgmt-framework   latest              859e14c40565        628MB
docker-sonic-mgmt-framework   master.0-cff37b00   859e14c40565        628MB
docker-sonic-telemetry        latest              7b1f3f4e07c3        496MB
docker-sonic-telemetry        master.0-cff37b00   7b1f3f4e07c3        496MB
docker-snmp                   latest              622a3804fe2d        448MB
docker-snmp                   master.0-cff37b00   622a3804fe2d        448MB
docker-platform-monitor       latest              be8186b6e493        704MB
docker-platform-monitor       master.0-cff37b00   be8186b6e493        704MB
docker-teamd                  latest              1fc7bc388609        418MB
docker-teamd                  master.0-cff37b00   1fc7bc388609        418MB
docker-nat                    latest              fa8225f12bc7        421MB
docker-nat                    master.0-cff37b00   fa8225f12bc7        421MB
docker-router-advertiser      latest              e2424c15620e        407MB
docker-router-advertiser      master.0-cff37b00   e2424c15620e        407MB
docker-lldp                   latest              16d78b4c39b2        447MB
docker-lldp                   master.0-cff37b00   16d78b4c39b2        447MB
docker-database               latest              1631b8041bfb        407MB
docker-database               master.0-cff37b00   1631b8041bfb        407MB
docker-orchagent              latest              75e2440bb81b        436MB
docker-orchagent              master.0-cff37b00   75e2440bb81b        436MB
docker-macsec                 latest              2dd475f51b8e        421MB
docker-macsec                 master.0-cff37b00   2dd475f51b8e        421MB
docker-fpm-frr                latest              3adf6fa9efdb        436MB
docker-fpm-frr                master.0-cff37b00   3adf6fa9efdb        436MB
docker-syncd-mlnx             latest              d1780d0d6d23        672MB
docker-syncd-mlnx             master.0-cff37b00   d1780d0d6d23        672MB
docker-sflow                  latest              a6170d41215c        419MB
docker-sflow                  master.0-cff37b00   a6170d41215c        419MB
docker-dhcp-relay             latest              e4c458cdcb0c        414MB
docker-dhcp-relay             master.0-cff37b00   e4c458cdcb0c        414MB
k8s.gcr.io/pause              3.2                 80d28bedfe5d        683kB

Output of show techsupport:

(paste your output here or download and attach the file here )

Additional information you deem important (e.g. issue happens only occasionally):

There is a W/A:

sudo mkdir -p /run/systemd/resolve/
sudo touch /run/systemd/resolve/resolv.conf
# Altough does not preserve this after reboot

sonic_dump_r-bulldog-03_20210608_102022.tar.gz

[zhangyanzhao]

@renukamanavalan

[renukamanavalan]

All we need is admin@str-s6000-acs-13:/etc/default$ cat kubelet KUBELET_EXTRA_ARGS="--resolv-conf=/etc/resolv.conf"

Can you please raise the PR?