search

containers / podman

Podman: A tool for managing OCI containers and pods.
https://podman.io
Apache License 2.0
23.76k stars 2.42k forks source link

libpod passes K8S_* environment variables to CNIs when not running under Kubernetes #12083

Closed ghost closed 3 years ago

ghost commented 3 years ago

/kind bug

Description

All containers launched via Podman/libpod pass a K8S_* environment variables, like K8S_POD_NAME to the underlying CNI. This causes CNI's like Calico to assume they are running within a Kubernetes context when they are not, which breaks things. It's not an unreasonable assumption either.

Podman shouldn't be setting these envvars, it's not a Kubernetes application. At the very least it should at least check it is also running in a Kubernetes context first if there is a use case for that.

Steps to reproduce the issue:

  1. Create a new CNI conflist
[root@calico-lab1 / ]# cat /etc/cni/net.d/10-calico.conflist 
{
    "name": "calico",
    "cniVersion": "0.3.1",
    "plugins": [
        {
            "type": "calico",
            "etcd_endpoints": "http://etcd.foo.bar.com:2379",
            "log_level": "debug",
            "ipam": {
                "type": "calico-ipam"
            }
        }
   ]
}
  1. Run and get the issue: (relevant snippet showing the arguments received by the CNI from Podman) 
    2020-12-07 18:11:16.982 [DEBUG][16711] utils.go 425: Getting WEP identifiers with arguments: IgnoreUnknown=1;IgnoreUnknown=1;K8S_POD_NAMESPACE=dazzling_zhukovsky;K8S_POD_NAME=dazzling_zhukovsky;K8S_POD_INFRA_CONTAINER_ID=18f054fb89c3c5e1e38907286b87ab2732e2b1c7f8f54ff3d1e0dc5be0ea0613, for node calico-lab1.foo.com
[root@calico-lab1 ]# podman run -it --network calico busybox /bin/sh
2020-12-07 18:11:16.982 [DEBUG][16711] plugin.go 118: /var/lib/calico/nodename exists
2020-12-07 18:11:16.982 [DEBUG][16711] utils.go 59: Read node name from file: calico-lab1.foo.com
2020-12-07 18:11:16.982 [DEBUG][16711] utils.go 69: Using node name calico-lab1.foo.com
2020-12-07 18:11:16.982 [DEBUG][16711] utils.go 425: Getting WEP identifiers with arguments: IgnoreUnknown=1;IgnoreUnknown=1;K8S_POD_NAMESPACE=dazzling_zhukovsky;K8S_POD_NAME=dazzling_zhukovsky;K8S_POD_INFRA_CONTAINER_ID=18f054fb89c3c5e1e38907286b87ab2732e2b1c7f8f54ff3d1e0dc5be0ea0613, for node calico-lab1.foo.com
2020-12-07 18:11:16.982 [DEBUG][16711] utils.go 426: Loaded k8s arguments: {{true} <nil> dazzling_zhukovsky dazzling_zhukovsky 18f054fb89c3c5e1e38907286b87ab2732e2b1c7f8f54ff3d1e0dc5be0ea0613}
2020-12-07 18:11:16.982 [INFO][16711] plugin.go 130: Extracted identifiers EndpointIDs=&utils.WEPIdentifiers{Namespace:"dazzling_zhukovsky", WEPName:"", WorkloadEndpointIdentifiers:names.WorkloadEndpointIdentifiers{Node:"calico-lab1.foo.com", Orchestrator:"k8s", Endpoint:"eth0", Workload:"", Pod:"dazzling_zhukovsky", ContainerID:"18f054fb89c3c5e1e38907286b87ab2732e2b1c7f8f54ff3d1e0dc5be0ea0613"}}
2020-12-07 18:11:16.982 [DEBUG][16711] load.go 71: Loading config from environment
2020-12-07 18:11:16.982 [DEBUG][16711] client.go 30: Using datastore type 'etcdv3'
2020-12-07 18:11:16.983 [DEBUG][16711] etcdv3.go 356: Processing Get request model-etcdKey=ClusterInformation(default) rev=""
2020-12-07 18:11:16.983 [DEBUG][16711] etcdv3.go 380: Calling Get on etcdv3 client etcdv3-etcdKey="/calico/resources/v3/projectcalico.org/clusterinformations/default" model-etcdKey=ClusterInformation(default) rev=""
2020-12-07 18:11:16.985 [DEBUG][16711] etcdv3.go 398: Processing List request list-interface=WorkloadEndpoint rev=""
2020-12-07 18:11:16.985 [DEBUG][16711] etcdv3.go 456: List options is a name prefix, don't add a / to the path list-interface=WorkloadEndpoint rev=""
2020-12-07 18:11:16.985 [DEBUG][16711] etcdv3.go 413: Calling Get on etcdv3 client etcdv3-etcdKey="/calico/resources/v3/projectcalico.org/workloadendpoints/dazzling_zhukovsky/calico--lab1.foo.com-k8s-dazzling_zhukovsky-" list-interface=WorkloadEndpoint rev="" 
2020-12-07 18:11:16.985 [DEBUG][16711] etcdv3.go 419: Processing response from etcdv3 etcdv3-etcdKey="/calico/resources/v3/projectcalico.org/workloadendpoints/dazzling_zhukovsky/calico--lab1.foo.com-k8s-dazzling_zhukovsky-" list-interface=WorkloadEndpoint numResults=0 rev="" 
2020-12-07 18:11:16.985 [DEBUG][16711] plugin.go 180: Retrieved list of endpoints: &{{WorkloadEndpointList projectcalico.org/v3} { 405  <nil>} []} ContainerID="18f054fb89c3c5e1e38907286b87ab2732e2b1c7f8f54ff3d1e0dc5be0ea0613" Namespace="dazzling_zhukovsky" Pod="dazzling_zhukovsky" WorkloadEndpoint="calico--lab1.foo.com-k8s-dazzling_zhukovsky-"
2020-12-07 18:11:16.985 [INFO][16711] k8s.go 61: Extracted identifiers for CmdAddK8s ContainerID="18f054fb89c3c5e1e38907286b87ab2732e2b1c7f8f54ff3d1e0dc5be0ea0613" Namespace="dazzling_zhukovsky" Pod="dazzling_zhukovsky" WorkloadEndpoint="calico--lab1.foo.com-k8s-dazzling_zhukovsky-eth0"
time="2020-12-07T18:11:16Z" level=error msg="Error adding network: invalid configuration: no configuration has been provided"
time="2020-12-07T18:11:16Z" level=error msg="Error while adding pod to CNI network \"calico\": invalid configuration: no configuration has been provided"
Error: error configuring network namespace for container 18f054fb89c3c5e1e38907286b87ab2732e2b1c7f8f54ff3d1e0dc5be0ea0613: invalid configuration: no configuration has been provided

Describe the results you received:

Container fails to be created as the CNI is being passed K8S variables when not in a K8s environment

Describe the results you expected:

Podman should not be setting K8S specific variables

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

$ podman --version
podman version 3.3.1

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.22.3
  cgroupControllers: []
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: Unknown
    path: <snip>
    version: 'conmon version 2.0.29, commit: '
  cpus: 28
  distribution:
    distribution: '"rhel"'
    version: "7.9"
  eventLogger: journald
  hostname: <snip>
  idMappings: <snip>
  kernel: 3.10.0-1160.42.2.el7.x86_64
  linkmode: dynamic
  memFree: 185802428416
  memTotal: 269088026624
  ociRuntime:
    name: crun
    package: Unknown
    path: <snip>
    version: |-
      crun version 1.0
      commit: 1.0
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: <snip>
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: ""
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    <snip>
    package: Unknown
    version: |-
      slirp4netns version 1.1.12
      commit: 7a104a101aa3278a2152351a082a6df71f57c9a3
      libslirp: 4.6.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.1
  swapFree: 8589930496
  swapTotal: 8589930496
  uptime: 49h 5m 46.8s (Approximately 2.04 days)
registries:
<snip>
store:
<snip>
version:
  APIVersion: 3.3.1
  Built: 315532800
  BuiltTime: Mon Dec 31 19:00:00 1979
  GitCommit: ""
  GoVersion: go1.16.7
  OsArch: linux/amd64
  Version: 3.3.1

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

Yes

Additional environment details (AWS, VirtualBox, physical, etc.): Physical

ghost commented 3 years ago

Sorry I meant to highlight the code snippet where I think this happens: https://github.com/containers/podman/blob/381ab48c2586a96932701a1fe270ebd2c32a04d4/libpod/network/cni/run.go#L192-L205

Luap99 commented 3 years ago

This is new code which is not yet included in any released podman version. As you can see it only sets K8S_POD_NAME because this is used by the dnsname plugin. Looking at your linked code it needs both K8S_POD_NAMESPACE and K8S_POD_NAME to bet set so with the new networking code it will work.

The old code used OCICNI which sets all K8s vars: https://github.com/cri-o/ocicni/blob/b38844812e649006b07a9063e2e7c4fa1a97d9b0/pkg/ocicni/ocicni.go#L803-L808

The next version which includes the new network code will be 4.0. It will be released in January/February.

ghost commented 3 years ago

That's good news - thanks !

Luap99 commented 3 years ago

Given that this is fixed in the main branch I am going to close this issue. I think you can overwrite the env var to an empty string if you set this CNI_ARGS=K8S_POD_NAME=