Error delegating ADD to CNI plugin:calico because:OS exec call failed:invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable

prnvkv commented 3 years ago

Is this a BUG REPORT or FEATURE REQUEST?: Issue:

What happened: While trying to install DANM using the deployment guide on k3s environment along with calico, webhook deployment fails with the below error:

Events:
  Type     Reason                  Age                        From     Message
  ----     ------                  ----                       ----     -------
  Warning  FailedCreatePodSandBox  8m56s (x56438 over 2d16h)  kubelet  (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "3b6e88fb9f9db7c0d97a0a0631ff40778072226e5fe1cf6424dc098d3efddcd7" network for pod "danm-webhook-deployment-74b8b8c599-6n2q4": networkPlugin cni failed to set up pod "danm-webhook-deployment-74b8b8c599-6n2q4_kube-system" network: CNI network could not be set up: CNI operation for network:calico failed with:CNI delegation failed due to error:Error delegating ADD to CNI plugin:calico because:OS exec call failed:invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable
  Normal   SandboxChanged          3m56s (x56728 over 2d16h)  kubelet  Pod sandbox changed, it will be killed and re-created.

What you expected to happen: Webhook deployment up and running How to reproduce it: Install k3s and deploy calico using: https://docs.projectcalico.org/getting-started/kubernetes/k3s/quickstart Follow deployment guide for DANM lightweight mode.

Anything else we need to know?:

# cat /etc/cni/net.d/10-calico.conf
{
  "name": "k8s-pod-network",
  "cniVersion": "0.3.1",
  "plugins":
    {
      "type": "calico",
      "datastore_type": "kubernetes",
      "mtu": 0,
      "nodename_file_optional": false,
      "log_level": "Info",
      "log_file_path": "/var/log/calico/cni/cni.log",
      "ipam": { "type": "calico-ipam", "assign_ipv4" : "true", "assign_ipv6" : "false"},
      "container_settings": {
          "allow_ip_forwarding": false
      },
      "policy": {
          "type": "k8s"
      },
      "kubernetes": {
          "k8s_api_root":"https://10.43.0.1:443",
          "kubeconfig": "/etc/cni/net.d/calico-kubeconfig"
      }
    }
}

# ls /etc/cni/net.d/calico-kubeconfig
/etc/cni/net.d/calico-kubeconfig

# cat /etc/cni/net.d/00-danm.conf
{
  "cniVersion": "0.3.1",
  "name": "meta_cni",
  "name_comment": "Mandatory parameter, but can be anything",
  "type": "danm",
  "type_comment": "Mandatory parameter according to CNI spec, MUST be set to danm",
  "kubeconfig": "/etc/rancher/k3s/k3s.yaml",
  "kubeconfig_comment": "Mandatory parameter, must point to a valid kubeconfig file containing the necessary RBAC setting for DANM's service account",
  "cniDir": "/etc/cni/net.d",
  "cniDir_comment": "Optional parameter, if defined CNI config files for static delegates are searched here. Default value is /etc/cni/net.d",
  "namingScheme": "awesome",
  "namingScheme_comment": "Optional parameter, if it is set to legacy container network interface names are set exactly to DanmNet.Spec.Options.container_prefix, otherwise prefix simply behaves as a prefix and is suffixed with a sequence ID. Default value is empty (e.g. not legacy)"
}

Environment:

DANM version (use danm -version):

Kubernetes version (use kubectl version):

# kubectl version
Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.4", GitCommit:"e87da0bd6e03ec3fea7933c4b5263d151aafd07c", GitTreeState:"clean", BuildDate:"2021-02-18T16:12:00Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.3+k3s1", GitCommit:"1d1f220fbee9cdeb5416b76b707dde8c231121f2", GitTreeState:"clean", BuildDate:"2021-07-22T20:52:14Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"linux/amd64"}

DANM configuration (K8s manifests, kubeconfig files, CNI config file):

# cat integration/bootstrap_networks/lightweight/calico.yaml
apiVersion: danm.k8s.io/v1
kind: DanmNet
metadata:
name: calico
namespace: kube-system
spec:
NetworkID: 10-calico
NetworkType: calico

OS (e.g. from /etc/os-release):

# cat /etc/os-release
NAME="Ubuntu"
VERSION="18.04.5 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.5 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

Kernel (e.g. uname -a):

# uname -a
Linux testuser-VirtualBox 5.4.0-81-generic #91~18.04.1-Ubuntu SMP Fri Jul 23 13:36:29 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Levovar commented 3 years ago

CNI chaining is not supported by DANM, see https://github.com/nokia/danm/issues/224 or https://github.com/nokia/danm/issues/135#issuecomment-525276610 having the plugins keyword in your config is totally unnecessary anyway as it only contains the config of one CNI

prnvkv commented 3 years ago

Thanks @Levovar :)

nokia / danm

Error delegating ADD to CNI plugin:calico because:OS exec call failed:invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable #260