projectcalico / calico

Cloud native networking and network security
https://docs.tigera.io/calico/latest/about/
Apache License 2.0
5.88k stars 1.31k forks source link

Can't config static pod ip with calico in my k8s cluster #5820

Closed kiddingl closed 2 years ago

kiddingl commented 2 years ago

Expected Behavior

The pod can set static ip

Current Behavior

The fe pod can't set static ip

Possible Solution

Steps to Reproduce (for bugs)

  1. I get calico.yml from https://docs.projectcalico.org/manifests/calico.yaml, set CALICO_IPV4POOL_IPIP value is Never, set CALICO_IPV4POOL_CIDR vaule is 10.244.0.0/16
  2. Appply the yaml
    [root@app1 starx_main]# kubectl apply  -f yaml

    Tene get the pod

    [root@app1 starx_main]# kubectl  get po  -n kube-system
    NAME                                       READY   STATUS    RESTARTS   AGE
    calico-kube-controllers-6fd7b9848d-g9q6r   1/1     Running   0          26m
    calico-node-xdx29                          1/1     Running   0          32m
    coredns-5b6bdcfc79-dfxxk                   1/1     Running   0          5h13m
    [root@app1 starx_main]#
  3. set static ip for fe statefulset:

image

  1. Install the pod with helm,I get pod status is always containerCreating

image

  1. I describe the pod:
    [root@app1 starx_main]# kubectl describe  po fe-0
    Name:           fe-0
    Namespace:      default
    Priority:       0
    Node:           app1/10.22.33.71
    Start Time:     Wed, 30 Mar 2022 16:20:02 +0800
    Labels:         app.kubernetes.io/instance=fe
                app.kubernetes.io/name=fe
                controller-revision-hash=fe-78c97b9794
                statefulset.kubernetes.io/pod-name=fe-0
    Annotations:    cni.projectcalico.org/ipAddrs: ["10.224.200.20"]
                cni.projectcalico.org/podIP:
                cni.projectcalico.org/podIPs:
    Status:         Pending
    IP:
    IPs:            <none>
    Controlled By:  StatefulSet/fe
    Containers:
    fe:
    Container ID:
    Image:         registry.starx.com/starx-main/fe:test
    Image ID:
    Ports:         8030/TCP, 9010/TCP, 9020/TCP, 9030/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP, 0/TCP
    Command:
      bash
      new_start_fe.sh
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /data from data (rw)
      /opt/module/fe/conf/fe.conf from config-fe (rw,path="config-fe")
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-mjqkm (ro)
    Conditions:
    Type              Status
    Initialized       True
    Ready             False
    ContainersReady   False
    PodScheduled      True
    Volumes:
    config-fe:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      fe
    Optional:  false
    data:
    Type:          HostPath (bare host directory volume)
    Path:          /data/fe/data
    HostPathType:  DirectoryOrCreate
    kube-api-access-mjqkm:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
    QoS Class:                   BestEffort
    Node-Selectors:              <none>
    Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
    Events:
    Type     Reason                  Age                    From               Message
    ----     ------                  ----                   ----               -------
    Normal   Scheduled               27m                    default-scheduler  Successfully assigned default/fe-0 to app1
    Warning  FailedCreatePodSandBox  27m                    kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "576743cafbee78d5270c646bdbc0a230d8c872f2fa588ce8b43cea0380d226a5" network for pod "fe-0": networkPlugin cni failed to set up pod "fe-0_default" network: error getting IP from IPAM: The provided IP address is not in a configured pool
    Warning  FailedCreatePodSandBox  27m                    kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "f6dfa50a5bd983cdb80014e206d4bb28c4bfaf62261d9c89ad6231a6a9a94279" network for pod "fe-0": networkPlugin cni failed to set up pod "fe-0_default" network: error getting IP from IPAM: The provided IP address is not in a configured pool
    Warning  FailedCreatePodSandBox  27m                    kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "b39caf96b6cb734bcbb002c4acfdb9528f63944a19933917a3665f5bfb677c11" network for pod "fe-0": networkPlugin cni failed to set up pod "fe-0_default" network: error getting IP from IPAM: The provided IP address is not in a configured pool
    Warning  FailedCreatePodSandBox  27m                    kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "951a5732524dc88bfb12f8906983022befee6684a837a818c7e43df4624db2b3" network for pod "fe-0": networkPlugin cni failed to set up pod "fe-0_default" network: error getting IP from IPAM: The provided IP address is not in a configured pool
    Warning  FailedCreatePodSandBox  27m                    kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "89f61f4f2f372be319924c3d7bac241300c0430c181652fe27e7987681b29a13" network for pod "fe-0": networkPlugin cni failed to set up pod "fe-0_default" network: error getting IP from IPAM: The provided IP address is not in a configured pool
    Warning  FailedCreatePodSandBox  27m                    kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "efc4efcec09b09a38a40953bc5951585c02a965c7d0376de98688ea9c1c9659d" network for pod "fe-0": networkPlugin cni failed to set up pod "fe-0_default" network: error getting IP from IPAM: The provided IP address is not in a configured pool
    Warning  FailedCreatePodSandBox  27m                    kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "16c3038474706b283b377723aa5955338dedbc60b01b313fa3ebedefe4106c07" network for pod "fe-0": networkPlugin cni failed to set up pod "fe-0_default" network: error getting IP from IPAM: The provided IP address is not in a configured pool
    Warning  FailedCreatePodSandBox  27m                    kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "b926c427c7db509e21fcb1271287cc7794c97551f7ec31aaa3b49173e7f75196" network for pod "fe-0": networkPlugin cni failed to set up pod "fe-0_default" network: error getting IP from IPAM: The provided IP address is not in a configured pool
    Warning  FailedCreatePodSandBox  27m                    kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "2e2110bef1ab905d13642a10fe77606f0703d8469eefad0d87d964711d651516" network for pod "fe-0": networkPlugin cni failed to set up pod "fe-0_default" network: error getting IP from IPAM: The provided IP address is not in a configured pool
    Normal   SandboxChanged          12m (x513 over 27m)    kubelet            Pod sandbox changed, it will be killed and re-created.
    Warning  FailedCreatePodSandBox  2m29s (x836 over 27m)  kubelet            (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "b8005413c70ea31c246857f0945cc219155625e67e652b1baa9dde4267a7f818" network for pod "fe-0": networkPlugin cni failed to set up pod "fe-0_default" network: error getting IP from IPAM: The provided IP address is not in a configured pool
    [root@app1 starx_main]#
  2. I get the IPPOOL:
    
    [root@app1 starx_main]# kubectl  get IPPOOL -o yaml
    apiVersion: v1
    items:
    - apiVersion: crd.projectcalico.org/v1
    kind: IPPool
    metadata:
    annotations:
      projectcalico.org/metadata: '{"uid":"acb2fb27-44c5-47aa-8850-b68590000ffd","creationTimestamp":"2022-03-30T08:04:35Z"}'
    creationTimestamp: "2022-03-30T08:04:35Z"
    generation: 1
    name: default-ipv4-ippool
    resourceVersion: "92711"
    selfLink: /apis/crd.projectcalico.org/v1/ippools/default-ipv4-ippool
    uid: b1f2ae4d-ac99-4701-b80d-e5632d8874dc
    spec:
    allowedUses:
    - Workload
    - Tunnel
    blockSize: 26
    cidr: 10.244.0.0/16
    ipipMode: Never
    natOutgoing: true
    nodeSelector: all()
    vxlanMode: Never
    kind: List
    metadata:
    resourceVersion: ""
    selfLink: ""
    [root@app1 starx_main]# cat /etc/cni/net.d/10-calico.conflist
    {
    "name": "k8s-pod-network",
    "cniVersion": "0.3.1",
    "plugins": [
    {
      "type": "calico",
      "log_level": "info",
      "log_file_path": "/var/log/calico/cni/cni.log",
      "datastore_type": "kubernetes",
      "nodename": "app1",
      "mtu": 0,
      "ipam": {
          "type": "calico-ipam"
      },
      "policy": {
          "type": "k8s"
      },
      "kubernetes": {
          "kubeconfig": "/etc/cni/net.d/calico-kubeconfig"
      }
    },
    {
      "type": "portmap",
      "snat": true,
      "capabilities": {"portMappings": true}
    },
    {
      "type": "bandwidth",
      "capabilities": {"bandwidth": true}
    }
    ]```

I confused that the fe pod can't set static IP, 10.244.200.20 is really in 10.244.0.0/16

Context

Your Environment

kiddingl commented 2 years ago

I have soled the probem, the 10.224.200.20 is worong not in 10.244.0.0/16

5glabadmin commented 1 year ago

Hi this post save me a millions times...thank you very much!!

Martin-Luther commented 3 weeks ago

@5glabadmin Same here, thanks to @kiddingl