Calico doesn't work if cni config hostname doesnt match calico-node nodename

bradbeam commented 7 years ago

Ran into an issue where I had a hostname mismatch between the cni config and calico-node hostname variable. This caused calico networking to appear functional but not actually work. Worked with @tmjd to get this identified and resolved.

cni config referenced the short name [1] and the calico-node nodename parameter was obtained via kubernetes downward api [2]. This caused an issue where calicoctl get nodes reported back both the fqdn and short name for the node [3].

[1]

# cat /etc/cni/net.d/10-calico.conf 
{
  "name": "calico-k8s-network",
  "hostname": "worker-02",
  "type": "calico",
  "etcd_endpoints": "https://10.232.255.8:2379,https://10.232.255.5:2379,https://10.232.255.7:2379",
  "etcd_cert_file": "/etc/ssl/etcd/ssl/node-worker-02.mydomain.com.pem",
  "etcd_key_file": "/etc/ssl/etcd/ssl/node-worker-02.mydomain.com-key.pem",
  "etcd_ca_cert_file": "/etc/ssl/etcd/ssl/ca.pem",
  "log_level": "info",
  "ipam": {
    "type": "calico-ipam"
  },
  "kubernetes": {
    "kubeconfig": "/etc/kubernetes/node-kubeconfig.yaml"
  }
}

[2]

...
        - name: NODENAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: spec.nodeName

[3]

$ calicoctl get nodes
NAME                                 
control-01.mydomain.com   
control-02                           
control-02.mydomain.com   
control-03.mydomain.com   
worker-01                            
worker-01.mydomain.com    
worker-02                            
worker-02.mydomain.com

Expected Behavior

Calico networking for pods works.

Current Behavior

Current behavior allows for you to have mismatching names with calico-node running happily and kubelet / cni happy as well, while in fact networking is not working as expected.

Possible Solution

It'd be swell if there was a way to unify the way the node names are defined. Some ideas:

Perhaps calico-node can use the hostname parameter from the cni config? I guess in my case this still wouldn't have mattered because I was doing an explicit set of the nodename in calico-node
Perhaps calico-node can check to see if the hostname supplied with cni is the first item of a fqdn name and match/merge the two together?
Or find some way to log an error message? Not really sure on the best way to go from here. Since the two are a bit independent its a bit tricky. Maybe the best approach is just to document the requirement?

Steps to Reproduce (for bugs)

1. 2. 3. 4.

Context

Your Environment

Calico version 2.5
Orchestrator version (e.g. kubernetes, mesos, rkt): k8s 1.7.4
Operating System and version: ubuntu 16.04
Link to your project (optional): kubespray https://github.com/kubernetes-incubator/kubespray/tree/e16b57aa055a576c6572cf27dcaf535a70692429/roles/network_plugin/calico/templates/cni-calico.conf.j2#L3 https://github.com/kubernetes-incubator/kubespray/tree/e16b57aa055a576c6572cf27dcaf535a70692429/roles/network_plugin/calico/templates/calico-node.yml.j2#L107

mattymo commented 7 years ago

Should kubespray stop defining hostname in /etc/cni/net.d/10-calico.conf?

heschlie commented 7 years ago

@mattymo It should use nodename as we deprecated hostname to better align with what we are looking for, and maybe should use ansible_nodename or ansible_fqdn? I'm not great with Ansible so I might be off target there.

caseydavenport commented 7 years ago

I think there's not a lot we can do today given the orthogonal nature of the two components besides make our documentation much better.

Note that we already set both the CNI and calico/node nodename configuration options in the KDD manifest using the k8s downward API.

This is probably the direction we want to point people since it guarantees a consistent value, although changing that on a running cluster has potential to cause some issues so I don't know that we can just update our manifests across-the-board to do this.

It should use nodename as we deprecated hostname

@heschlie I'd wait until https://github.com/projectcalico/cni-plugin/pull/375 is merged and released though if you're using calico IPAM!

ghost commented 6 years ago

We gave full stack cni/node Calico a try yesterday. We got hit hard by this.

At my scenario, I found out that:

Calico-node default behaviour is to register the calico host with its ip address if NODENAME is not set.
Calico cni default behaviour is to register the calico host with its FQDN hostname if NODENAME is not set.

(my lab setup is 2.6.5 with calico node as DS from yaml provided, cni installed from secondary install-cni container).

Although I noticed its listed pretty clear here that if calico-node registers itself with something other than hostname you are going to have a bad time, I couldn't find out anywhere in the documentation that the default behaviour from calico-node is to register with IP.

If you folks find appropriate, I'd suggest to put a note in big bold letters on both calico-node/cni documentation stating what is going to happen if you don't define your nodenames manually on both places.

caseydavenport commented 6 years ago

Calico-node default behaviour is to register the calico host with its ip address if NODENAME is not set.

Calico cni default behaviour is to register the calico host with its FQDN hostname if NODENAME is not set

That's interesting, and not what I'd have expected. I'd expect both to use the same logic, defaulting to the value provided by os.Hostname(). Looking at the code though, the two bits are unfortunately not identical...

It's a shame this logic isn't common between the two. The main difference seems to be this bit of code where the HOSTNAME env var is checked. I wonder if that is the root of the discrepancy you were seeing, @mrrandrade .

reachlin commented 6 years ago

we hit this problem too.

caseydavenport commented 6 years ago

I'd like to change the manfiests to that these are always the same:

In CNI config:

"nodename": "__KUBERNETES_NODE_NAME__",

In the DaemonSet:

# Set based on the k8s node name.
- name: NODENAME
  valueFrom:
    fieldRef:
      fieldPath: spec.nodeName

However, if the chosen value differs from the value that was previously autodetected, this will break things on upgrade. So, we'd need to find a way to support upgrade to the new manfiest.

As it is, I can't really think of anything workable - making this sort of change is going to require downtime on any cluster performing an upgrade. Any ideas?

jryberg commented 6 years ago

This one was not very fun to discover. It took me two days reading documentation, debugging and a lot of frustration.

When it's a mismatch between CNI nodename and calico-node nodename the interface is created but it gets no ip address assigned.

I did not define any nodename in CNI configuration but it was defined in my calico-node configuration.

Once I added nodename config to CNI my cluster started work work.

caseydavenport commented 6 years ago

I'm working on a fix for this.

I've got two PRs:

calico/node: https://github.com/projectcalico/calico/pull/1722
cni-plugin: https://github.com/projectcalico/cni-plugin/pull/480

Together, those PRs will let the two components coordinate on a node name to use.

projectcalico / calico