Test how linkerd and cillium interact

[dadjeibaah]

Based on https://github.com/linkerd/linkerd2/issues/6238#issuecomment-858089072 and https://github.com/linkerd/linkerd2/issues/6238#issuecomment-858796177, it appears that there may be issues between linkerd proxies and Cillium's load balancing features. In order to get a better understanding on what these issues are and possibly how we can fix them, we should write new integration tests that exercise running Linkerd with Cillium.

We should make sure that some of the following features work in a cillium environment:

• gRPC load balancing
• service profiles
• destination resolution

There may be other issues like multicluster but that might be out of scope for these tests.

[kleimkuhler]

Another issue I have observed this interaction in is #5932

Not all users who have commented on the issue use Cilium, but if you look at my comment I was able to observe this behavior and fix it by disabling hostServices on Cilium.

[alex-berger]

For reference these are the corresponding Cilium issue/PR for this problem

• https://github.com/cilium/cilium/issues/16583
• https://github.com/cilium/cilium/pull/13882

[jastBytes]

I am using cilium and tried to set up multicluster with linkerd today. My problem is that the linkerd-service-mirror proxy is using the external ip address of the 2nd cluster to contact the multicluster gateway instead of the local probe service. That’s why the identity is not correct when connecting to the 2nd cluster and the 2nd cluster's gateway refuses the connection. Special thanks to @mateiidavid which tried to help me here. Maybe this is a cilium issue as well. https://linkerd.slack.com/archives/C89RTCWJF/p1629357555443900

[jastBytes]

So I tried the exact same configuration but switched both clusters to calico and it works right away. So there is definitely something going on with cilium and multicluster linkerd.