Inconsistent service availability

[ThomasBoutelier]

We use cloudmap controller in multiple clusters and have the same problem on them. Clusters are linked by vpc peering. Half of the time, an imported service cannot be reached in a pod.

• Service name is correctly resolved in an internal kube adress.
• If I use the endpointslice adress of the remote service, it works on every pod.
• I created a DaemonSet for testing purposes to see if it can be a network/node problem. A pod that have worked on a node can fail after a restart and vice versa.
• We used mcs-controller in v0.2.3 and updated in v0.3.0 and have the same problems.
• Kubernetes 1.21 on all clusters.

I have a feeling that there is some trouble between the service and endpointslice resolution but it don't know how can I debug it.

[runakash]

Hello @ThomasBoutelier - Which version of coredns are you using?

Also, if you can share the logs, below is the command

kubectl logs -f -l control-plane=controller-manager -c manager -n cloud-map-mcs-system

[ThomasBoutelier]

Hello,

It's the version v1.8.4.

Here's the output of the logs:

{"level":"info","ts":1664418556.359755,"logger":"controllers.ServiceExport","msg":"no changes to export to Cloud Map","namespace":"bidder-es-inte","name":"matcher-es-http"}
{"level":"info","ts":1664423785.0764976,"logger":"controllers.ServiceExport","msg":"updating Cloud Map service","namespace":"bidder-inte","name":"redis-master"}
{"level":"info","ts":1664423785.0765486,"logger":"cloudmap","msg":"fetching a service","namespace":"bidder-inte","name":"redis-master"}
{"level":"info","ts":1664423785.0766342,"logger":"controllers.ServiceExport","msg":"no changes to export to Cloud Map","namespace":"bidder-inte","name":"redis-master"}
{"level":"info","ts":1664423785.076691,"logger":"controllers.ServiceExport","msg":"updating Cloud Map service","namespace":"bidder-es-inte","name":"matcher-es-http"}
{"level":"info","ts":1664423785.0766985,"logger":"cloudmap","msg":"fetching a service","namespace":"bidder-es-inte","name":"matcher-es-http"}
{"level":"info","ts":1664423785.076791,"logger":"controllers.ServiceExport","msg":"no changes to export to Cloud Map","namespace":"bidder-es-inte","name":"matcher-es-http"}
{"level":"info","ts":1664423785.0768418,"logger":"controllers.ServiceExport","msg":"updating Cloud Map service","namespace":"bidder-inte","name":"iasp"}
{"level":"info","ts":1664423785.0768507,"logger":"cloudmap","msg":"fetching a service","namespace":"bidder-inte","name":"iasp"}
{"level":"info","ts":1664423785.0769007,"logger":"controllers.ServiceExport","msg":"no changes to export to Cloud Map","namespace":"bidder-inte","name":"iasp"}

[runakash]

Hey - I can help you troubleshoot. Can you join our slack channel? https://github.com/aws/aws-cloud-map-mcs-controller-for-k8s#slack-community

[runakash]

Should be fixed by the https://github.com/aws/aws-cloud-map-mcs-controller-for-k8s/issues/254.