Error calling mutations.agones.dev due to x509: cert signed by unknown authority.

[MoonBarc]

What happened: When adding a new Fleet after Agones has started up fully, it tries to call the mutations.agones.dev endpoint and fails due to

x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "admission-controller-ca")

What you expected to happen: The kubectl apply to work and deploy a Fleet. How to reproduce it (as minimally and precisely as possible):

1. install tilt
2. install agones
3. deploy a fleet with tilt.

Environment:

• Agones version: 1.9.0
• Kubernetes version:

  Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.5", GitCommit:"e6503f8d8f769ace2f338794c914a96fc335df0f", GitTreeState:"clean", BuildDate:"2020-06-26T03:47:41Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"windows/amd64"}
  Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.13", GitCommit:"39a145ca3413079bcb9c80846488786fed5fe1cb", GitTreeState:"clean", BuildDate:"2020-07-15T16:10:14Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration: Minikube, Docker Driver, and Tilt
• Install method (yaml/helm): Helm
• Troubleshooting guide log(s): Full error (in tilt):

  Build Failed: kubectl apply:
  stderr: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "mutations.agones.dev": Post https://agones-controller-service.agones-system.svc:443/mutate?timeout=30s: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "admission-controller-ca")

• Others: I'm also having the agones-controller spam this sporadically in the logs:

  TLS handshake error from 172.17.0.2:37024: remote error: tls: bad certificateTLS handshake error from 172.17.0.2:37024: remote error: tls: bad certificate

Thanks!

[markmandel]

I'm not familiar with Tilt - what happens if you run Agones on Minikube without it?

[MoonBarc]

Sorry, I should have explained more of what it does. It's basically Skaffold, except it uses Starlark and has a cool web interface. From what I know, Tilt is trying to build the image, and then deploy it using kubectl.

I will try it without tilt as soon as I can, but I'm not at home currently.

[MoonBarc]

Just tried it on kind instead of minikube and was pleasantly surprised that it works. Must just be a minikube thing. I will close for now but will also do some further investigation.

[xxtanisxx]

Wondering if you solved this? We are running into similar issues with ArgoCD

[xxtanisxx]

Just in case someone stumble onto this issue with ArgoCD. Hopefully it can save someone 2 weeks of hair pulling. We were able to resolve this by setting static controller TLS cert and key with the below field.

agones.controller.tlsCert agones.controller.tlsKey

The initial merge from ArgoCD works flawlessly. Subsequent sync from ArgoCD somehow modified the Agones certificates and all related ca-bundle. As a result, we got the error x509: cert signed by unknown authority. You can observe the behavior in ArgoCD UI with out-of-sync on mutating webhook, validating webhook, agones-cert, and/or allocator API service despite being in sync. Below is an image of what it looks like

After inputing a static TLS certificate and key, you can use the below command to validate that secret cert is identical to ca-bundle.

kubectl get mutatingwebhookconfiguration agones-mutation-webhook -o yaml -o jsonpath='{.webhooks[0].clientConfig.caBundle}' | base64 -d kubectl get secrets -n agones-system agones-cert -o yaml -o jsonpath='{.data.server\.crt}' | base64 -d

Can also run this command to verify the CA and Cert. openssl verify x509_strict -CAfile build/ca.crt build/server.crt

Note: agones-cert = { helm release name }-cert

Potential solution is to use cert-manager if you don't want TLS data in github. PR