googleforgames / agones

Dedicated Game Server Hosting and Scaling for Multiplayer Games on Kubernetes
https://agones.dev
Apache License 2.0
6.08k stars 808 forks source link

Error calling mutations.agones.dev due to x509: cert signed by unknown authority. #1883

Closed MoonBarc closed 3 years ago

MoonBarc commented 3 years ago

What happened: When adding a new Fleet after Agones has started up fully, it tries to call the mutations.agones.dev endpoint and fails due to

x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "admission-controller-ca")

What you expected to happen: The kubectl apply to work and deploy a Fleet. How to reproduce it (as minimally and precisely as possible):

  1. install tilt
  2. install agones
  3. deploy a fleet with tilt.

Environment:

Thanks!

markmandel commented 3 years ago

I'm not familiar with Tilt - what happens if you run Agones on Minikube without it?

MoonBarc commented 3 years ago

Sorry, I should have explained more of what it does. It's basically Skaffold, except it uses Starlark and has a cool web interface. From what I know, Tilt is trying to build the image, and then deploy it using kubectl.

I will try it without tilt as soon as I can, but I'm not at home currently.

MoonBarc commented 3 years ago

Just tried it on kind instead of minikube and was pleasantly surprised that it works. Must just be a minikube thing. I will close for now but will also do some further investigation.

xxtanisxx commented 2 years ago

Wondering if you solved this? We are running into similar issues with ArgoCD

xxtanisxx commented 2 years ago

Just in case someone stumble onto this issue with ArgoCD. Hopefully it can save someone 2 weeks of hair pulling. We were able to resolve this by setting static controller TLS cert and key with the below field.

agones.controller.tlsCert agones.controller.tlsKey

The initial merge from ArgoCD works flawlessly. Subsequent sync from ArgoCD somehow modified the Agones certificates and all related ca-bundle. As a result, we got the error x509: cert signed by unknown authority. You can observe the behavior in ArgoCD UI with out-of-sync on mutating webhook, validating webhook, agones-cert, and/or allocator API service despite being in sync. Below is an image of what it looks like

Screen Shot 2022-01-31 at 3 02 53 PM

After inputing a static TLS certificate and key, you can use the below command to validate that secret cert is identical to ca-bundle.

kubectl get mutatingwebhookconfiguration agones-mutation-webhook -o yaml -o jsonpath='{.webhooks[0].clientConfig.caBundle}' | base64 -d kubectl get secrets -n agones-system agones-cert -o yaml -o jsonpath='{.data.server\.crt}' | base64 -d

Can also run this command to verify the CA and Cert. openssl verify x509_strict -CAfile build/ca.crt build/server.crt

Note: agones-cert = { helm release name }-cert

Potential solution is to use cert-manager if you don't want TLS data in github. PR