search

Azure / open-service-broker-azure

The Open Service Broker API Server for Azure Services
https://osba.sh
MIT License
248 stars 100 forks source link

TLS handshake error from xxxx: remote error: tls: bad certificate #595

Open jldeen opened 6 years ago

jldeen commented 6 years ago

I have created multiple clusters on AKS (with RBAC) and setup both service catalog and OSBA. Sometimes it works, sometimes it doesn't. Both pods for OSBA will fire up and run, but when I do a get on clusterservicebrokers, there's an error fetching the catalog. Diving deeper, the logs for the OSBA pod is riddled with errors like this:

2018/10/05 13:42:29 http: TLS handshake error from xxxxxxx remote error: tls: bad certificate.

More logs on the OSBA clusterservicebroker:

Warning ErrorFetchingCatalog 4s (x4 over 16s) service-catalog-controller-manager Error getting broker catalog: Get https://osba-open-service-broker-azure.osba.svc.cluster.local/v2/catalog: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "osba-ca")

Obviously, if it can't pull the catalog, it can't see which classes are available so when I go to deploy a chart for OSBA, I get a "ReferencesNonExistentServiceClass" error.

If I run kubectl get clusterserviceclasses -o=custom-columns=NAME:.metadata.name,EXTERNAL\ NAME:.spec.externalName I see nothing, which would make sense on the "nonexistentserviceclass" error.

I have purged the deployments for both catalog and OSBA several times and, again, I've tried on multiple clusters.

Any idea how to solve this? Is there a step in the docs I'm missing?

bodewig commented 5 years ago

At least in my case this happens (all the time) when I do a helm upgrade over an existing installation.

The helm chart creates a new certificate and stores it in a new secret with each upgrade but the broker pod is not restarted so it still uses the old certificate. Using helm upgrade --recreate-pods works for us, maybe you can use this as a workaround as well.

The proper fix would be to add an annotation inside of deployment.yml like the one suggested in https://github.com/helm/helm/blob/master/docs/charts_tips_and_tricks.md#automatically-roll-deployments-when-configmaps-or-secrets-change - I'll try to carve out some time for a PR.

bodewig commented 5 years ago

Actually I'm not sure whether the include approach of https://github.com/helm/helm/blob/master/docs/charts_tips_and_tricks.md#automatically-roll-deployments-when-configmaps-or-secrets-change wouldn't end up reading yet another certificate than the one that is actually used.

As it stands the secret will always be rewritten so always using --recreate-pods is the logical choice. Using annotations to trigger pod reloads only when a secret changes really doesn't make any sense in the current state, so I won't create a PR. It would probably be good to address this together with #547.