Testing web hooks locally

[gowrisankar22]

Hi,

I am trying out ASO and want to test the webhook locally. I can see below make target creates the self-signed certificates but how do you instruct kube-apiserver to trust these self-signed certs?

https://github.com/Azure/azure-service-operator/blob/master/Makefile#L36-L56

BRs, Gowrisankar

[matthchr]

It depends what you mean by "locally"

One way to do this is documented in our development guide.

You also can just run make kind-create and then use that cluster to install ASO by going through the quickstart steps, except that instead of using helm to install the operator in steps 2 and 4, just run make deploy instead.

Why do you want to test the webhooks locally anyway?

[gowrisankar22]

@matthchr Let me explain a bit,

1. Yeah I am testing locally via kind cluster. First thing what i noticed is you are using v0.9.0 and mine is v0.10.0.
2. make deploy deploy installs all the resources including the webhooks and i can see that validating webhook created for couple of CRDs https://github.com/Azure/azure-service-operator/search?q=ValidateCreate . This step to work i need the cert manager to deployed prior hand to provision the tls certificates for webhook. As you are mounting these tls certficates to the POD, webhook validation works for me.
3. I am testing it using make run and created certs using make generate-test-certs and before applying https://github.com/Azure/azure-service-operator/blob/master/config/webhook/manifests.yaml i replaced the CABUDLE from CA_BUNDLE=$(kubectl get configmap -n kube-system extension-apiserver-authentication -o=jsonpath='{.data.client-ca-file}' | base64 | tr -d '\n')
4. make generate-test-certs creates the self signed certs but how do you instruct kube-apiserver to trust these self-signed certs? because validating webhook gives me below error Error from server (InternalError): Internal error occurred: failed calling webhook ....... x509: certificate relies on legacy Common Name field, use SANs or temporarily enable Common Name matching with GODEBUG=x509ignoreCN=0
5. Also, local testing https://github.com/Azure/azure-service-operator/blob/master/config/webhook/manifests.yaml#L14 this wont work and probably you should use url: "https://host.docker.internal:9443/validate-azure-microsoft-com-v1alpha1-azuresqlmanageduser" to talk to api server
6. In a nutshell, make generate-test-certs you are creating just for manager to start w/o any issues and this is not really used for webhook testing.

[gowrisankar22]

@matthchr any update this ?

[matthchr]

The following steps allow webhooks to work for me:

1. make kind-create
2. kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v0.12.0/cert-manager.yaml
3. Modify /home/matthchr/work/github/azure-service-operator/config/default/manager_image_patch.yaml line 11 to mcr.microsoft.com/k8s/azureserviceoperator:1.0.22275 (having to do this is awkward and we should probably do it automatically in a make target).
4. make deploy

I then tested by modifying config/samples/azure_v1alpha1_azuresqluser.yaml to have a DBName of master (which should be rejected). k apply -f config/samples/azure_v1alpha1_azuresqluser.yaml confirms that it is indeed rejected:

$ k apply -f config/samples/azure_v1alpha1_azuresqluser.yaml
Error from server ('master' is a reserved database name and cannot be used): error when creating "config/samples/azure_v1alpha1_azuresqluser.yaml": admission webhook "vazuresqluser.kb.io" denied the request: 'master' is a reserved database name and cannot be used

In our automated testing, the webhooks are run as part of envtest, so this path is really only useful for manual testing. See azuresqlmanageduser_controller_test for an example automated test for the webhooks.

Why do you want to test the webhooks like this anyway?

[matthchr]

In a nutshell, make generate-test-certs you are creating just for manager to start w/o any issues and this is not really used for webhook testing.

That's correct - that step is only used if you want to run main.go of the operator locally on your machine (i.e. not hosting it in a container in Kubernetes/kind at all). If you do that you can't run webhooks because the operator isn't actually running "in" the cluster (it doesn't have an endpoint in the cluster it's listening to webhook requests for).

Installing the operator to kind via make deploy means the container is running in the kind cluster and whatever certificates you have locally don't mean anything anyway (unless you mount them into the container hosting the operator).

[matthchr]

Closing this as there hasn't been a response for a while and I believe the original issue was answered. Feel free to reopen if need be.