Allow provider-specific validation of ProviderConfig in a Cluster resource

[csrwng]

The Problem:

Currently the cluster API server validates only the known fields of a Cluster resource. ProviderConfig is taken at face value and left for the Machine controller to interpret. Validation is needed to ensure that the data passed in ProviderConfig is well-formed. There are 2 use cases for this:

• ProviderConfig only references an external resource Although we don't yet have fields for it, the Cluster could simply reference an external resource as its provider config. In this case, whatever API server owns the resource can handle that resource's validation.
• ProviderConfig is embedded in the resource This is what's allowed today. Something would have to intercept the cluster resource and validate the embedded config, or the cluster API apiserver would have to call out to something to validate it.

Given that an external ProviderConfig can be validated by an external server/controller, the following will only deal with the case in which a ProviderConfig is embedded in a Cluster resource.

Possible Solutions:

• A separate controller validates the ProviderConfig - A provider-specific controller could watch Cluster resources as they're created and set their status appropriately if the embedded ProviderConfig is found to be invalid.

  Pros:

  • Nothing to be done. Whoever implements a cloud provider can write their own controller that performs validation.

  Cons:

  • Validation is done late and will not give the client immediate feedback on a badly formed ProviderConfig

• Cluster API apiserver validates the ProviderConfig itself - Assuming that the ProviderConfig is backed by a CRD, the API server could look up the CRD definition, and if it includes a validation definition, reuse the CRD validation code to validate the resource. CRD validation is now a gated feature in Kubernetes 1.10.

  Pros:

  • Validation done at the same time other resource validation happens.

  Cons:

  • Validation is limited to what CRD validation allows
  • There is likely a good bit of work integrating code from CRD validation into the cluster API code.
  • Assumes the ProviderConfig is a CRD. If the resource type is backed by another API server, this type of validation is still not supported.

• Cluster API apiserver delegates to dynamic admission control to validate - Dynamic admission webhooks allow dynamically registering external HTTP endpoints that will be called with an API resource on admission and give them a chance to validate the resource before it makes it to the API server's regular validation logic. This requires that the apiserver be configured to include the ValidatingAdmissionWebhook admission control via command line argument.

  Pros:

  • No additional code required. The deployment of the apiserver would need its command arguments modified to include the additional admission control.
  • Validation done on admission, so the user would get immediate feedback of resources that fail validation.
  • Supports all kinds of embedded data. Not required to be backed by a CRD or API server.

  Cons:

  • A cloud provider implementation that wants to have its own validation would need to provide its own webhook and register it with the clusterapi server. A starter repo for creating your own webhook provider can be found here: https://github.com/openshift/generic-admission-server

Many thanks to @deads2k who provided a good bit of information on this.

[csrwng]

@dgoodwin @staebler fyi

[dlipovetsky]

Doesn't this proposal apply to Machine ProviderConfig, too?

[dlipovetsky]

ProviderConfig only references an external resource Although we don't yet have fields for it, the Cluster could simply reference an external resource as its provider config.

Did you mean the ProviderConfigSource field? From what I understand, once the field is implemented (ongoing work is part of https://github.com/kubernetes/kube-deploy/pull/659), a ProviderConfig will be able to reference an external resource.

[csrwng]

Doesn't this proposal apply to Machine ProviderConfig, too?

Yes it does

Did you mean the ProviderConfigSource field?

Yes

[alvaroaleman]

I started to toy around a little with the Webhook functionality of Kubebuilder by trying to let it do Defaulting + Validation for MachineDeployments.

While I got it ultimately working, there are some issues:

• This functionality is not part of a released kubebuilder version yet, one must build it from master
• The cert bootstrapping is not really usable yet, Kubebuilder will generate a secret with cert+key, but it fails to start up without that cert+key being already present. I worked around this by temporarily disabling the secret volume mount and doing a touch on the respective files so it generates the secret, then re-deploying it with the mount enabled again
• There some to be some issues with the generated patches, on my first attempt I got an Internal error occurred: jsonpatch add operation does not apply: doc is missing path: /spec/strategy/type. Adding the spec.strategy.type field with an empty string value fixed that and then the rest of the defaulting worked properly

CC @Lion-Wei

[fejta-bot]

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

[vincepri]

/remove-lifecycle stale

[vincepri]

/area api

[ncdc]

v1alpha2 removes the RawExtension fields and providers can now using OpenAPI and/or validating webhooks to validate their types.

/close

[k8s-ci-robot]

@ncdc: Closing this issue.

In response to [this](https://github.com/kubernetes-sigs/cluster-api/issues/147#issuecomment-522707423): >v1alpha2 removes the `RawExtension` fields and providers can now using OpenAPI and/or validating webhooks to validate their types. > >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.