Support custom Kubernetes resources

[tortila]

Currently Zelt only supports the following Kubernetes resources: https://github.com/zalando-incubator/zelt/blob/5ab87ddca18b5208872e731a5788f8670a78ce09/zelt/kubernetes/manifest.py#L14-L19

Recently there was a use case within Zalando to deploy the resource PlatformCredentialsSet alongside the standard Locust resources, and it was not possible - Zelt throws the error Unsupported resource type.

Describe the solution you'd like. Zelt should allow users to define manifests for existing custom resources (i.e. not Custom Resource Definitions) and be able to deploy them.

[thilp]

https://github.com/zalando-incubator/zelt/blob/5ab87ddca18b5208872e731a5788f8670a78ce09/zelt/kubernetes/manifest_set.py#L8-L18

I suppose we don't really need to be so restrictive in our handling of user-provided manifests? My Zelt knowledge is a bit rusted but I think we can simply continue to check (as we do today) that we have all the required manifests, and also add unrecognized manifests to the manifest set.

Reading the code, I seem to recall that one reason for not handling custom resources is that we're using the kubernetes lib, which makes us use specialized functions like NetworkingV1beta1Api.create_namespaced_ingress — we can't just cover everything a user could need with that. I remember that Kopf had the same problem. A solution for Kopf was to replace kubernetes with pykube-ng, but in our case perhaps we can simply use the HTTP client provided by kubernetes (handling auth) and talk HTTP to the API server. After all, we just need to POST or DELETE arbitrary resources.

What do you think?

[tortila]

The current Kubernetes Python client supports the custom objects API, which I haven't tried yet, but it looks like it could handle all custom resources manifests. The only limitation that I see is that the API for creating custom objects is very explicit and expects: group, version, plural. These values (except for the version) are not provided in the manifest of the custom object, but rather belong to the CRD itself. So it looks like - in the example of creating a PlatformCredentialsSet resource, we would have to:

• check the manifest and get the type of the resource (PlatformCredentialsSet),
• call the Kubernetes API to describe this CRD and get: group and plural,
• make the actual call to create the custom object with the correct values.

I haven't checked pykube-ng - are you saying that it allows to create custom objects without providing the type or the extra values that are not present in the manifests? If so, I'm sold. This would really simplify the overall code for handling Kubernetes logic. Otherwise I would be interested in what you think about the approach with the currently used Kubernetes client that I described above.

[thilp]

pykube-ng has an object_factory function that only needs the apiVersion and kind fields, and internally it does the lookup with the API server for you (to get the "plural" and whether the resource supports namespacing).

If kubernetes supports custom resources as well now, then the only difference I see with pykube-ng is that we need to fetch the list of plurals manually, which is not (should not be) a big deal. In that case, I wouldn't recommend investigating pykube-ng further, as mixing two Kubernetes clients will become confusing and I'm not sure there's reason enough to replace kubernetes with pykube-ng globally.

• call the Kubernetes API to describe this CRD and get: group and plural,

We can get group and version from the apiVersion field with something like:

group, version = api_version.rsplit("/", 1)

so I think the only dependency we have on the API server is the plural (what pykube-ng calls endpoint).