Generic/dynamic resource client

[felixfbecker]

For my implementation of kubectl apply, I get any kind of KubeResourceV1 as input. I then have to retrieve the current state from the server, compute a three-way patch (using reflection) and apply it.

Unfortunately, since the resource clients seem to only be accessible through extension methods, and KubeResourceClient only has protected methods. Therefor I have to write a big if/else and handle every possible resource type:

if (Resource is DeploymentV1Beta1) {
    DeploymentV1Beta1 modified = Resource as DeploymentV1Beta1;
    DeploymentV1Beta1 current = await client.DeploymentsV1Beta1().Get(Resource.Metadata.Name, Resource.Metadata.Namespace, cancellationToken);
    DeploymentV1Beta1 original;
    string originalJson = current.Metadata.Annotations[lastAppliedConfigAnnotation];
    if (!String.IsNullOrEmpty(originalJson)) {
        original = JsonConvert.DeserializeObject<DeploymentV1Beta1>(originalJson);
    } else {
        original = modified;
    }

    Action<JsonPatchDocument<DeploymentV1Beta1>> patchAction = deploymentPatch => {
        var patch = new JsonPatchDocument();
        diff(current, modified, patch, ignoreDeletions: true);
        diff(original, modified, patch, ignoreAdditionsAndModifications: true);
        foreach (var operation in patch.Operations) {
            deploymentPatch.Operations.Add(new Operation<DeploymentV1Beta1>(operation.op, operation.path, operation.from, operation.value));
        }
    };

    if (ShouldProcess(Resource.Metadata.Name, "patch")) {
        await client.DeploymentsV1Beta1().Update(Resource.Metadata.Name, patchAction, Resource.Metadata.Namespace, cancellationToken);
    }
} else if (Resource is ServiceV1) {
    ServiceV1 modified = Resource as ServiceV1;
    ServiceV1 current = await client.ServicesV1().Get(Resource.Metadata.Name, Resource.Metadata.Namespace, cancellationToken);
    ServiceV1 original;
    string originalJson = current.Metadata.Annotations[lastAppliedConfigAnnotation];
    if (!String.IsNullOrEmpty(originalJson)) {
        original = JsonConvert.DeserializeObject<ServiceV1>(originalJson);
    } else {
        original = modified;
    }

    Action<JsonPatchDocument<ServiceV1>> patchAction = deploymentPatch => {
        var patch = new JsonPatchDocument();
        diff(current, modified, patch, ignoreDeletions: true);
        diff(original, modified, patch, ignoreAdditionsAndModifications: true);
        foreach (var operation in patch.Operations) {
            deploymentPatch.Operations.Add(new Operation<ServiceV1>(operation.op, operation.path, operation.from, operation.value));
        }
    };

    if (ShouldProcess(Resource.Metadata.Name, "patch")) {
        await client.ServicesV1().Update(Resource.Metadata.Name, patchAction, Resource.Metadata.Namespace, cancellationToken);
    }
    await client.ServicesV1().Update(Resource.Metadata.Name, patchAction, Resource.Metadata.Namespace, cancellationToken);
} else if (Resource is PersistentVolumeClaimV1) {
    // ... repeat for every single resource type ...

It would be awesome if instead, I could use a dynamic KubeResouceClient like this:

IKubeResourceClient resourceClient = client.ResourceClient(Resource.GetType());
if (!(resourceClient is IGetClient)) {
    throw new Exception($"Resource kind {Resource.Kind} is not gettable");
}
if (!(resourceClient is IPatchClient)) {
    throw new Exception($"Resource kind {Resource.Kind} is not patchable");
}
KubeResourceV1 current = await (resourceClient as IGetClient).Get(Resource.Metadata.Name, Resource.Metadata.Namespace, cancellationToken);
KubeResourceV1 original;
string originalJson = current.Metadata.Annotations[lastAppliedConfigAnnotation];
if (!String.IsNullOrEmpty(originalJson)) {
    original = JsonConvert.DeserializeObject(originalJson, Resource.GetType()) as KubeResourceV1;
} else {
    original = Resource;
}

Action<JsonPatchDocument> patchAction = patch => {
    diff(current, Resource, patch, ignoreDeletions: true);
    diff(original, Resource, patch, ignoreAdditionsAndModifications: true);
};

if (ShouldProcess(Resource.Metadata.Name, "patch")) {
    await (resourceClient as IPatchClient).Update(Resource.Metadata.Name, patchAction, Resource.Metadata.Namespace, cancellationToken);
}

I am not sure if this actually works or if this is idiomatic API design in C# - wdyt?

[tintoy]

Hmm - I'm pretty sure we can do something along these lines. Let me have a think about it this afternoon and get back to you?

Another possibility might be to have DynamicResourceClientV1 which is effectively equivalent to KubeResourceClient<KubeResourceV1> (but I think the idea needs a little TLC).

[tintoy]

The tricky part is dynamically figuring out the API URLs for each resource type. But it can be done (there's an API for that :wink:)

[tintoy]

The tricky part is dynamically figuring out the API URLs for each resource type. But it can be done (there's an API for that 😉)

Looks promising:

https://gist.github.com/tintoy/49b7c400fb89325c3a914667af4427b5

[tintoy]

And a quick-and-dirty metadata cache API:

using System;
using System.Collections.Concurrent;
using System.Collections.Generic;
using System.Linq;
using System.Threading;
using System.Threading.Tasks;

namespace KubeClient.Metadata
{
    using Models;

    public sealed class KubeMetadataCache
    {
        static readonly IReadOnlyCollection<string> ApiGroupPrefixes = new string[] { "api", "apis" };

        readonly object _stateLock = new object();

        readonly Dictionary<string, KubeApiMetadata> _metadata = new Dictionary<string, KubeApiMetadata>();

        public KubeMetadataCache()
        {
        }

        public bool IsEmpty
        {
            get
            {
                lock (_stateLock)
                {
                    return _metadata.Count == 0;
                }
            }
        }

        public KubeApiMetadata Get<TModel>()
            where TModel : KubeObjectV1
        {
            return Get(
                typeof(TModel)
            );
        }

        public KubeApiMetadata Get(Type modelType)
        {
            if (modelType == null)
                throw new ArgumentNullException(nameof(modelType));

            (string kind, string apiVersion) = KubeObjectV1.GetKubeKind(modelType);
            if (String.IsNullOrWhiteSpace(kind))
                throw new ArgumentException($"Model type {modelType.FullName} has not been decorated with KubeResourceAttribute or KubeResourceListAttribute.", nameof(modelType));

            return Get(kind, apiVersion);
        }

        public KubeApiMetadata Get(string kind, string apiVersion)
        {
            if (String.IsNullOrWhiteSpace(kind))
                throw new ArgumentException("Argument cannot be null, empty, or entirely composed of whitespace: 'kind'.", nameof(kind));

            if (String.IsNullOrWhiteSpace(apiVersion))
                throw new ArgumentException("Argument cannot be null, empty, or entirely composed of whitespace: 'apiVersion'.", nameof(apiVersion));

            lock (_stateLock)
            {
                string cacheKey = CreateCacheKey(kind, apiVersion);
                if (_metadata.TryGetValue(cacheKey, out KubeApiMetadata metadata))
                    return metadata;
            }

            return null;
        }

        public string GetPrimaryPath<TModel>()
            where TModel : KubeObjectV1
        {
            return GetPrimaryPath(
                typeof(TModel)
            );
        }

        public string GetPrimaryPath(Type modelType)
        {
            if (modelType == null)
                throw new ArgumentNullException(nameof(modelType));

            (string kind, string apiVersion) = KubeObjectV1.GetKubeKind(modelType);
            if (String.IsNullOrWhiteSpace(kind))
                throw new ArgumentException($"Model type {modelType.FullName} has not been decorated with KubeResourceAttribute or KubeResourceListAttribute.", nameof(modelType));

            return GetPrimaryPath(kind, apiVersion);
        }

        public string GetPrimaryPath(string kind, string apiVersion)
        {
            if (String.IsNullOrWhiteSpace(kind))
                throw new ArgumentException("Argument cannot be null, empty, or entirely composed of whitespace: 'kind'.", nameof(kind));

            if (String.IsNullOrWhiteSpace(apiVersion))
                throw new ArgumentException("Argument cannot be null, empty, or entirely composed of whitespace: 'apiVersion'.", nameof(apiVersion));

            lock (_stateLock)
            {
                KubeApiMetadata metadata = Get(kind, apiVersion);
                if (metadata == null)
                    throw new KeyNotFoundException($"No API metadata found for '{kind}/{apiVersion}'");

                return metadata.Paths.Select(pathMetadata => pathMetadata.Path).FirstOrDefault();
            }
        }

        public void Clear()
        {
            lock (_stateLock)
            {
                _metadata.Clear();
            }
        }

        public async Task Load(KubeApiClient kubeClient, CancellationToken cancellationToken = default)
        {
            if (kubeClient == null)
                throw new ArgumentNullException(nameof(kubeClient));

            var loadedMetadata = new List<KubeApiMetadata>();

            foreach (string apiGroupPrefix in ApiGroupPrefixes)
            {
                APIGroupListV1 apiGroups = await kubeClient.APIGroupsV1().List(apiGroupPrefix, cancellationToken);
                if (apiGroupPrefix == "api")
                {
                    // Special case for old-style ("api/v1") APIs.
                    apiGroups.Groups.Add(new APIGroupV1
                    {
                        Name = "Core",
                        PreferredVersion = new GroupVersionForDiscoveryV1
                        {
                            GroupVersion = "v1"
                        }
                    });
                }

                foreach (APIGroupV1 apiGroup in apiGroups.Groups)
                {
                    List<string> groupVersions = new List<string>();
                    if (apiGroupPrefix == "api")
                    {
                        groupVersions.Add("v1");
                    }
                    else
                    {
                        groupVersions.AddRange(
                            apiGroup.Versions.Select(
                                version => version.GroupVersion
                            )
                        );
                    }

                    foreach (string groupVersion in groupVersions)
                    {
                        APIResourceListV1 apis = await kubeClient.APIResourcesV1().List(apiGroupPrefix, groupVersion, cancellationToken);
                        foreach (var apisForKind in apis.Resources.GroupBy(api => api.Kind))
                        {
                            string kind = apisForKind.Key;

                            var apiPaths = new List<KubeApiPathMetadata>();

                            foreach (APIResourceV1 api in apisForKind)
                            {
                                string apiPath = $"{apiGroupPrefix}/{apiGroup.PreferredVersion.GroupVersion}/{api.Name}";

                                apiPaths.Add(
                                    new KubeApiPathMetadata(apiPath,
                                        verbs: api.Verbs.ToArray()
                                    )
                                );
                            }

                            loadedMetadata.Add(
                                new KubeApiMetadata(kind, groupVersion, apiPaths)
                            );
                        }
                    }
                }
            }

            lock (_stateLock)
            {
                _metadata.Clear();

                foreach (KubeApiMetadata apiMetadata in loadedMetadata)
                {
                    string cacheKey = CreateCacheKey(apiMetadata.Kind, apiMetadata.ApiVersion);
                    _metadata[cacheKey] = apiMetadata;
                }
            }
        }

        static string CreateCacheKey(string kind, string apiVersion)
        {
            if (String.IsNullOrWhiteSpace(kind))
                throw new ArgumentException("Argument cannot be null, empty, or entirely composed of whitespace: 'kind'.", nameof(kind));

            if (String.IsNullOrWhiteSpace(apiVersion))
                throw new ArgumentException("Argument cannot be null, empty, or entirely composed of whitespace: 'apiVersion'.", nameof(apiVersion));

            return $"{apiVersion}/{kind}";
        }
    }

    public class KubeApiMetadata
    {
        public KubeApiMetadata(string kind, string apiVersion, IReadOnlyList<KubeApiPathMetadata> paths)
        {
            Kind = kind;
            ApiVersion = apiVersion;
            Paths = paths;
        }

        public string Kind { get; }
        public string ApiVersion { get; }
        public IReadOnlyList<KubeApiPathMetadata> Paths { get; }
    }

    public class KubeApiPathMetadata
    {
        public KubeApiPathMetadata(string path, IReadOnlyCollection<string> verbs)
        {
            Path = path;
            Verbs = verbs;
        }

        public string Path { get; }
        public IReadOnlyCollection<string> Verbs { get; }
    }
}

[tintoy]

So this API should enable you, given a resource type's kind and apiVersion, to find the primary API path for that resource type.

Currently this doesn't handle resource types that aren't served up by the K8s apiserver but it'll do for a start.

Does the API look like it would do what you need?

If so, I can then create a ResourceClient that will dynamically discover API paths and support basic operations only (CRUD, basically, where U = PATCH).

[tintoy]

You'd get at it by calling kubeClient.Dynamic().Get("Pod", "v1", "my-pod").

[tintoy]

Or kubeClient.Dynamic().List("Pod", "v1", "my-namespace").

[felixfbecker]

Ha, that's even more dynamic than I expected. I think I will still want to decode the YAML as the actual model classes, because I want the objects to be typed strongly, so they can be formatted with Format.ps1xml files. So I can give you the type dynamically with resource.GetType() if that makes it easier? Of course, I can also call client.Patch(resource.Kind, resource.ApiVersion.split('/').Last(), patch).

[tintoy]

If you have a resource object already then I can reflect on its Type as required. Or are you deserialising and want a CLR Type given kind and apiVersion?

[felixfbecker]

I see it as a two step process. #26 would allow me to deserialise anything that has kind and apiVersion to model instances (I would want that in any case to implement a ConvertFrom-KubeYaml cmdlet). Then once I have that, I can easily use it as the first step of the kubectl apply implementation (i.e. Update-KubeResource will take a strongly-typed model object as input, which you can get from ConvertFrom-KubeYaml). The diffing logic in the apply implementation is written with reflection so it just takes in object and generates a JsonPatchDocument. The statically known type will be KubeResource (since it's any subclass of that), which is why I can't use one of the static clients, but I can give you the runtime type through resource.GetType().

[tintoy]

No worries - if the runtime type is something derived from KubeResourceV1 then the dynamic client can work with it, figuring out the API paths on the fly.

[tintoy]

BTW, it should be pretty easy to write a JsonConverter that uses the model-type lookups to dynamically select the type to deserialise based on kind and apiVersion in the incoming JSON.

Example: https://github.com/tintoy/aykay-deekay/blob/master/src/AKDK/Messages/DockerEvents/Converters/JsonCreationConverter.cs and https://github.com/tintoy/aykay-deekay/blob/master/src/AKDK/Messages/DockerEvents/Converters/DockerEventConverter.cs

[tintoy]

Still need to implement create / delete but we're getting there :)

[tintoy]

Will also work on making the API metadata cache read-through (and make the pre-load behaviour optional).