Enable batched creation of GenericKubernetesResource

[syr]

Is your enhancement related to a problem? Please describe

As I didn't find it in the docs, would be very useful to create a list of GenericKubernetesResource with a single command.

With a List<io.fabric8.kubernetes.api.model.batch.v1.Job> jobList this works using: kubernetesClient.resourceList(jobList).createOrReplace()

With List<io.fabric8.kubernetes.api.model.GenericKubernetesResource> jobList it compiles, but fails as ResourceDefinitionContext must be passed but I don't see how.

Specifically, this is the call which I call n times: kubernetesClient.genericKubernetesResources(myContext).resource(myGKR).createOrReplace()

I want to migrate it to a single call by passing a List<io.fabric8.kubernetes.api.model.GenericKubernetesResource> gkrList like: kubernetesClient.genericKubernetesResources(myContext).resourceList(gkrList).createOrReplace();

Describe the solution you'd like

List<io.fabric8.kubernetes.api.model.GenericKubernetesResource> gkrList = createMyGenericKubernetesResource();
kubernetesClient.genericKubernetesResources(myContext).resourceList(gkrList).createOrReplace();

Describe alternatives you've considered

List<GenericKubernetesResource> gkrList = new ArrayList<>();
gkrList.add(new GenericKubernetesResourceBuilder()...withResourceDefinitionContext(myContext)...build());
kubernetesClient.resourceList(gkrList).createOrReplace();

Additional context

No response

[shawkins]

but fails as ResourceDefinitionContext must be passed but I don't see how

What version are you using? That should not fail on recent releases (6.x?) as long as the apiVersion and kind are set on your resources.

[syr]

Using version 6.3.1 Testing by unit test via @QuarkusTest using @WithKubernetesTestServer (Quarkus version 2.16.4.Final) GenericKubernetesResource has kind and apiVersion set (besides others)

The ResourceDefinitionContext (which I can pass at a single create kubernetesClient.genericKubernetesResources(myContext).resource(myGKR).createOrReplace() but AFAIK not by the intended kubernetesClient.resourceList(gkrList).createOrReplace()) is built by

ResourceDefinitionContext.Builder()
                .withGroup(GROUP)
                .withVersion(VERSION)
                .withKind(KIND)
                .withNamespaced(true)

I would assume the problem is related to on of these fields but I also don't see how to set them at the GenericKubernetesResource instead.

Stack trace (some values replaced by uppercase names)

io.fabric8.kubernetes.client.KubernetesClientException: Could not find a registered handler for item: [GenericKubernetesResource(apiVersion=VERSION, kind=KIND, metadata=ObjectMeta(annotations={}, creationTimestamp=null, deletionGracePeriodSeconds=null, deletionTimestamp=null, finalizers=[], generateName=null, generation=null, labels={myLabel=LABEL}, managedFields=[], name=NAME, namespace=NAMESPACE, ownerReferences=[], resourceVersion=44ffd377-466b-460b-9b81-d8ec96dc4050, selfLink=null, uid=c22d0c37-31b3-41e3-8246-697635318400, additionalProperties={}), additionalProperties={status={...})].

    at io.fabric8.kubernetes.client.impl.Handlers.get(Handlers.java:77)
    at io.fabric8.kubernetes.client.impl.KubernetesClientImpl.resource(KubernetesClientImpl.java:353)
    at io.fabric8.kubernetes.client.dsl.internal.NamespaceVisitFromServerGetWatchDeleteRecreateWaitApplicableListImpl.getResource(NamespaceVisitFromServerGetWatchDeleteRecreateWaitApplicableListImpl.java:114)
    at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
    at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
    at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
    at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
    at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
    at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
    at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
    at io.fabric8.kubernetes.client.dsl.internal.NamespaceVisitFromServerGetWatchDeleteRecreateWaitApplicableListImpl.getResources(NamespaceVisitFromServerGetWatchDeleteRecreateWaitApplicableListImpl.java:106)
    at io.fabric8.kubernetes.client.dsl.internal.NamespaceVisitFromServerGetWatchDeleteRecreateWaitApplicableListImpl.createOrReplace(NamespaceVisitFromServerGetWatchDeleteRecreateWaitApplicableListImpl.java:196)
    at io.fabric8.kubernetes.client.dsl.internal.NamespaceVisitFromServerGetWatchDeleteRecreateWaitApplicableListImpl.createOrReplace(NamespaceVisitFromServerGetWatchDeleteRecreateWaitApplicableListImpl.java:61)

[shawkins]

Using version 6.3.1

Are there more details missing here - are you using a mock server for instance?

There are two cases. Built-in types and custom types. For built-in types based upon your original comment I can confirm the following works on master in the JobIT test:

List<GenericKubernetesResource> asList = Arrays
        .asList(Serialization.jsonMapper().convertValue(job, GenericKubernetesResource.class));

List<HasMetadata> result = client
        .resourceList(asList)
        .createOrReplace();

That is as long as the resource has the appropriate apiVersion and kind set, we'll use the same ResourceDefinitionContext as if the Job class were being used.

For custom types, the api-server metadata will be used to lookup the ResourceDefinitionContext - if you are using a mock server it may be setup to respond appropriately to the metadata request.

[syr]

Yes I am using io.fabric8.kubernetes.client.server.mock.KubernetesServer and trying to create a list of CRD (custom types). I did not find documentation on how to do it...no idea how to configure mock server to "respond appropriately to the metadata request.". Don't know how the metadata request looks like..Sorry I am new in this. :-/

What we have working atm is creation of single CRD: kubernetesClient.genericKubernetesResources(myContext).resource(crd).createOrReplace()

Wondering how to get rid of the .genericKubernetesResources(myContext), then I could do kubernetesClient.resourceList(crdList).createOrReplace()

[shawkins]

6.2.0 added KubernetesMockServer.expectCustomResource. If you use that, then you don't need to specify the context with dsl.

[syr]

OK, I tried

kubernetesServer.getKubernetesMockServer().expectCustomResource(myCustomResourceDefinitionContext);
kubernetesServer.getClient().resourceList(myGenericKubernetesResourceList).createOrReplace();

get the same stacktrace :/

io.fabric8.kubernetes.client.KubernetesClientException: Could not find a registered handler for item: [GenericKubernetesResource(apiVersion=VERSION, kind=KIND, metadata=ObjectMeta(annotations={}, creationTimestamp=null, deletionGracePeriodSeconds=null, deletionTimestamp=null, finalizers=[], generateName=null, generation=null, labels={myLabel=LABEL}, managedFields=[], name=NAME, namespace=NAMESPACE, ownerReferences=[], resourceVersion=44ffd377-466b-460b-9b81-d8ec96dc4050, selfLink=null, uid=c22d0c37-31b3-41e3-8246-697635318400, additionalProperties={}), additionalProperties={status={...})].

    at io.fabric8.kubernetes.client.impl.Handlers.get(Handlers.java:77)
    at io.fabric8.kubernetes.client.impl.KubernetesClientImpl.resource(KubernetesClientImpl.java:353)
    at io.fabric8.kubernetes.client.dsl.internal.NamespaceVisitFromServerGetWatchDeleteRecreateWaitApplicableListImpl.getResource(NamespaceVisitFromServerGetWatchDeleteRecreateWaitApplicableListImpl.java:114)
    at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
    at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
    at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
    at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
    at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
    at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
    at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
    at io.fabric8.kubernetes.client.dsl.internal.NamespaceVisitFromServerGetWatchDeleteRecreateWaitApplicableListImpl.getResources(NamespaceVisitFromServerGetWatchDeleteRecreateWaitApplicableListImpl.java:106)
    at io.fabric8.kubernetes.client.dsl.internal.NamespaceVisitFromServerGetWatchDeleteRecreateWaitApplicableListImpl.createOrReplace(NamespaceVisitFromServerGetWatchDeleteRecreateWaitApplicableListImpl.java:196)
    at io.fabric8.kubernetes.client.dsl.internal.NamespaceVisitFromServerGetWatchDeleteRecreateWaitApplicableListImpl.createOrReplace(NamespaceVisitFromServerGetWatchDeleteRecreateWaitApplicableListImpl.java:61)

Even when now ignoring all the test mocks problems etc., how to get this working in productive code? How to do .genericKubernetesResources(myContext) before and call kubernetesClient.resourceList(crdList).createOrReplace() afterwards?

[shawkins]

OK, I tried

You still probably have an issue with your kind / apiVersion. You can also see how this works in the project tests https://github.com/fabric8io/kubernetes-client/blob/v6.3.1/kubernetes-tests/src/test/java/io/fabric8/kubernetes/client/mock/KubernetesMockServerTest.java#L80 - note in the following test that the crud mock also supports installing the crd as a way of supplying the metadata.

Even when now ignoring all the test mocks problems etc., how to get this working in productive code?

You do not need to use the context at all. Just like in the tests if you need a list context, you can use client.genericKubernetesResources(apiVersion, kind), otherwise if you already have the resource, you can use client.resource(object)

[syr]

@shawkins While testing against a dev k8s (no mock server) I found out how to fix it, so how to change it that I don't have to pass

myContext= ResourceDefinitionContext.Builder()
                .withGroup(GROUP)
                .withVersion(VERSION)
                .withKind(KIND)
                .withNamespaced(true)

I "migrated" these settings to the My CRD class by class annotions and interfaces, like

@Group(GROUP)
@Plural(PLURAL)
@Version(VERSION)
public class MyCRD extends GenericKubernetesResource implements Namespaced...

After this I could finally create the CRDs with a single call like

kubernetesClient.resourceList(myCrdList).createOrReplace()

just to find out, that fabric8 client makes actually single calls for every single list element :*(

Trying to work around by merging all CRDs in a single yaml file, separated by ---, also seems not to work, getting error expected a single document in the stream...but found another document

Is it generally possible to create multiple resources with a single yaml file with a single fabric8 client call / single http post under the hood?

[shawkins]

I "migrated" these settings to the My CRD class by class annotions and interfaces, like

That is definitely not necessary - for CRDs or CRs. For one there are already CustomResourceDefinition pojos. Just for clarification, are you dealing with CRD's or CR's?

As mentioned above, with generic in a list it's sufficient if the resource has the appropriate kind and api version set on the resource - new GenericKubernetesResourceBuilder().withNewMetadata().withKind(kind).withApiVersion(apiVersion)...

just to find out, that fabric8 client makes actually single calls for every single list element :*(

That is correct. In particular there's no api-server REST endpoint for batch create operations, and certainly nothing that matches the semantics of createOrReplace.

[syr]

Thanks for the clarification. Closing the issue as the whole idea of batched resource creation got obsolete by this limitation of the k8s REST API