Informer is unable to deserialize Kubernetes resource in native mode

[metacosm]

Describe the bug

Would appreciate help figuring out https://github.com/quarkiverse/quarkus-operator-sdk/issues/359. It seems that some resources (not even sure which) are not properly deserialized, resulting in an NPE downstream (the observed exception). /cc @iocanel @manusa

Expected behavior

No response

Actual behavior

No response

How to Reproduce?

See: https://github.com/quarkiverse/quarkus-operator-sdk/issues/359#issuecomment-1151126745

• Clone repo at: https://github.com/cdmikechen/Operator-Dependent-Test
• Change property https://github.com/cdmikechen/Operator-Dependent-Test/blob/test-native-run/src/main/resources/application.properties#L5 to true
• Compile natively: mvn package -Dnative
• Connect to your favorite Kubernetes cluster
• Launch the app: ./target/bigdata-operator-1.0.0-SNAPSHOT-runner
• You should observe a whole bunch of exceptions:

  2022-06-14 14:03:08,741 ERROR [io.fab.kub.cli.inf.cac.SharedProcessor] (main) Failed invoking io.javaoperatorsdk.operator.processing.event.source.informer.InformerEventSource@7e6f916d event handler: null: java.lang.NullPointerException
      at io.javaoperatorsdk.operator.processing.event.ResourceID.fromFirstOwnerReference(ResourceID.java:17)
      at io.javaoperatorsdk.operator.processing.event.source.informer.Mappers.lambda$fromOwnerReference$0(Mappers.java:35)
      at io.javaoperatorsdk.operator.processing.event.source.informer.PrimaryToSecondaryIndex.onAddOrUpdate(PrimaryToSecondaryIndex.java:20)
      at io.javaoperatorsdk.operator.processing.event.source.informer.InformerEventSource.onAdd(InformerEventSource.java:98)
      at io.javaoperatorsdk.operator.processing.event.source.informer.InformerEventSource.onAdd(InformerEventSource.java:66)
      at io.fabric8.kubernetes.client.informers.cache.ProcessorListener$AddNotification.handle(ProcessorListener.java:96)
      at io.fabric8.kubernetes.client.informers.cache.ProcessorListener.add(ProcessorListener.java:47)
      at io.fabric8.kubernetes.client.informers.cache.SharedProcessor.lambda$distribute$0(SharedProcessor.java:79)
      at io.fabric8.kubernetes.client.informers.cache.SharedProcessor.lambda$distribute$1(SharedProcessor.java:101)
      at io.fabric8.kubernetes.client.utils.SerialExecutor.lambda$execute$0(SerialExecutor.java:40)
      at io.fabric8.kubernetes.client.utils.SerialExecutor.scheduleNext(SerialExecutor.java:52)
      at io.fabric8.kubernetes.client.utils.SerialExecutor.execute(SerialExecutor.java:46)
      at io.fabric8.kubernetes.client.informers.cache.SharedProcessor.distribute(SharedProcessor.java:98)
      at io.fabric8.kubernetes.client.informers.cache.SharedProcessor.distribute(SharedProcessor.java:79)
      at io.fabric8.kubernetes.client.informers.cache.ProcessorStore.update(ProcessorStore.java:50)
      at io.fabric8.kubernetes.client.informers.cache.ProcessorStore.update(ProcessorStore.java:28)
      at io.fabric8.kubernetes.client.informers.cache.Reflector.lambda$listSyncAndWatch$0(Reflector.java:85)
      at java.util.ArrayList.forEach(ArrayList.java:1511)
      at io.fabric8.kubernetes.client.informers.cache.Reflector.listSyncAndWatch(Reflector.java:82)
      at io.fabric8.kubernetes.client.informers.impl.DefaultSharedIndexInformer.run(DefaultSharedIndexInformer.java:146)
      at io.javaoperatorsdk.operator.processing.event.source.informer.InformerWrapper.start(InformerWrapper.java:36)
      at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
      at java.util.concurrent.ConcurrentHashMap$ValueSpliterator.forEachRemaining(ConcurrentHashMap.java:3612)
      at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
      at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290)
      at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:754)
      at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373)
      at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:686)
      at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159)
      at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173)
      at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
      at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596)
      at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:765)
      at io.javaoperatorsdk.operator.processing.event.source.informer.InformerManager.start(InformerManager.java:44)
      at io.javaoperatorsdk.operator.processing.event.source.informer.ManagedInformerEventSource.start(ManagedInformerEventSource.java:75)
      at io.javaoperatorsdk.operator.processing.event.NamedEventSource.start(NamedEventSource.java:20)
      at io.javaoperatorsdk.operator.processing.event.EventSourceManager.start(EventSourceManager.java:68)
      at io.javaoperatorsdk.operator.processing.Controller.start(Controller.java:305)
      at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
      at java.util.HashMap$ValueSpliterator.forEachRemaining(HashMap.java:1779)
      at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
      at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290)
      at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:754)
      at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373)
      at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:686)
      at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159)
      at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173)
      at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
      at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596)
      at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:765)
      at io.javaoperatorsdk.operator.Operator$ControllerManager.start(Operator.java:219)
      at io.javaoperatorsdk.operator.Operator.start(Operator.java:100)
      at io.quarkiverse.operatorsdk.runtime.AppEventListener.onStartup(AppEventListener.java:26)
      at io.quarkiverse.operatorsdk.runtime.AppEventListener_Observer_onStartup_9c5d3dd7eff7dbe02976f59f48aec5b70eb9c99f.notify(Unknown Source)
      at io.quarkus.arc.impl.EventImpl$Notifier.notifyObservers(EventImpl.java:323)
      at io.quarkus.arc.impl.EventImpl$Notifier.notify(EventImpl.java:305)
      at io.quarkus.arc.impl.EventImpl.fire(EventImpl.java:73)
      at io.quarkus.arc.runtime.ArcRecorder.fireLifecycleEvent(ArcRecorder.java:130)
      at io.quarkus.arc.runtime.ArcRecorder.handleLifecycleEvents(ArcRecorder.java:99)
      at io.quarkus.deployment.steps.LifecycleEventsBuildStep$startupEvent1144526294.deploy_0(Unknown Source)
      at io.quarkus.deployment.steps.LifecycleEventsBuildStep$startupEvent1144526294.deploy(Unknown Source)
      at io.quarkus.runner.ApplicationImpl.doStart(Unknown Source)
      at io.quarkus.runtime.Application.start(Application.java:101)
      at io.quarkus.runtime.ApplicationLifecycleManager.run(ApplicationLifecycleManager.java:103)
      at io.quarkus.runtime.Quarkus.run(Quarkus.java:67)
      at io.quarkus.runtime.Quarkus.run(Quarkus.java:41)
      at io.quarkus.runtime.Quarkus.run(Quarkus.java:120)
      at io.quarkus.runner.GeneratedMain.main(Unknown Source)

Output of uname -a or ver

No response

Output of java -version

No response

GraalVM version (if different from Java)

No response

Quarkus version or git rev

No response

Build tool (ie. output of mvnw --version or gradlew --version)

No response

Additional information

No response

[quarkus-bot[bot]]

/cc @geoand, @iocanel

[metacosm]

Note that a similar issue can be seen in https://github.com/marcnuri-demo/kubernetes-controller-jl-port if you comment out these lines: https://github.com/marcnuri-demo/kubernetes-controller-jl-port/blob/main/src/main/java/com/marcnuri/demo/booternetes/port/KubernetesControllerApplication.java#L34-L39 which seems to point out to GraalVM needing some "warm-up" code to load the classes required by the informers to work properly. Note also that these classes should already be properly registered for reflection, since the code works fine with the "warm-up" code.

[geoand]

Where is the registration of the model class for reflection coming from?

[metacosm]

Where is the registration of the model class for reflection coming from?

In the case of the application linked to in the description field, the error occurs before any custom resources are created so the registration for reflection is supposed to be handled by the Kubernetes client extension, as far as I'm aware: the only resources dealt with are CRDs and ConfigMap.

It appears the deserialisation error happens when the informer starts which results in it trying to list all existing config maps and deserialise them in a list. The list is created but the items are null which leads to the NPE later on when the operator SDK tries to manipulate the list item when it's notified by the informer that a resource has been added to the watched set.

[geoand]

so the registration for reflection is supposed to be handled by the Kubernetes client extension

This is sort of true, but it's not the whole story. Quarkus does not register the entire Kubernetes model for reflection, as that model is huge. It needs to know what is actually being used in order to register the proper classes. The way it knows is by looking at the parameter types, return types or generic types of known methods / interfaces. If the application in question does not fall under one of the known categories, it will need to register reflection manually.

[metacosm]

So that might be what's going on… and hence, why "warming" the code results in the proper result… That said, which interfaces / methods are considered? I think it would make sense to add the Informer registration methods to that set since it seems they're not part of it and Informers are quite widely used, in particular for operators.

[geoand]

That said, which interfaces / methods are considered?

See https://github.com/quarkusio/quarkus/blob/main/extensions/kubernetes-client/deployment/src/main/java/io/quarkus/kubernetes/client/deployment/KubernetesClientProcessor.java#L42

I think it would make sense to add the Informer registration methods to that set since it seems they're not part of it and Informers are quite widely used, in particular for operators

If you show me a simple example, we can work from there to make it happen.

[metacosm]

There are some issues with the current code. In particular, when it comes to processing CustomResource "implementors", which looking at git blame, is actually my own fault! 😅 Indeed, the code tries to extract types from interface implementations but CustomResource is an abstract class, not an interface so the code doesn't work as expected. Also, in the case of CustomResource (and, ultimately, abstract classes that we want to process in a similar way), we also want to register the extending class itself, not just its type parameters. This is addressed by #26188.

There are more issues at play but, after discussing with @geoand, we agreed that it makes more sense to provide the required support as part of the Quarkus extension for the Java Operator SDK as the required changes would not be appropriate, nor practical, in the Kubernetes client extension.

[metacosm]

Note that the issue is still occurring after changes to the Quarkus operator SDK extension and I would appreciate help solving this issue.

[geoand]

What changes were made and what is the current problem?

[metacosm]

26188 was implemented in Quarkus and https://github.com/quarkiverse/quarkus-operator-sdk/pull/364 was merged to better register classes for reflection but the core issue of the informers not being able to deserialize the resources still persist.

[geoand]

So there is still an issue where reflection cannot determine the properties / methods of the classes?

If so, do the steps outlined in the description still reflect how the issue can be reproduced?

[metacosm]

Let's close this for now, since I'm not sure this still happens. Will open a new issue if needed.