No httpclient implementations found on the context classloader

[jchipmunk]

Hi,

I use kafka-kubernetes-config-provider v.1.0.0 deployed in the directory specified in the plugin.path worker configuration property. This version works great!

After updating to v.1.0.1, the Kafka Connect application startup failed with the following error:

[2022-08-15T14:08:43,248][ERROR][category=org.apache.kafka.connect.cli.ConnectDistributed] Stopping due to errorio.fabric8.kubernetes.client.KubernetesClientException: No httpclient implementations found on the context classloader, please ensure your classpath includes an implementation jar
    at io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:171)
    at io.fabric8.kubernetes.client.DefaultKubernetesClient.<init>(DefaultKubernetesClient.java:153)
    at io.fabric8.kubernetes.client.DefaultKubernetesClient.<init>(DefaultKubernetesClient.java:145)
    at io.strimzi.kafka.AbstractKubernetesConfigProvider.configure(AbstractKubernetesConfigProvider.java:63)
    at io.strimzi.kafka.KubernetesConfigMapConfigProvider.configure(KubernetesConfigMapConfigProvider.java:17)
    at org.apache.kafka.common.config.AbstractConfig.instantiateConfigProviders(AbstractConfig.java:572)
    at org.apache.kafka.common.config.AbstractConfig.resolveConfigVariables(AbstractConfig.java:515)
    at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:107)
    at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:129)
    at org.apache.kafka.connect.runtime.WorkerConfig.<init>(WorkerConfig.java:452)
    at org.apache.kafka.connect.runtime.distributed.DistributedConfig.<init>(DistributedConfig.java:405)
    at org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:95)
    at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:80)

The reason for the error is that io.fabric8.kubernetes.client.utils.HttpClientUtils class loads HttpClient implementation using context classloader for current thread:

public static HttpClient createHttpClient(Config config) {
  ServiceLoader<HttpClient.Factory> loader = ServiceLoader.load(HttpClient.Factory.class);
  ...

In this particular case the current thread is org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader class because org.apache.kafka.connect.runtime.isolation.PluginClassLoader class (who knows about libs of kafka-kubernetes-config-provider) is only used to instantiate the ConfigProvider. For more details, see org.apache.kafka.common.config.AbstractConfig class:

private Map<String, ConfigProvider> instantiateConfigProviders(Map<String, String> indirectConfigs, Map<String, ?> providerConfigProperties) {
  ...
  ConfigProvider provider = Utils.newInstance(entry.getValue(), ConfigProvider.class);  // PluginClassLoader is used here
  provider.configure(configProperties);                                                 // DelegatingClassLoader is used here
  ...

[scholzj]

How do you add it to the plugin directory? The libraries are int he ZIp file and they are in the dependency tree:

[INFO] +- io.strimzi:kafka-kubernetes-config-provider:jar:1.0.1:compile
[INFO] |  \- io.fabric8:kubernetes-client:jar:6.0.0:compile
[INFO] |     +- io.fabric8:kubernetes-client-api:jar:6.0.0:compile
[INFO] |     |  +- io.fabric8:kubernetes-model-core:jar:6.0.0:compile
[INFO] |     |  |  \- io.fabric8:kubernetes-model-common:jar:6.0.0:compile
[INFO] |     |  +- io.fabric8:kubernetes-model-apps:jar:6.0.0:compile
[INFO] |     |  +- io.fabric8:kubernetes-model-batch:jar:6.0.0:compile
[INFO] |     |  +- io.fabric8:kubernetes-model-certificates:jar:6.0.0:compile
[INFO] |     |  \- io.fabric8:kubernetes-model-extensions:jar:6.0.0:compile
[INFO] |     +- io.fabric8:kubernetes-httpclient-okhttp:jar:6.0.0:runtime
[INFO] |     |  +- com.squareup.okhttp3:okhttp:jar:3.12.12:runtime
[INFO] |     |  |  \- com.squareup.okio:okio:jar:1.15.0:runtime
[INFO] |     |  \- com.squareup.okhttp3:logging-interceptor:jar:3.12.12:runtime
[INFO] |     \- io.fabric8:zjsonpatch:jar:0.3.0:compile

So whether you use the ZIP file or pull from Maven, it should be there without the need for any changes. I do not seem to have any problems with using it without adding / specifying the additional dependency.

[jchipmunk]

How do you add it to the plugin directory? The libraries are int he ZIp file and they are in the dependency tree:

[INFO] +- io.strimzi:kafka-kubernetes-config-provider:jar:1.0.1:compile
[INFO] |  \- io.fabric8:kubernetes-client:jar:6.0.0:compile
[INFO] |     +- io.fabric8:kubernetes-client-api:jar:6.0.0:compile
[INFO] |     |  +- io.fabric8:kubernetes-model-core:jar:6.0.0:compile
[INFO] |     |  |  \- io.fabric8:kubernetes-model-common:jar:6.0.0:compile
[INFO] |     |  +- io.fabric8:kubernetes-model-apps:jar:6.0.0:compile
[INFO] |     |  +- io.fabric8:kubernetes-model-batch:jar:6.0.0:compile
[INFO] |     |  +- io.fabric8:kubernetes-model-certificates:jar:6.0.0:compile
[INFO] |     |  \- io.fabric8:kubernetes-model-extensions:jar:6.0.0:compile
[INFO] |     +- io.fabric8:kubernetes-httpclient-okhttp:jar:6.0.0:runtime
[INFO] |     |  +- com.squareup.okhttp3:okhttp:jar:3.12.12:runtime
[INFO] |     |  |  \- com.squareup.okio:okio:jar:1.15.0:runtime
[INFO] |     |  \- com.squareup.okhttp3:logging-interceptor:jar:3.12.12:runtime
[INFO] |     \- io.fabric8:zjsonpatch:jar:0.3.0:compile

So whether you use the ZIP file or pull from Maven, it should be there without the need for any changes. I do not seem to have any problems with using it without adding / specifying the additional dependency.

Some details of my configuration

I have the directory that is used for plugins (connectors, converters, transformations, config providers) and is configured accordingly in the connect-distributed.properties file:

plugin.path=/opt/kafka/plugins

The kafka-kubernetes-config-provider plugin with all the necessary libraries is located in this directory:

bash-5.1$ ls -la /opt/kafka/plugins/kafka-kubernetes-config-provider-1.0.1
total 9452
drwxr-xr-x 2 root root    4096 Aug 15 13:50 .
drwxr-xr-x 3 root root      50 Aug 15 13:50 ..
-rw-r--r-- 1 1001  116   75714 Aug  3 19:30 jackson-annotations-2.13.3.jar
-rw-r--r-- 1 1001  116  374895 Aug  3 19:30 jackson-core-2.13.3.jar
-rw-r--r-- 1 1001  116 1536542 Aug  3 19:30 jackson-databind-2.13.3.jar
-rw-r--r-- 1 1001  116   52020 Aug  3 19:30 jackson-dataformat-yaml-2.13.3.jar
-rw-r--r-- 1 1001  116  121201 Aug  3 19:30 jackson-datatype-jsr310-2.13.3.jar
-rw-r--r-- 1 1001  116   11054 Aug  3 19:30 kafka-kubernetes-config-provider-1.0.1.jar
-rw-r--r-- 1 1001  116  393908 Aug  3 19:30 kubernetes-client-6.0.0.jar
-rw-r--r-- 1 1001  116  405972 Aug  3 19:30 kubernetes-client-api-6.0.0.jar
-rw-r--r-- 1 1001  116   36110 Aug  3 19:30 kubernetes-httpclient-okhttp-6.0.0.jar
-rw-r--r-- 1 1001  116  444555 Aug  3 19:30 kubernetes-model-apps-6.0.0.jar
-rw-r--r-- 1 1001  116  248112 Aug  3 19:30 kubernetes-model-batch-6.0.0.jar
-rw-r--r-- 1 1001  116  145404 Aug  3 19:30 kubernetes-model-certificates-6.0.0.jar
-rw-r--r-- 1 1001  116   19677 Aug  3 19:30 kubernetes-model-common-6.0.0.jar
-rw-r--r-- 1 1001  116 4331115 Aug  3 19:30 kubernetes-model-core-6.0.0.jar
-rw-r--r-- 1 1001  116  544079 Aug  3 19:30 kubernetes-model-extensions-6.0.0.jar
-rw-r--r-- 1 1001  116   12483 Aug  3 19:30 logging-interceptor-3.12.12.jar
-rw-r--r-- 1 1001  116  427674 Aug  3 19:30 okhttp-3.12.12.jar
-rw-r--r-- 1 1001  116   88732 Aug  3 19:30 okio-1.15.0.jar
-rw-r--r-- 1 1001  116  331605 Aug  3 19:30 snakeyaml-1.30.jar
-rw-r--r-- 1 1001  116   35518 Aug  3 19:30 zjsonpatch-0.3.0.jar

i.e. the kubernetes-httpclient-okhttp library is also provided.

Regarding my explanation above, the DelegatingClassLoader loads any classes other than plugins (connectors, converters, transformations, config providers) using the application classloader (/opt/kafka/libs). So if you place the kafka-kubernetes-config-provider plugin with all the necessary libraries into Kafka's libs directory, then you will not have such a problem.

I also note that DefaultKubernetesClient class is now marked as deprecated.

[scholzj]

Ok, I guess I get the point now. But it also makes it impossible to use it with the other than OkHttp implementations because you hardcode a single implementation I guess which might be a problem for other users.

[jchipmunk]

It seemed to me that this is unlikely but if such user scenarios exist, then the problem should be fixed deeper at the kubernetes-client library, perhaps in the same way as it was done for KubernetesClientBuilder.

I will try to create the required request to the kubernetes-client repository.

[scholzj]

TBH, I did not tried the other clients. One of the appeals would be that the Jetty client is already part of Kafka. So using that would decrease the footprint. But it depends on Jetty 11, so not sure it would work with Jetty 9 which is used by Kafka (and probably will be used until Kafka 4.0 when Java 8 is dropped). So I guess until then we can maybe proceed with your PR.

[scholzj]

Fixed in #13