Exception: "No NetworkModule installed" in Mqtt-Source-Connector when building version 5.0.0 from source

[LarissaVerhuelsdonk]

We're currently planning on extending the stream-reactor framework with some of our own code. Obviously, this requires us to be able to build from source. However, Version 5.0.1 (using git tag "5.0.1") does not seem to build the "mqtt" project correctly.

The build process (sbt assembly) finishes without any errors and produces a fat-JAR, as expected. When installing the JAR as a Kafka-Connect plugin, it is loaded normally and without error. When we start an Mqtt-Source-Connector, however, it instantly reduces the below exception:

java.lang.IllegalArgumentException: no NetworkModule installed for scheme \"ssl\" of URI \"ssl://redacted-mqtt-host:8883\"\n\tat org.eclipse.paho.client.mqttv3.internal.NetworkModuleService.validateURI(NetworkModuleService.java:74)\n\tat org.eclipse.paho.client.mqttv3.MqttConnectOptions.setServerURIs(MqttConnectOptions.java:589)\n\tat com.datamountaineer.streamreactor.connect.mqtt.connection.MqttClientConnectionFn$.buildBaseClient(MqttClientConnectionFn.scala:81)\n\tat com.datamountaineer.streamreactor.connect.mqtt.connection.MqttClientConnectionFn$.apply(MqttClientConnectionFn.scala:37)\n\tat com.datamountaineer.streamreactor.connect.mqtt.source.MqttSourceTask.$anonfun$start$6(MqttSourceTask.scala:85)\n\tat com.datamountaineer.streamreactor.connect.mqtt.source.MqttManager.<init>(MqttManager.scala:47)\n\tat com.datamountaineer.streamreactor.connect.mqtt.source.MqttSourceTask.start(MqttSourceTask.scala:85)\n\tat org.apache.kafka.connect.runtime.AbstractWorkerSourceTask.initializeAndStart(AbstractWorkerSourceTask.java:271)\n\tat org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:187)\n\tat org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:244)\n\tat org.apache.kafka.connect.runtime.AbstractWorkerSourceTask.run(AbstractWorkerSourceTask.java:73)\n\tat java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)\n\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat java.base/java.lang.Thread.run(Thread.java:829)\n

I managed to reliably produce the above exception for all schemes (tcp, ssl, etc.) for versions 5.0.0 and 5.0.1, while another developer (@DerPate) managed to reproduce the exception for version 5.0.0 on his own, completely separate setup.

[davidsloan]

@LarissaVerhuelsdonk can you share a configuration that has this problem please? Also please state which broker and what version you are using. There is a working configuration in the MqttTest that shows that this is working with TCP. https://github.com/lensesio/stream-reactor/blob/fd798a5b3101084b7c1409ce223feb276ea5d276/kafka-connect-mqtt/src/fun/scala/com/datamountaineer/streamreactor/connect/MqttTest.scala#L20

[DerPate]

HI @davidsloan i will answer for @LarissaVerhuelsdonk. We use following configuration:

          connect.mqtt.hosts: "ssl://{{ mqtt_cluster_dns_name }}:8883"
          connect.mqtt.username: "kafka_connect_emobv2_symbols"
          connect.mqtt.client.id: "kafka_connect_emobv2_symbols"

          connect.mqtt.ssl.cert: "/usr/share/certs/{{ env }}-kafka-zookeeper.crt"
          connect.mqtt.ssl.key: "/usr/share/certs/{{ env }}-kafka-zookeeper.pem"
          connect.mqtt.ssl.ca.cert: "/usr/share/certs/cumulative-{{ env }}-ca.crt"

And i checked against the test cases and saw no difference between the two. Our problem is more in the category that we cannot reproduce the build and use it rather that out configuration is broken. As we both are not able to build the connector on our machines and use it due to the aforementioned bug.

[davidsloan]

@DerPate & @LarissaVerhuelsdonk just to confirm, if you try it with the released jar from github does it work?

Can you tell us which operating system and which JDK are you using, and what process are you using to build the assembly?

For our CI build we are using JDK 17 ('temurin' distribution) running on Ubuntu.

To build the command is

sbt "project mqtt" assembly

[DerPate]

@davidsloan Yes the precomplied version from GitHub works indeed. On my system I use following setup: sbt 1.6.2 (Eclipse Adoptium Java 17.0.9) on Arch Linux

[LarissaVerhuelsdonk]

@davidsloan For completeness' sake, I've got the same problems with: sbt 1.6.2 (Eclipse Adoptium Java 17.0.9) on Windows 10 Enterprise, Version 22H2

For the build process I'm using the IntelliJ sbt-Plugin. Commands are project mqtt to move into the sub-project and assembly to build the JAR. However, @DerPate also tried standard sbt from the command line and we get the same error in both cases.

[davidsloan]

Hi @DerPate and @LarissaVerhuelsdonk .

Sorry, I'm not sure about this one but I can offer some tips to help you troubleshoot the issues?

1. Try reproducing our build environment exactly including the same JDK.
2. Compare the Jar file you are producing with the jar file we released. You should see what the differences are. The error makes me think that maybe one of the META-INF/services files is being skipped? There's no obfuscation or anything like that you have added?
3. I think it might be worth taking this one upstream to Eclipse Paho MQTT Client There are some similar issues open:

https://github.com/eclipse/paho.mqtt.java/issues?q=is%3Aissue+is%3Aopen+networkmodule

4. Also you might consider fiddling with the version of Paho client (in Dependencies.scala) and see if you can find an older one that doesn't give you the problem.

Please keep us updated!

[LarissaVerhuelsdonk]

@davidsloan it's taken us a while as other things gained higher priority in late December, but we managed to find the fault on our side. Apparently, we had a faulty dependency in our artifact repository, which consistently messed up our build.

I'll close this issue, as there are no other problems. Sorry for the confusion.