search

lensesio / fast-data-dev

Kafka Docker for development. Kafka, Zookeeper, Schema Registry, Kafka-Connect, , 20+ connectors
https://lenses.io
Apache License 2.0
2.02k stars 331 forks source link

Connectivity Error for Connect UI #43

Open skynyrd opened 7 years ago

skynyrd commented 7 years ago

Hi,

I am trying to use fast-data-dev on Ubuntu 17.10 by sudo docker run --rm -it --net=host landoop/fast-data-dev:cp3.3 (I also used latest tag)

And Connect UI returns following error:

Kafka Connect : /api/kafka-connect
Kafka Connect Version : 0.11.0.0-cp1
Kafka Connect UI Version : 0.9.2
CONNECTIVITY ERROR

Topics UI and other components work well. What can be the problem?

skynyrd commented 7 years ago

In fact, it worked after I reboot.. Closing the issue..

skynyrd commented 7 years ago

Nope it happened again.. Any ideas?

andmarios commented 7 years ago

Hi @skynyrd

In the web interface we expose the logs directory (the link is at the bottom of the page). So maybe you could check out the Schema Registry logs to help us pinpoint the issue?

E.g try to open http://localhost:3030/logs/schema-registry.log

Please keep in mind that because supervisord doesn't give us a way to stall a program at startup, Schema Registry some times starts before the broker and exits. This isn't a problem, because supervisord will restart it. It means though that in the log you will see a few errors. We are interested only in the last one.

skynyrd commented 7 years ago

Thanks for quick response @andmarios 

Exception in thread "main" java.lang.NoClassDefFoundError: com/github/jcustenborder/kafka/connect/utils/VersionUtil
    at com.github.jcustenborder.kafka.connect.rabbitmq.RabbitMQSourceConnector.version(RabbitMQSourceConnector.java:35)
    at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.getPluginDesc(DelegatingClassLoader.java:242)
    at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:223)
    at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:198)
    at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.registerPlugin(DelegatingClassLoader.java:190)
    at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:150)
    at org.apache.kafka.connect.runtime.isolation.Plugins.<init>(Plugins.java:47)
    at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:63)
Caused by: java.lang.ClassNotFoundException: com.github.jcustenborder.kafka.connect.utils.VersionUtil
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at org.apache.kafka.connect.runtime.isolation.PluginClassLoader.loadClass(PluginClassLoader.java:62)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 8 more

This exception occurs in the logs, but it is related to the rabbitmq connector, not connect ui or another fast-data-dev component. At least it is not about connectivity error.

I have tried to run the same command in Windows and it just works.

Will going to update here if I find any solution...

Antwnis commented 7 years ago

@skynyrd have you tried to use the MQTT connector that comes with fast-data-dev?

The documentation is here, and is available on the kafka-connect-ui as MQTT source http://docs.datamountaineer.com/en/latest/mqtt.html

You will most probably, need to add the rabbitmq client lib in the classpath to provide the client connectivity to the generic MQTT source. i.e. https://github.com/Landoop/fast-data-dev#enable-additional-connectors

skynyrd commented 7 years ago

Hey @Antwnis, sorry for the late response. No, I didn't. I am going to try it. Thanks!

Antwnis commented 7 years ago

Hey @skynyrd

About the connectivity error, what happens is that we have included a collection of 25+ Kafka Connectors in fast-data-dev that we run integration tests

On Kafka 0.11 Kafka Connect added classpath isolation in order to allow via OSGI connectors with conflicting libraries to be isolated 👍 The side effect of that is that it takes ~10-15 seconds for "Kafka Connect" to load any connector in the classpath

Given that there are 25+ connectors, that results into 3-4 minutes of classpath loading by Kafka Connect, before it's operational. If you give it a bit of time, everything will start-up

Regarding rabbitmq, i'm curious if you gave the MQTT connector a try

skynyrd commented 6 years ago

@Antwnis thanks for your reply. Yes, after giving the time, it worked. However, I have also realized that if a custom connector had some problems, connectivity error can occur. In fact, I couldn't try MQTT connector as we have chosen rabbitmq source connector as a team, I am going to try it in my next project. Thanks a lot.

andmarios commented 6 years ago

Between, you can disable connectors using the DISABLE flag. This could speed things up considerably.

E.g to disable all connectors but the file one:

-e DISABLE=azure-documentdb,blockchain,bloomberg,cassandra,coap,druid,elastic,elastic5,ftp,hazelcast,hbase,influxdb,jms,kudu,mongodb,mqtt,redis,rethink,voltdb,yahoo,hdfs,jdbc,elasticsearch,s3,twitter
psmod2 commented 6 years ago

@skynyrd - I'm currently facing the exact same issue as you per above. How did you resolve this? Exception in thread "main" java.lang.NoClassDefFoundError: com/github/jcustenborder/kafka/connect/utils/VersionUtil

moutasemalakkad commented 2 years ago

getting this error:

org.apache.kafka.common.config.ConfigException: No supported Kafka endpoints are configured. Either kafkastore.bootstrap.servers must have at least one endpoint matching kafkastore.security.protocol or broker endpoints loaded from ZooKeeper must have at least one endpoint matching kafkastore.security.protocol. at io.confluent.kafka.schemaregistry.storage.KafkaStore.endpointsToBootstrapServers(KafkaStore.java:350) at io.confluent.kafka.schemaregistry.storage.KafkaStore.(KafkaStore.java:138) at io.confluent.kafka.schemaregistry.storage.KafkaSchemaRegistry.(KafkaSchemaRegistry.java:150) at io.confluent.kafka.schemaregistry.rest.SchemaRegistryRestApplication.setupResources(SchemaRegistryRestApplication.java:54) at io.confluent.kafka.schemaregistry.rest.SchemaRegistryRestApplication.setupResources(SchemaRegistryRestApplication.java:38) at io.confluent.rest.Application.createServer(Application.java:157) at io.confluent.kafka.schemaregistry.rest.SchemaRegistryMain.main(SchemaRegistryMain.java:44)

lifeofzi commented 1 year ago

Try restarting after docker pull landoop/fast-data-dev Initially, it did show the same connectivity issue for me, but on waiting it was cleared and now I am able to see the connect UI. I checked the logs and could not see an error logged. On the console, when it was showing the connectivity error it was showing a 502. Also check if you have any other service running on this port.