Connector issue for Confluent Platform 6.0.1

[brifordwylie]

Hi, thanks for making this cool Redis connector. So I'm fairly new at using Connectors so this issue could certainly be pilot error. I'm getting an error when the connector spins up. I'm using the Confluent Platform Docker compose for 6.0.1 (https://github.com/confluentinc/cp-all-in-one) and I've added a redis container. I'll attach the docker-compose.yml file so you can replicate. I'll also attach the config I'm using for the connector. I'm zipping them so Github will allow the attachment... redis_connector_issue.zip

Here's the error I'm getting when the connector spins up...

connect            | [2021-02-08 16:30:51,255] ERROR WorkerSourceTask{id=redis_connector-0} Task threw an uncaught and unrecoverable exception (org.apache.kafka.connect.runtime.WorkerTask)
connect            | java.lang.ClassCastException: class io.netty.channel.epoll.EpollEventLoopGroup cannot be cast to class
 io.netty.channel.EventLoopGroup (io.netty.channel.epoll.EpollEventLoopGroup is in unnamed module of loader 'app';
 io.netty.channel.EventLoopGroup is in unnamed module of loader
 org.apache.kafka.connect.runtime.isolation.PluginClassLoader @2d532099)
connect            |    at io.lettuce.core.resource.DefaultEventLoopGroupProvider.getOrCreate(DefaultEventLoopGroupProvider.java:151)
connect            |    at io.lettuce.core.resource.DefaultEventLoopGroupProvider.allocate(DefaultEventLoopGroupProvider.java:89)
connect            |    at io.lettuce.core.AbstractRedisClient.doGetEventExecutor(AbstractRedisClient.java:275)
connect            |    at io.lettuce.core.AbstractRedisClient.getEventLoopGroup(AbstractRedisClient.java:264)
connect            |    at io.lettuce.core.AbstractRedisClient.channelType(AbstractRedisClient.java:246)
connect            |    at io.lettuce.core.RedisClient.connectStatefulAsync(RedisClient.java:315)
connect            |    at io.lettuce.core.RedisClient.connectPubSubAsync(RedisClient.java:407)
connect            |    at io.lettuce.core.RedisClient.connectPubSub(RedisClient.java:328)
connect            |    at io.github.jaredpetersen.kafkaconnectredis.source.RedisSourceTask.start(RedisSourceTask.java:76)
connect            |    at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:232)
connect            |    at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:185)
connect            |    at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:235)
connect            |    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
connect            |    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
connect            |    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
connect            |    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
connect            |    at java.base/java.lang.Thread.run(Thread.java:834)

[jaredpetersen]

@brifordwylie Sorry to hear that you're having a hard time getting everything working.

I took a look at your docker-compose.yml file but I'm not seeing how the connector is installed. You have a Kafka Connect service using the image cnfldemos/cp-server-connect-datagen:0.4.0-6.0.1 but that image only contains Kafka Connect Datagen. I also see that there aren't any volume mounts defined so I don't believe you're mounting any other connectors via the file system.

I'm not familiar with that error and the stacktrace is unfortunately not helpful. It looks like the connector is running but there's not enough here for me to be able to reproduce the issue so that I can investigate. If you can provide a Docker Compose file that is ready to go with instructions, I'm happy to look further 🙂

Have you tried the demo? While it uses Kubernetes instead of Docker Compose, I still think it would be worth your time and has instructions suitable for those new to Kubernetes. It's using a very pared down Kafka Connect image that just contains jaredpetersen/kafka-connect-redis. You can see how it's configured here and how the plugin is installed here.

[brifordwylie]

@jaredpetersen so the reason I'm using https://github.com/confluentinc/cp-all-in-one is because it seems like the best 'standard' to start from.

So starting from the CP Platform as a base.. to install your connector just hop onto the docker container like this...

$ docker exec -it connect bash

[~]$ confluent-hub install jaredpetersen/kafka-connect-redis:1.1.0

then restart the container

docker restart connect

tell connect to activate the connector

curl -sX POST http://localhost:8083/connectors -d @redis_connector.json --header "Content-Type: application/json" | jq

So, I think that's it... having your connector running/tested on the CP 'standard' setup might be a good thing... let me know if you can reproduce with these instructions.

[jaredpetersen]

Ran your example tonight but the connect container dies during startup before I even get to the point of installing jaredpetersen/kafka-connect-redis. I've tried this multiple times and seems to die at different points during the startup process.

Death during plugin load (nothing else logged after this):

[2021-02-10 05:33:21,014] INFO Added plugin 'org.apache.kafka.connect.rest.basic.auth.extension.BasicAuthSecurityRestExtension' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
[2021-02-10 05:33:21,015] INFO Loading plugin from: /usr/share/java/kafka-rest-bin (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
[2021-02-10 05:33:21,022] INFO Registered loader: PluginClassLoader{pluginLocation=file:/usr/share/java/kafka-rest-bin/} (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
[2021-02-10 05:33:21,024] INFO Loading plugin from: /usr/share/java/rest-utils (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)

Another death:

[2021-02-10 05:38:59,998] INFO Loading plugin from: /usr/share/java/rest-utils (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
[2021-02-10 05:39:00,236] INFO Registered loader: PluginClassLoader{pluginLocation=file:/usr/share/java/rest-utils/} (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
[2021-02-10 05:39:00,237] INFO Loading plugin from: /usr/share/java/confluent-security (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)

The exit code is 137.

The broker itself also keeps dying:

[2021-02-10 05:38:11,478] INFO [GroupCoordinator 1]: Stabilized group _confluent-controlcenter-6-0-1-1 generation 2 (__consumer_offsets-40) (kafka.coordinator.group.GroupCoordinator)
[2021-02-10 05:38:11,505] INFO [GroupCoordinator 1]: Assignment received from leader for group _confluent-controlcenter-6-0-1-1 for generation 2 (kafka.coordinator.group.GroupCoordinator)
[2021-02-10 05:38:26,101] INFO Waiting for 32 seconds for metric reporter topic _confluent-telemetry-metrics to become available. (io.confluent.cruisecontrol.metricsreporter.ConfluentMetricsSamplerBase)

The exit code is also 137.

I've attempted this startup process several times without any success. I'm not particularly confident in this docker-compose setup so I will try to provide a docker-compose setup as a companion to the Kubernetes demo.

[jaredpetersen]

@brifordwylie I created a Docker Compose setup based on the Kubernetes demo. You can check it out on the docker-compose-mvp branch. There's a README there that has the basic steps for using the source connector since that's what you were interested in with your example.

I did not see any of the errors that you mentioned so I'm assuming that something is just broken with your example.

It's unlikely that this Docker Compose setup will be an official alternative for the demo. I'm not a huge fan of Docker Compose since it's harder to create clusters of apps than a tool like Kubernetes. Please clone this branch since it won't be up forever.

[brifordwylie]

@jaredpetersen thank for the effort on this. I've cloned the repo and pulled the docker-compose-mvp branch. I'll spin it all up in the next day or two, I'm sure it will work fine and I'll learn something along the way :)

Feel free to close this issue and thanks again for taking a deeper dive on this.

[jaredpetersen]

@brifordwylie Absolutely. Please do not hesitate to reach out again in the future -- I'm happy to help 😄

[ocus]

I had the same issue as the one described by @brifordwylie using https://github.com/confluentinc/cp-all-in-one.

Turns out removing CLASSPATH, CONNECT_PRODUCER_INTERCEPTOR_CLASSES and CONNECT_CONSUMER_INTERCEPTOR_CLASSES from https://github.com/confluentinc/cp-all-in-one/blob/6655d3bd9add605a016b250b5a487f2809d981ce/cp-all-in-one/docker-compose.yml#L81 fixed the ClassCastException and the connector is now working fine.

[brifordwylie]

@ocus so I tried out your suggestion (removing CLASSPATH,CONNECT_PRODUCER_INTERCEPTOR_CLASSES and CONNECT_CONSUMER_INTERCEPTOR_CLASSES) and I'm still getting this error when I activate the connector...

connect            | [2021-02-11 17:22:56,511] ERROR WorkerSourceTask{id=redis_connector-0} Task threw an uncaught and unrecoverable exception. Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask)
connect            | java.lang.ClassCastException: class io.netty.channel.epoll.EpollEventLoopGroup cannot be cast to class io.netty.channel.EventLoopGroup (io.netty.channel.epoll.EpollEventLoopGroup is in unnamed module of loader 'app'; io.netty.channel.EventLoopGroup is in unnamed module of loader org.apache.kafka.connect.runtime.isolation.PluginClassLoader @1062338c)

It's okay.. I'll try the compose file from @jaredpetersen and do some debugging to see where/when things go awry..

[Chebura]

I have same problem with cnfldemos/cp-server-connect-datagen:0.5.0-6.2.0 and jaredpetersen/jaredpetersen-redis-connector:1.2.0 installed from confluent-hub as described by instruction). Ive installed CP all-in-one.