search

neo4j-contrib / neo4j-streams

Neo4j Kafka Connector
https://neo4j.com/docs/kafka
Apache License 2.0
173 stars 71 forks source link

[3.5.5] Error initializing the streaming producer: Failed to create new KafkaAdminClient #277

Closed Tin-Nguyen closed 4 years ago

Tin-Nguyen commented 4 years ago

Guidelines

I'm trying to upgrade the Neo4j Streams plugin from 3.5.4 to 3.5.5 that has just been released 4 days ago for my Neo4j 3.5.6-enterprise. However I got the below error when starting Neo4j

org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient
        at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:407)
        at org.apache.kafka.clients.admin.AdminClient.create(AdminClient.java:55)
        at streams.kafka.KafkaAdminService.<init>(KafkaAdminService.kt:18)
        at streams.kafka.KafkaEventRouter.start(KafkaEventRouter.kt:46)
        at streams.StreamsEventRouterLifecycle.start(StreamsExtensionFactory.kt:48)
        at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:452)
        at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:111)
        at org.neo4j.kernel.extension.AbstractKernelExtensions.start(AbstractKernelExtensions.java:82)
        at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:452)
        at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:111)
        at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:452)
        at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:111)
        at org.neo4j.kernel.NeoStoreDataSource.start(NeoStoreDataSource.java:444)
        at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:452)
        at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:111)
        at org.neo4j.kernel.impl.transaction.state.DataSourceManager.start(DataSourceManager.java:116)
        at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:452)
        at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:111)
        at org.neo4j.graphdb.facade.GraphDatabaseFacadeFactory.initFacade(GraphDatabaseFacadeFactory.java:225)
        at com.neo4j.commercial.edition.CommercialGraphDatabase.<init>(CommercialGraphDatabase.java:20)
        at com.neo4j.server.database.CommercialGraphFactory.newGraphDatabase(CommercialGraphFactory.java:40)
        at org.neo4j.server.database.LifecycleManagingDatabase.start(LifecycleManagingDatabase.java:90)
        at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:452)
        at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:111)
        at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:180)
        at org.neo4j.server.ServerBootstrapper.start(ServerBootstrapper.java:124)
        at org.neo4j.server.ServerBootstrapper.start(ServerBootstrapper.java:91)
        at com.neo4j.server.enterprise.CommercialEntryPoint.main(CommercialEntryPoint.java:22)
Caused by: org.apache.kafka.common.config.ConfigException: No resolvable bootstrap urls given in bootstrap.servers
        at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:88)
        at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:47)
        at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:367)
        ... 27 more

Currently used versions

Versions

conker84 commented 4 years ago

hi @Tin-Nguyen can you share your compose file?

Tin-Nguyen commented 4 years ago

hi @conker84 , please check the docker compose file here

Also, I created a custom Dockerfile as below

FROM neo4j:3.5.6-enterprise

ADD neo4j-streams-3.5.5.jar /var/lib/neo4j/plugins/neo4j-streams-3.5.5.jar
ADD entity-manager-0.0.66.jar /var/lib/neo4j/plugins/entity-manager-0.0.66.jar
ADD ./docker-setup.sh /docker-setup.sh

RUN /docker-setup.sh
conker84 commented 4 years ago

You're missing two env params into the neo4j service:

      NEO4J_kafka_zookeeper_connect: zookeeper:2181
      NEO4J_kafka_bootstrap_servers: broker:29092
Tin-Nguyen commented 4 years ago

@conker84 , I did add it into neo4j.conf already. Is it deprecated on the latest version of the plugin (3.5.5)? Below is my current configuration in neo4j.conf


# Neo4j-Streams Kafka Connection
kafka.zookeeper.connect=zookeeper:2181
kafka.bootstrap.servers=kafka:29092
kafka.schema.registry.url=http://schema-registry:8081

# Neo4j-Streams Producer
# streams.source.topic.nodes.<TOPIC_NAME>=<PATTERN>
# streams.source.topic.relationships.<TOPIC_NAME>=<PATTERN>
streams.source.topic.nodes.events.pmn-node-goal=Goal{*}
streams.source.topic.relationships.events.pmn-rel-goalof=GOAL_OF{*}
streams.source.enabled=true
# polling interval (ms)
streams.source.schema.polling.interval=10000

# Neo4j-Streams Consumer
kafka.auto.offset.reset=latest
kafka.group.id=neo4j-streams
kafka.enable.auto.commit=true
kafka.value.deserializer=io.confluent.kafka.serializers.KafkaAvroDeserializer
Tin-Nguyen commented 4 years ago

it works on my local now after applied the ENV vars you mentioned above @conker84 . It seems the plugin deprecated those configurations below in neo4j.conf.

kafka.zookeeper.connect=zookeeper:2181
kafka.bootstrap.servers=kafka:29092
kafka.schema.registry.url=http://schema-registry:8081

Anyway, thanks for supporting on this 👍

conker84 commented 4 years ago

We didn't deprecate those configs, I'm guessing where you're using the neo4j.conf file because I can't see any reference into the compose file and the Dockerfile. I think that there is a problem in your container configuration. Properties like NEO4J_kafka_zookeeper_connect are transformed into kafka.zookeeper.connect into the neo4j.conf file by the Neo4j Container.