Neo4j-streams Sink execution hang

nylqd commented 4 years ago

I discovered that when I config neo4j-streams sink to consume cdc schema event from kafka, the sink execution is not working properly.

Expected Behavior (Mandatory)

all the create/update/delete cdc events should be handle properly.

Actual Behavior (Mandatory)

create event hang in the db.execute lambda, will not return the queryStatistics and proceed. the sink daemon job just hang in there and not consume any data from kafka.

How to Reproduce the Problem

Steps (Mandatory)

the neo4j instance using plugin sink, just use neo4j-admin restore to restore data from other db.
config plugin as sink, w/ cdc schema strategy, config log level to DEBUG
found debug log does not contains consume detail, so add debug log in function write of class StreamsEventSinkQueryExecution to log the detail of query and params, no other change.
build the plugin from branch 4.0.
restart neo4j service, and start produce cdc event to kafka topic.

Screenshots (where it's possibile)

I have add debug log in write function and you may see the log above, the result of the execute lambda is showing, but the service just hang in there, will not proceed.

Specifications (Mandatory)

Currently used versions

Versions

OS: RHEL 7.4
Neo4j: 4.0.4 enterprise
Neo4j-Streams: 4.0.3 (self build)

mroiter-larus commented 4 years ago

Hi @nylqd, could you please try with the jar provided here?

nylqd commented 4 years ago

Hi @nylqd, could you please try with the jar provided here?

thx for ur reply, I have tried the official 4.0.3 release package, and still stuck at write data at target instance.

I have done to step to verify, 1) delete a node from source instance, see reading data from topic, query statistics: <nothing happened> in target instance neo4j.log file; 2) create a node from source instance, see only reading data fro topic and then stuck here forever.

mroiter-larus commented 4 years ago

Could you please share your neo4j.conf? In particular the streams configurations? Thanks in advance

nylqd commented 4 years ago

here are the streams conf, hope it helps

# source instance conf in neo4j.conf
kafka.zookeeper.connect=....
kafka.bootstrap.servers=....

streams.sink.enabled=false
streams.source.enabled=true
streams.source.topic.nodes.<our topic>=<pattern>
streams.source.topic.relationships.<our topic>=<pattern>

# target instance conf in neo4j.conf
kafka.zookeeper.connect=....
kafka.bootstrap.servers=....

streams.source.enabled=false
streams.sink.enabled=false
streams.sink.enabled.to.<our db name>=true
streams.sink.topic.cdc.schema.to.<our db name>=<our topic>

streams.sink.errors.tolerance=all
streams.sink.errors.log.enable=true
streams.sink.errors.log.include.messages=true

mroiter-larus commented 4 years ago

@nylqd following the step you have provided i'm not able to reproduce the issue. I'm using the official 4.0.3 release package. However i noticed a difference between my source instance configuration and yours. Could you please add in the neo4j.conf of your source instance the following property and retry?

streams.source.schema.polling.interval=10000

mroiter-larus commented 4 years ago

@nylqd Any news? Thanks

nylqd commented 4 years ago

@nylqd Any news? Thanks

@mroiter-larus sorry for the delay, have been working on other project. I have just apply the config u suggest, and the problem still exists.

I'm wondering this might be an issue with write data to neo4j? target instance log shows reading data from topic then nothing but stuck.

I have test with another scenario all the same but only target instance with an empty db, and it works . The scenario described above is what we want, we just want sync data of an existing db to a different neo4j instance, is there any other solution may meet our need?

thx ur reply a lot.

mroiter-larus commented 4 years ago

@nylqd Sorry for the delay. Could you please try with the latest pre-release? You can find it here.

Thanks in advance!

conker84 commented 4 years ago

@nylqd did you create the constraints on the target database?

conker84 commented 4 years ago

@nylqd PING

nylqd commented 4 years ago

@mroiter-larus @conker84 thx for ur reply

@nylqd Sorry for the delay. Could you please try with the latest pre-release? You can find it here.

Thanks in advance!

been busy on another project, I'd like to try out the pre-release in tomorrow's office hour.

@nylqd did you create the constraints on the target database?

what I have done is using neo4j-admin backup from the source db and restore to the target db, last time I checked, the constraints are intact.

conker84 commented 4 years ago

Since we use it in our customers would be great dive into your issue to understand where is the problem. Please @nylqd let us know if you're still interested in this or we can close it

neo4j-contrib / neo4j-streams