Expire module not working correctly for multi-node Neo4j cluster

[rittinger]

We are having issues with the expire module (3.0.6.43.3) running in a multi-node, neo4j cluster (3.0.6). When setting the attribute 'ttl' on a node in the graph with the value of 10000 (10 seconds), the node is deleted after 10 seconds in a single node neo4j cluster (which is correct). The same test with multiple neo4j nodes results in the node being deleted instantly (not waiting for 10 seconds).

Test cypher: create (t:Test) return t // node is returned match (t:Test) return t // again node is returned match (t:Test) set t.ttl = 10000 return t // node is returned with new ttl attribute match (t:Test) return t // Even though this cypher was executed with 1 second of the previous, the node is already gone

Setting the ttl value to 40000 did result in the node existing longer, but it was not 40 seconds (more like 5-6 seconds). What is strange, we were having this problem in test/stage/prod and now the problem only exists in stage. We're wondering if maybe the servers are out of sync (their clocks), maybe this could cause the problem (as we noticed a couple of the prod boxes had been bounced).

Switching a single node from High Availability (multi-node cluster) to a single stand alone node results in the expire module working as expected. Switching back to the multi-node and the test no longer works.

Our entries in neo4j.conf: com.graphaware.runtime.enabled=true com.graphaware.module.EM.1=com.graphaware.neo4j.expire.ExpirationModuleBootstrapper com.graphaware.module.EM.relationshipTtlProperty=ttl com.graphaware.module.EM.nodeTtlProperty=ttl com.graphaware.runtime.stats.disabled=true com.graphaware.server.stats.disabled=true dbms.unmanaged_extension_classes=com.graphaware.server=/graphaware

[rittinger]

We have an update. Inadvertently we performed our test against a slave node. When adding the ttl attribute to a node on a slave instance, the graph node is deleted instantly. The same test performed against the master neo4j instance works as expected (graph node is removed 10 seconds later). Another question for Neo Tech is why are we able to write to slave nodes (missing configuration setting perhaps)? We need Graphware to confirm.

[dgordon5483]

Hi @rittinger,

Neo4j allows writes to master or slave. Writing to master directly is encouraged, since it is less overhead per write, and will allow for better write performance. Unless you specify a slave as "read only" (which you would only want to do if it was also "slave only"), it would not prevent you from writing to that slave.

I hope that clears your Neo question up.

Thanks, Dave

[rittinger]

@dgordon5483 That makes sense and we knew about the ability to make a node read only. We need input from graphaware to see why this is failing and if they have any advice for a multi-node cluster.

[ikwattro]

Hi @rittinger @dgordon5483

On the other hand, the module should only be "started" when on a master instance to avoid this type of issues, we will make a patch release of the module in the next days.

[rittinger]

Hi @ikwattro @dgordon5483 Leader election (when a slave node becomes the master) happens automatically. I don't know of any way to automatically disable / enable the expire module when this happens. or are you saying when this happens, the expire module will handle this automatically (that would be awesome!)

[bachmanm]

It will. In the next release (few days). Thanks!

[rittinger]

That's awesome, thanks!

[bachmanm]

Thanks @jasperblues for the fix. @rittinger could you try with http://products.graphaware.com/download/expire/graphaware-expire-3.0.6.43.4.jar ?

[rittinger]

tested the new expiry module on Prod and this is working as expected. We can create & set ttl from both master and slave and the node lives so long as the TTLs are set. @bachmanm

Thank you for the quick response!

[ikwattro]

Awesome, thanks for letting us know !