Trigger with assignedNodeProperties is blocking further DB writes

[foobargeez]

neo4j: 3.4.1
apoc: 3.4.0.1

Created the following trigger:

CALL apoc.trigger.add('set-updated','UNWIND apoc.trigger.propertiesByKey({assignedNodeProperties},"updated") as prop
WITH prop.node as n
SET n.updatedAt = apoc.date.currentTimestamp()', {phase:'after'});

Ran the below to test the above trigger: MATCH (n) WHERE n.foo="bar" SET n.updated="geez" RETURN n.updated,n.updatedAt

The above query keeps spinning in neo4j browser and nothing in debug.log.

Appreciate any help!

[alexiudice]

Try running just: MATCH (n) WHERE n.foo="bar" SET n.updated="geez" This causes a write transaction that should set off the trigger. If this hangs then there is another problem.

Once we've verified you can SET the updated property try running: MATCH (n) WHERE n.foo="bar" RETURN n.updated, n.updatedAt This gets the nodes that were affected by the trigger. They should have the updatedAt property.

You will not see the changes from the trigger in the same transaction that "fires" the trigger, thus you need that second statement to make sure the trigger worked. Let me know what happens!

[jexp]

@foobargeez did you check the suggestion?

[gvasend]

APOC 3.5.0.1 NEO4j 3.5.0

I am seeing a similar issue. My trigger started out more complex:

CALL apoc.trigger.add('get_text','UNWIND apoc.trigger.propertiesByKey({assignedNodeProperties},"doc_type") AS prop WITH prop.node as f, id(prop.node) as id MATCH (f:File) CREATE (a:Trigger {dag_id:"GetText", node_id:id})-[:PROCESSING]->(f) ',{phase:'after'});

Then I simplified to this for testing purposes:

CALL apoc.trigger.add('get_text','UNWIND apoc.trigger.propertiesByKey({assignedNodeProperties},"doc_type") AS prop WITH prop.node as f MATCH (f:File) SET f.test=1 ',{phase:'after'});

When I try to set the property programatically or via the browser the operation hangs. Other operations still work fine. I can create nodes, etc. When I check the node property, it has been set so the set command worked but hung before completing (i.e. still getting the hourglass in the browser) and the test property is not set.

[gvasend]

I also tried assignedLabels with the same result:

CALL apoc.trigger.add('get_text1','UNWIND apoc.trigger.nodesByLabel({assignedLabels},"Document") AS f WITH f SET f.test=1 ',{phase:'after'});

Node triggers seem to work fine as this one:

CALL apoc.trigger.add('cat_doc', 'UNWIND {createdNodes} AS n WITH n, id(n) as id MATCH (n:File) CREATE (a:Trigger {dag_id:"CategorizeDocument", node_id:id})-[:PROCESSING]->(n) ',{phase:'after'});

[jexp]

how do you trigger those?

please note that this does not work

WITH prop.node as f MATCH (f:File)

you need a where id(f) = prop.node

[gvasend]

I primarily create the trigger conditions from py2neo but I sometimes do it from the browser for testing purposes. I get the hanging problem in more than one situation. I can imagine that it would be easy to create triggers that would recursively create triggers but I have tried to be careful to avoid that. One case was that I was triggering on all new Nodes/Relationships and writing that data to Mongo. It would work fine until I tried to import a 2000+node/relationship graph and it would hang. The graph is imported fine but the execution hangs. When I turn those two triggers off it does not hang. My latest hanging problem involves multiple triggers on a specific relationship. One thing I have tried that often helps is to write the graph-changing part using apoc.periodic.submit. I am guessing that allows the conflicting activity to happen outside the thread that initiates the trigger.

Here are the two triggers that when enabled hangs the completion of the GraphML import:

CALL apoc.trigger.add('new_obj', 'UNWIND {createdNodes} AS n WITH n, localdatetime() as currentDateTime, id(n) as id, labels(n) as label_list CALL apoc.mongodb.insert("....","aif_document_db","change_log",[{type:"new",subtype:"node",node_id:id,config_key:"{config_key}", labels: label_list, createdAt: currentDateTime}]) RETURN 0 ',{phase:'after'});

CALL apoc.trigger.add('new_rel', 'UNWIND {createdRelationships} AS r WITH r MATCH (from)-[r]-(to) WITH r, id(r) as id, id(from) as from_n, id(to) as to_n, type(r) as rel_type, localdatetime() as currentDateTime CALL apoc.mongodb.insert("...","aif_document_db","change_log",[ {type:"new",subtype:"relationship", rel_id:id, config_key:"{config_key}", rel: rel_type, from:from_n, to:to_n, createdAt: currentDateTime}]) RETURN 0 ',{phase:'after'});

I am also guessing that the triggering condition blocks until the trigger has completed which I suppose could cause long delays on an import operation (would be nice if one could disable triggers for the import). However, I waited a long time (hours) and it never completed so I doubt it was just being slow.

[gvasend]

Follow-up

It looks like the submit jobs help but it just kicks the problem down the road a little further. Now the import submit jobs are hanging and the actions in the submit are not being executed. FYI I tried on both N4j 3.5.0 and 3.5.2

[mojo2go]

I experience the same. Running Neo4j Desktop with 3.5.3 on my MacBook Pro.

With a trigger defined that monitors a certain label, I never have a problem with the trigger being invoked for new nodes being created with the matching label...

But if I try to assign the label named by that trigger to a node that already exists, I get a spinner. And once that has happened I can't assign any label to any existing node. Restarting the server allows me to add labels not specified in my trigger, but then as soon as I do try to assign the trigger specified label to an existing node, I get the spinner again, and the inability to assign any label to any node, again.

Here's the example trigger I am working with:

CALL apoc.trigger.add('testLabelTrigger', " UNWIND apoc.trigger.nodesByLabel({assignedLabels}, 'A') AS node SET node.aLabel = true ", { phase: 'after' })

[jexp]

@gvasend yes it blocks the job, we could add an option for async execution.

You can suspend/resume triggers already, there are procedures for that. But even on a 2000 element insert it should be done quickly.

Perhaps it's related to handling of the mongodb connection. Something you could try is to insert all 2k documents in one go into mongo?

[jexp]

@mojo2go hmm good question, the trigger triggers new commits which calls new triggers so I guess you get into a loop.

You should try phase:'before' instead so it becomes part of your original commit.

[mojo2go]

@jexp That worked! Changing the selector from 'after' to 'before' does fix the hanging problem completely in every case I can think to test. And it doesn't cause problems with my program logic.

I am happy to assist in any further testing of this. I have big plans for triggers, like to enforce contract coherency where all member nodes of a contract trigger a rules check upon update, to make sure a person doesn't get removed from a contract without the contract first being re-labeled.

[marci543]

With @kerimovscreations we use APOC 4.0.0.2 with Neo4j Community Server 4.0.0. conf/neo4j.conf:

dbms.security.procedures.unrestricted=apoc.*
dbms.security.procedures.whitelist=apoc.*
apoc.trigger.enabled=true

We run the first example of the manual from browser. The modification query hangs indefinitely. Only stopping the query helps, but before that the trigger runs anyway, thus it produces the correct result.

Queries:

CREATE (d:Person {name:'Daniel'})
CREATE (l:Person {name:'Mary'})
CREATE (t:Person {name:'Tom'})
CREATE (j:Person {name:'John'})
CREATE (m:Person {name:'Michael'})
CREATE (a:Person {name:'Anne'})
CREATE (l)-[:DAUGHTER_OF]->(d)
CREATE (t)-[:SON_OF]->(d)
CREATE (t)-[:BROTHER]->(j)
CREATE (a)-[:WIFE_OF]->(d)
CREATE (d)-[:SON_OF]->(m)
CREATE (j)-[:SON_OF]->(d)

New parameter syntax is needed for Neo4j 4.0 ({assignedNodeProperties} → $assignedNodeProperties):

CALL apoc.trigger.add('setAllConnectedNodes','UNWIND apoc.trigger.propertiesByKey($assignedNodeProperties,"surname") as prop
WITH prop.node as n
MATCH(n)-[]-(a)
SET a.surname = n.surname', {phase:'after'});

The modification and the trigger succeeds, but never finishes:

MATCH (d:Person {name:'Daniel'})
SET d.surname = 'William'

[alexiudice]

@marci543 The trigger you proposed

CALL apoc.trigger.add('setAllConnectedNodes','UNWIND apoc.trigger.propertiesByKey($assignedNodeProperties,"surname") as prop
WITH prop.node as n
MATCH(n)-[]-(a)
SET a.surname = n.surname', {phase:'after'});

will definitely produce an infinite loop which is why the modification and trigger succeeds but never "finishes".

If you update a single node, the trigger fires and updates its surrounding nodes. Updating those surrounding nodes sets the trigger off for them. The trigger then updates the original node, resulting in an infinite loop.

I'm not sure that this is an issue with apoc, since the trigger is really acting how it should. I think the user may just need to be aware of how triggers work to avoid situations like this.

To avoid an infinite loop you could edit the trigger to be:

CALL apoc.trigger.add('setAllConnectedNodes','UNWIND apoc.trigger.propertiesByKey($assignedNodeProperties,"surname") as prop
WITH prop.node as n
MATCH(n)-[]-(a)
WHERE NOT EXISTS(a.surname)
SET a.surname = n.surname', {phase:'after'});

The WHERE NOT EXISTS(a.surname) ensures that it will not loop indefinitely. I tried this out and it solves the issue, the modification and triggers succeed and the query execution finishes.

[marci543]

@alexiudice Thank you for your help. It makes sense now. So the example in the documentation should only change the attribute if it doesn't exist or it is different.

CALL apoc.trigger.add('setAllConnectedNodes','UNWIND apoc.trigger.propertiesByKey($assignedNodeProperties,"surname") as prop
WITH prop.node as n
MATCH (n)-[]-(a)
WHERE NOT EXISTS(a.surname) OR a.surname <> n.surname
SET a.surname = n.surname', {phase:'after'});

[alexiudice]

@marci543 Happy to help!

And yes that documentation should probably be changed to avoid confusing people in the future. Feel free to either open a new issue to let others know it needs to be changed, or make a small PR editing it.

[conker84]

closed as it's solved