Make sure async propagation does not block other requests

[smadbe]

Motivations

We moved some propagation to async because there were too long. But of course, we have to make sure these propagation may not block other sync requests (which would wait for a 15min propagation to complete).

As propagation works by iteration, it would convenient if:

• propagation would commit from time to time: maybe commit at every iteration would make it slow... a rule may be "every iteration if >0.1sec has passed since the last one
• start the lock related to the iteration with low priority: the locking should be in low priority, not the queries in the lock

Note: that would also solve the problem of a propagation which would run for >15min and so which would not progress when killed after 15min by lambda.

Subtasks

• check what would be relevant and possible

[GeoffreyHuck]

As discussed the other day, the propagation should schedule the run of another propagation if it's getting killed at 15min.

[GeoffreyHuck]

Notes on the problem

Two processes that can cause the problem

1. A service is called that schedules a big propagation
2. Another service is called that wants to run a normally small propagation, but since the big propagation is in the propagations tables, it ends up calling a big propagation, and timeout

or

1. A service schedule a big async results propagation (this one sets a lock)
2. The propagation is run is async, and the lock is taken
3. Another service wants to run the results propagation but cannot get the lock, and timeout

Solutions

We can either:

• Make all propagations async, or
• Allow to execute a propagation on a subset, but in this case we would also have to handle the lock for the results propagation

We already talked about it on the phone and we decided to make all propagation async.

[GeoffreyHuck]

Tasks

• [x] make all the propagation async, except the groups ancestors that is fast and which have services who needs the result right away
• [ ] commit at the end of the step if >0.1sec has passed since the last one
• [ ] start the lock related to the iteration with low priority: the locking should be in low priority, not the queries in the lock (need clarification)
• [ ] re-schedule a propagation if it's getting killed at 15min.