France-ioi / AlgoreaBackend

Backend for the new Algorea platform
MIT License
1 stars 2 forks source link

Propagations step by step after transaction #1058

Closed GeoffreyHuck closed 3 months ago

GeoffreyHuck commented 3 months ago

For each propagations:

Instead of calling the propagation right away, we schedule it. Scheduling a propagation marks a flag, and then execute it AFTER the current transaction. This way, we can split propagations into several mini-transactions, because they are not themselves in a propagation anymore.

Note: The propagations are still done in sync, after the end of the transaction in which they are scheduled.

There are two exceptions, details below.

On the slowness of itemCreate/itemUpdate:

The process was slow even with async propagation because there was also an items ancestors propagation that was done synchronously. For those 2 services, the items ancestors propagation have been added in the async propagation, making the call to the service faster.

Step by step transactions:

For now, only the permissions propagation have been split into smaller transactions. If need to, the other propagations are ready to be split also. We might want to do this only if they become too slow.

Deleted tests:

Two tests that didn't pass anymore were deleted: TestGroupGroupStore_CreateRelation and TestGroupGroupStore_CreateRelationsWithoutChecking. They didn't pass because we don't execute the same queries anymore. I'm not sure they have any value because they only check the SQL that is executed, and not the result. In my opinion they're useless.

Two exceptions in the process

We have two exception were we need the groups ancestors propagation to be called inside a transaction, because their result is used for other queries inside the same transaction.

Those 2 exceptions are:

Depending on how fast this propagation is, we might want to rewrite those processes so they don't rely of doing this intermediate propagation.

How to review

It's easier to review all commits at once because many changes are repetitive. Further information were added in the comments of the commits.

codecov[bot] commented 3 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 100.00%. Comparing base (ad0b4dc) to head (a8656c3).

Additional details and impacted files ```diff @@ Coverage Diff @@ ## fixNilPointerschedulePropagation #1058 +/- ## ================================================================== Coverage 100.00% 100.00% ================================================================== Files 236 236 Lines 14226 14225 -1 ================================================================== - Hits 14226 14225 -1 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.