flux-framework / flux-accounting

bank/accounting interface for the Flux resource manager
https://flux-framework.readthedocs.io/projects/flux-accounting/en/latest/index.html
GNU Lesser General Public License v3.0
3 stars 10 forks source link

job-archive interface: wrap job usage updates into a single SQL transaction #452

Closed cmoussa1 closed 6 months ago

cmoussa1 commented 6 months ago

Problem

When flux-accounting needs to do a job usage update for all of the associations in the association_table, it needs to make a number of updates to a number of tables in the DB. If there are a lot of associations, this can slow down the performance of a number of other, much simpler commands, such as flux account view-user. This is because the job usage updates makes a separate transaction for each UPDATE statement performed, forcing other reads from the DB to wait until the updates are completed.


This PR drops the separate commit() calls from the various helper functions that execute UPDATE statements in job_archive_interface.py and instead wraps the entire set of database updates into a single SQLite transaction, which will only be committed to the DB after all of the updates are done. This should hopefully begin to relieve some of the headache mentioned in #451.

cmoussa1 commented 6 months ago

Thanks for reviewing @ryanday36 - the sharness test added in t1011-job-archive-interface.t is in fact checking job usage values for correctness. I was also able to confirm that wrapping up the updates into a single transaction is OK for calculating both user and bank usage values - subsequent updates made into a single transaction should be readable by each other. 👍 I'll set MWP here