catalyst / moodle-tool_objectfs

Object file storage system for Moodle
https://moodle.org/plugins/tool_objectfs
88 stars 72 forks source link

Make the generate_status_report work using deltas to make it many orders of magnitude faster #572

Open brendanheywood opened 1 year ago

brendanheywood commented 1 year ago

generate_status_report can take hours on a large site and it runs 4 reports which are each expensive in different ways.

I'm fairly certain that all of these reports can be refactored one by one to to all run in a delta mode. Each of the reports is saved so we know the previous results from when it last ran. We should be able to run stats on only the files which have changed since the last run time and append them. Files which have been deleted we can keep in the tool_objectfs_objects marked as deleted so we can properly track removals, and we can only clean them out of that table via delete_orphaned_object_metadata when for records which we know have been processed by generate_status_report.

In a round about way this should also fix #556 / #411

anton-catalyst-eu commented 10 months ago

Hi Brendan :) How are you? Do you reckon this one could be progressed sometime soon or your focus needs to be elsewhere? Not chasing, just checking. We could increase temp table memory allocation for affected clients but your solution sounds way better

brendanheywood commented 9 months ago

@JoshAudette had some success here #596 on this front