Open brendanheywood opened 1 year ago
Hi Brendan :) How are you? Do you reckon this one could be progressed sometime soon or your focus needs to be elsewhere? Not chasing, just checking. We could increase temp table memory allocation for affected clients but your solution sounds way better
@JoshAudette had some success here #596 on this front
generate_status_report can take hours on a large site and it runs 4 reports which are each expensive in different ways.
I'm fairly certain that all of these reports can be refactored one by one to to all run in a delta mode. Each of the reports is saved so we know the previous results from when it last ran. We should be able to run stats on only the files which have changed since the last run time and append them. Files which have been deleted we can keep in the tool_objectfs_objects marked as deleted so we can properly track removals, and we can only clean them out of that table via delete_orphaned_object_metadata when for records which we know have been processed by generate_status_report.
In a round about way this should also fix #556 / #411