Example local results (transaction errors are to ensure I can retest quickly - will omit in other results):
Pre-patch
❯ ctrl php admin/tool/excimer/dedupe-test.php
Number of duplicates found: 139058
Number of duplicates to remove: 115942
Number of duplicates remaining: 0
Execution time: 1.8240051269531 seconds!!! Database transaction error !!!
!! Transactions already disposed
Error code: dmltransactionexception !!
!! Stack trace: * line 103 of /lib/dml/moodle_transaction.php: dml_transaction_exception thrown
* line 121 of /admin/tool/excimer/dedupe-test.php: call to moodle_transaction->rollback()
!!
After precision reorder:
❯ ctrl php admin/tool/excimer/dedupe-test.php
Number of duplicates found: 139058
Number of duplicates to remove: 115942
Number of duplicates remaining: 0
Execution time: 1.4179699420929 seconds
After SQL + PHP change to pure SQL + optimisations
❯ ctrl php admin/tool/excimer/dedupe-test.php
Execution time: 0.41718292236328 seconds
Number of records in tool_excimer_page_groups: 36146
The numbers might be skewed slightly based on what my machine was doing at the time, but is a rough estimate. I also generated more duplicates at some point to make it take longer, but it wasn't that significant.
Resolves #352 (would be good to have confirmation with larger site)
Script used to generate some local test data - which isn't as great as live data but better than no data!
Example local results (transaction errors are to ensure I can retest quickly - will omit in other results):
Pre-patch
After precision reorder:
After SQL + PHP change to pure SQL + optimisations
The numbers might be skewed slightly based on what my machine was doing at the time, but is a rough estimate. I also generated more duplicates at some point to make it take longer, but it wasn't that significant.