This PR massively improves report generation performance overall (up to 10x faster compared to current version 5.3.0, and up to 4x faster compared to master branch right now). It also adds the ability to chunk generation into smaller jobs for super large sites with thousands of entries.
For example, report generation on 5.3.0 (current version) for even 1000 pages is 60+ seconds with a request timeout...
But with this PR, 1000 pages takes me ~7 seconds, which is about 10x faster 🏎️...
But where this PR really shines is when generating reports for massive sites with thousands of entries, which you can't even do on 5.3.0 (as shown above in our 1000 page example).
On master (untagged), we can at least generate a report for 5000 pages, but it takes me about ~52 seconds...
With this PR though, the same 5000 pages takes about ~29 seconds, which is nearly 2x faster 🏎️
And if I enable async queue driver w/ 5 workers (chunked in jobs of 1000 pages each), the report generation endpoint takes about ~13 seconds, which is about 4x faster compared to master, and not even possible on 5.3.0 🏎️
Configuration
I have made the chunk size configurable in config/statamic/seo-pro.php under reports.queue_chunk_size, so that users can balance chunk size vs. worker count how they like.
Note on CPU
Of course your mileage will vary. It's pretty CPU intensive when generating a report for this many entries. The above numbers are just locally on my M1 Macbook.
This PR massively improves report generation performance overall (up to 10x faster compared to current version 5.3.0, and up to 4x faster compared to master branch right now). It also adds the ability to chunk generation into smaller jobs for super large sites with thousands of entries.
For example, report generation on 5.3.0 (current version) for even 1000 pages is 60+ seconds with a request timeout...
But with this PR, 1000 pages takes me ~7 seconds, which is about 10x faster 🏎️...
But where this PR really shines is when generating reports for massive sites with thousands of entries, which you can't even do on 5.3.0 (as shown above in our 1000 page example).
On master (untagged), we can at least generate a report for 5000 pages, but it takes me about ~52 seconds...
With this PR though, the same 5000 pages takes about ~29 seconds, which is nearly 2x faster 🏎️
And if I enable async queue driver w/ 5 workers (chunked in jobs of 1000 pages each), the report generation endpoint takes about ~13 seconds, which is about 4x faster compared to master, and not even possible on 5.3.0 🏎️
Configuration
I have made the chunk size configurable in
config/statamic/seo-pro.phpunderreports.queue_chunk_size, so that users can balance chunk size vs. worker count how they like.Note on CPU
Of course your mileage will vary. It's pretty CPU intensive when generating a report for this many entries. The above numbers are just locally on my M1 Macbook.