Data exports: Fine-tuning historical export batch sizes

[macobo]

Is your feature request related to a problem?

Exports can be slow. One area to fine-tune is how many events are exported in a given "batch" which is currently 500.

The ideal number depends on a few factors:

• ClickHouse load - we don't want to overload our instances
• App we're exporting into
  • BigQuery accepts up to 10k items in a batch: https://docs-snaplogic.atlassian.net/wiki/spaces/SD/pages/80904212/Google+BigQuery+Bulk+Load+Streaming
  • S3 limits us to 160GB per export
  • Redshift app uses SQL to insert, the max limit being 16MB per insert query
  • For posthog migrator today, 500 events per batch might be too much already

Describe the solution you'd like

Allow specifying the batch size default in the plugin source code.

Describe alternatives you've considered

• Making this a configuration option via pluginConfig: Would still need good defaults to have a realistic effect, validations are tricky
• Making this an instance config: there doesn't seem to be a one-size-fits-all solution here

Additional context

cc @yakkomajuri who worked on exports previously for thoughts

Thank you for your feature request – we love each and every one!

[yakkomajuri]

Update the issue name just for clarity.

I think allowing devs to specify batch sizes is valid. Two considerations though:

1. We should specify a clear max and min
2. It's important to note that the limiting factor here might become the plugin execution time. Some export plugins already take 10-20s to run (q: for real or are metrics skewed because of asynchronous stuff?) so increasing the batch sizes a lot might be fine from the perspective of the third party service but not from the plugin execution limit side

[posthog-bot]

This issue hasn't seen activity in two years! If you want to keep it open, post a comment or remove the stale label – otherwise this will be closed in two weeks.