Batch Exports Stability

[EDsCODE]

This ticket will track the assortment of changes that are in progress for making batch exports more resilient against failures

Stability

• [ ] Autoscaling based on certain metrics from dashboard (from incident analysis) Example: scale on task slots
• [ ] Query cancellation — Make sure that retries cancel any ongoing queries https://github.com/PostHog/posthog/issues/26317
• [ ] Single producer multiple consumer Batch Exports. Move all exports to async https://github.com/PostHog/posthog/issues/26314

Clickhouse

• [ ] Create new CH users that are dedicated for batch exports use

Documentation

• [ ] Update documentation
• [ ] Consistency across the docs on what’s allowed
• [ ] Whitelisting IPs

Features in consideration

• [ ] File size limits https://github.com/PostHog/posthog/issues/26315
• [ ] Connection test
• [ ] Partitioning exports based on event names https://github.com/PostHog/posthog/issues/26316