search

influxdata / influxdb

Scalable datastore for metrics, events, and real-time analytics
https://influxdata.com
Apache License 2.0
29.08k stars 3.56k forks source link

Persist should run in parallel #24658

Open pauldix opened 9 months ago

pauldix commented 9 months ago

Persisting snapshots and their parquet files is not done in parallel. This process is kicked off here: https://github.com/influxdata/influxdb/blob/35e29d14087b26a8433c8c24240cb5c2cb494d18/influxdb3_write/src/write_buffer/queryable_buffer.rs#L135

This could be persisting potentially many tables at the same time. How many to run at a time should be something that can be configured.

hiltontj commented 3 weeks ago

I updated the issue description to reflect the change from segments to snapshots.