search

parseablehq / parseable

Open Source ElasticSearch Alternative. Parseable helps you search and get insights from your logs in the most simple way possible.
https://parseable.com
GNU Affero General Public License v3.0
1.87k stars 105 forks source link

Optimize Ingestion #948

Open parmesant opened 2 days ago

parmesant commented 2 days ago

During ingestion we accept a batch of events but still ingest them sequentially. Try to optimize that flow to improve ingestion.

parmesant commented 2 days ago

One naive way could be to spawn tokio tasks for each record but will the task overhead be greater than the resulting throughput improvement?