metrico / qryn

Polyglot Observability Stack. Lightweight & Drop-in compatible with Loki, Prometheus, Tempo, Pyroscope, Opentelemetry, Datadog & more! WASM powered ⭐️ Star to Support

https://qryn.dev

GNU Affero General Public License v3.0

1.05k stars 63 forks source link

BULK_MAXAGE / BULK_MAXSIZE - insert limits!? #497

Closed bzed closed 1 week ago

bzed commented 2 months ago

Hi,

according to the documentation there is are BULK_MAXSIZE BULK_MAXCACHE settings, but I can't find them in the code.

Is there any undocumented way I didn't find to limit the size of the inserts?

Thanks,

Bernd

akvlad commented 2 months ago

Now it sends as many as you can ingest during 100ms. Also we have a limit for ingestion of 50MB/sec before a request proceeds to be parsed.

akvlad commented 2 months ago

@bzed can you suggest the best solution you suggest about the BULK_MAXSIZE configuration? What should happen if the request overflows the configured value?

akvlad commented 2 weeks ago

Bulk size configuration

since 3.2.23

Two env vars were introduced:

BULK_MAX_SIZE_BYTES - max size of the bulk in bytes (default - infinite)
BULK_MAX_AGE_MS - max age of the bulk in milliseconds (default - 100), can't be less than 100

lmangani commented 1 week ago

@bzed this was implemented. Let us know if you have any feedback and/or comments.

Cloding as completed!