On Airmail, there is a ratio of approximately 3 persist request attempts for 1 ingest request received. However, we validate docs before actually enforcing rate limits per shard. Consequently, we waste a lot of CPU validating docs not persisted to disk. We should inverse the order with which the current logic is performed and apply rate limits before validation.
On Airmail, there is a ratio of approximately 3
persistrequest attempts for 1ingestrequest received. However, we validate docs before actually enforcing rate limits per shard. Consequently, we waste a lot of CPU validating docs not persisted to disk. We should inverse the order with which the current logic is performed and apply rate limits before validation.