[discovery] Backpressure

[arielshaqed]

What

Discover how lakeFS can apply backpressure on clients during load.

Requirements

• Observability: Do not hide load or backpressure from lakeFS server dependencies (DynamoDB, S3, network, ...). Otherwise we just lost an important vector for DevOps: we will not know that there is an issue.
• Improvement: Backpressure should end up improving latency, and/or throughput, and/or bandwidth, at least for clients that succeed. In particular it should improve probability of success and/or latency of intensive Spark jobs.
• Clients: Ideally client involvement will not be required or can be automated. For instance AWS "Boto" clients can handle some existing backpressure signals, can we use those? What about the lakeFS API?

[github-actions[bot]]

This issue is now marked as stale after 90 days of inactivity, and will be closed soon. To keep it, mark it with the "no stale" label.