oxidecomputer / crucible

A storage service.
Mozilla Public License 2.0
166 stars 17 forks source link

Adjust backpressure tuning #1440

Closed mkeeter closed 3 weeks ago

mkeeter commented 1 month ago

Testing with crutest bufferbloat reveals that we can get multi-second delays if we send a single read after filling up the write queues.

This PR turns up backpressure gain to make steady-state queue lengths less egregiously long.

I've done some casual testing, and now see turnaround times in the 10-100s of milliseconds. @faithanalog it would be great to get an iodriver run on this PR as well.

mkeeter commented 1 month ago

Here's main running a 30-seconds crutest rand-write with 4 KiB blocks. Notice that it takes an additional 30 seconds to actually finish the work; if someone sent a read at that point, it would be delayed by up to 30 seconds.

Screenshot 2024-08-30 at 11 47 42 AM

Here's this branch, which finishes the jobs at a reasonable speed:

Screenshot 2024-08-30 at 11 48 05 AM

mkeeter commented 3 weeks ago

Closing in favor of a larger stack of small changes