The tower::limit::ConcurrencyLimit middleware does not work properly when used with services that return response with streaming body. The middleware considers concurrency only from call to service to response future resolution.
But response with streaming body can't be considered finished until body has been consumed.
Solution
Add concurrency_limit module with ConcurrencyLimit middleware implementation that holds on the semaphore permit until response and its body has been consumed.
It uses original middleware from tower crate and tower-http::metrics::InFlightRequests middleware as inspiration.
Questions/Considerations
As far as module organization goes, it would be nicer to add this as tower-http::limit::concurrency, similar to the module layout in tower crate. But the tower-http::limit module is fully dedicated to RequestBody middleware now and moving it into sub-module would probably be breaking change.
I have considered leaving RequestBody middleware where it is and just add tower-http::limit::concurrency sub-module, but it felt confusing. Adding another top level module felt as cleanest alternative I had.
Motivation
The tower::limit::ConcurrencyLimit middleware does not work properly when used with services that return response with streaming body. The middleware considers concurrency only from call to service to response future resolution.
But response with streaming body can't be considered finished until body has been consumed.
Solution
Add concurrency_limit module with ConcurrencyLimit middleware implementation that holds on the semaphore permit until response and its body has been consumed.
It uses original middleware from
towercrate andtower-http::metrics::InFlightRequestsmiddleware as inspiration.Questions/Considerations
As far as module organization goes, it would be nicer to add this as
tower-http::limit::concurrency, similar to the module layout intowercrate. But thetower-http::limitmodule is fully dedicated toRequestBodymiddleware now and moving it into sub-module would probably be breaking change.I have considered leaving
RequestBodymiddleware where it is and just addtower-http::limit::concurrencysub-module, but it felt confusing. Adding another top level module felt as cleanest alternative I had.