feat: change the logic to handle dynamic batching

[JoanFM]

This PR fixes the behavior of dynamic batching.

• It makes sure that no batch larger than preferred batch size gets into the Executor.
• It allows the requests to be returned to the client as they are processed without waiting for other requests to be ready, this should improve the average latency of the system.

[codecov[bot]]

Codecov Report

Attention: 4 lines in your changes are missing coverage. Please review.

Comparison is base (810d8d4) 76.75% compared to head (91a8a78) 76.87%. Report is 1 commits behind head on master.

Additional details and impacted files

```diff @@ Coverage Diff @@ ## master #6066 +/- ## ========================================== + Coverage 76.75% 76.87% +0.12% ========================================== Files 145 145 Lines 13925 13986 +61 ========================================== + Hits 10688 10752 +64 + Misses 3237 3234 -3 ``` | [Flag](https://app.codecov.io/gh/jina-ai/jina/pull/6066/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=jina-ai) | Coverage Δ | | |---|---|---| | [jina](https://app.codecov.io/gh/jina-ai/jina/pull/6066/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=jina-ai) | `76.87% <95.83%> (+0.12%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=jina-ai#carryforward-flags-in-the-pull-request-comment) to find out more. | [Files](https://app.codecov.io/gh/jina-ai/jina/pull/6066?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=jina-ai) | Coverage Δ | | |---|---|---| | [jina/serve/runtimes/worker/batch\_queue.py](https://app.codecov.io/gh/jina-ai/jina/pull/6066?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=jina-ai#diff-amluYS9zZXJ2ZS9ydW50aW1lcy93b3JrZXIvYmF0Y2hfcXVldWUucHk=) | `96.59% <97.61%> (+0.99%)` | :arrow_up: | | [jina/serve/runtimes/worker/request\_handling.py](https://app.codecov.io/gh/jina-ai/jina/pull/6066?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=jina-ai#diff-amluYS9zZXJ2ZS9ydW50aW1lcy93b3JrZXIvcmVxdWVzdF9oYW5kbGluZy5weQ==) | `81.36% <83.33%> (+0.17%)` | :arrow_up: | ... and [5 files with indirect coverage changes](https://app.codecov.io/gh/jina-ai/jina/pull/6066/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=jina-ai)

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[JohannesMessner]

btw i think it is fair to call this a feature or at least a performance improvement, the old implementation was correct (we never made any guarantees that the actual batch size would not be bigger than the preferred one), just not optimized for this scenario.