HumanSignal / Adala

Adala: Autonomous DAta (Labeling) Agent framework
https://humansignal.github.io/Adala/
Apache License 2.0
979 stars 77 forks source link

fix: DIA-1584: Use send_and_wait + batches for output topic #245

Closed hakan458 closed 2 weeks ago

hakan458 commented 3 weeks ago

Back to using send_and_wait, which while using before we never lost a message in this stage.

However, we switched to send for speed purposes. To mitigate this, we are now sending batches of records to the output topic, instead of calling send_and_wait once per record. Even if we have the small delay while awaiting the result, that will now happen once per batch (e.g. 50 records) instead of once per record, greatly reducing the time it takes.

codecov-commenter commented 3 weeks ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 66.29%. Comparing base (c6da54b) to head (2e5b589).

Additional details and impacted files ```diff @@ Coverage Diff @@ ## master #245 +/- ## ========================================== - Coverage 66.33% 66.29% -0.05% ========================================== Files 47 47 Lines 2439 2436 -3 ========================================== - Hits 1618 1615 -3 Misses 821 821 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.