Closed negezor closed 2 months ago
Oh. Really? I've been using hint::black_box
for so long (probably in nightly)
Thank you, first and foremost, for such a rich ecosystem of modules that I use in my projects.
Currently, I am using sea-streamer with Kafka backend with a connection to Redpanda for:
AutoOffsetReset::Earliest
)AutoOffsetReset::Earliest
)async-graphql
(without transaction, AutoOffsetReset::Latest
)The inconvenience arises when it is necessary not only to accumulate messages, but also to add a minimum waiting time for debounce
and a maximum for throttle
(for example, like this in this demo with maxWait
), deserialize messages, and then process events and commit the identifier to the consumer. At the same time, it is necessary to explicitly handle each type of backend for the identifier commit, which already sounds like the implementation of a first-class abstraction.
I wrote a small helper, the implementation idea of which I borrowed a bit from futures-batch and tokio::time::timeout. It partially eliminates the inconveniences mentioned above, but I am not entirely confident in its reliability. Because committing the identifier to the consumer should only occur after successful execution, otherwise retries should occur with backoff (for example module) and no new events should be received until then.
Maybe it's not really the module's area of responsibility, but I think it would be handy. Otherwise, I am completely fine with the implementation of the module in terms of usability.
Thank you for your input. I think may be we can have a BufferedProducer<P>
that wraps a Producer
and have some buffering parameters: 1. max batch size 2. min send interval 3. max send interval
Speaking of which, I actually have implemented a similar thing somewhere in one of my projects.
The difficulty, though, as you already experienced, is to handle max send interval, because it means we'd have to set up a callback to wake us up after some time.
Btw, I think Kafka already has something similar https://stackoverflow.com/questions/51878751/send-bulk-of-messages-kafka-producer
So far, it is the batch processing of messages in the consumer that is causing the problem. I currently rely on batching from librdkafka in producer, but it won't work in Redis.
Thank you everyone for the contribution! This feature is now available in the latest release. Now is a good time to upgrade! Your participation is what makes us unique; your adoption is what drives us forward. You can support SeaQL 🌊 by starring our repos, sharing our libraries and becoming a sponsor ⭐.
PR Info
Just fixes the clippy warrnings. I would like to point out that the declared MSVR does not correspond to the functions used, namely:
std::hint::black_box
(min MSVR1.66
)std::thread::JoinHandle::is_finished
(min MSVR1.61
)