Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
If you are interested in working on this issue or have submitted a pull request, please leave a comment
Problem
I'm testing Vector's azure_blob to upload logs from Kafka. I realized ERROR logs like below, and when that happens, I actually miss the logs from Azure Blob. I've set acknowledgement as enabled, but Vector just keeps on uploading the new logs, so in the end, the log /container/topics/audit-logs-v2/year=2023/month=12/day=31/vector-1703996260-6d0af193-9f7d-4671-91e6-be18466bd55a.log.gz was not uploaded, and we lost the log.
I think this is due to Vector mistreats the event and acknowledgement is processed even in this ERROR.
Additionally, Vector ensures that the batch notifier for an event is always updated, whether or not the event made it to a sink. This ensures that if an event is intentionally dropped (for example, by using a [filter][filter] transform) or even unintentionally dropped (maybe Vector had a bug, uh oh!), we still update the batch notifier to indicate the processing status of the event.`
A note for the community
Problem
I'm testing Vector's
azure_blob
to upload logs from Kafka. I realized ERROR logs like below, and when that happens, I actually miss the logs from Azure Blob. I've setacknowledgement
as enabled, but Vector just keeps on uploading the new logs, so in the end, the log/container/topics/audit-logs-v2/year=2023/month=12/day=31/vector-1703996260-6d0af193-9f7d-4671-91e6-be18466bd55a.log.gz
was not uploaded, and we lost the log.I think this is due to Vector mistreats the event and acknowledgement is processed even in this ERROR.
https://vector.dev/docs/about/under-the-hood/architecture/end-to-end-acknowledgements/
Configuration
Version
0.34.1
Debug Output
No response
Example Data
No response
Additional Context
No response
References
No response