janmg / logstash-input-azure_blob_storage

This is a plugin for Logstash to fetch files from Azure Storage Accounts
Other
29 stars 8 forks source link

Fix file offset tracking #30

Closed nttoshev closed 2 years ago

nttoshev commented 2 years ago

This commit fixes an issue reported by @Sera4000 regarding duplicate logs. The problem was with the way the currently read offset was tracked. In the case when the blob had grown since the last time we read it, it happened that the size was set to 0 and not updated after reading, leading to the registry being saved with offset = 0 for the file. On the next iteration, the file would be read in full.

This PR makes it so that in each case (full read / partial read / empty file / error) we track the size of the chunk read in a variable delta_size and add that to the current file[:offset] to get the new offset.