search

janmg / logstash-input-azure_blob_storage

This is a plugin for Logstash to fetch files from Azure Storage Accounts
Other
29 stars 8 forks source link

i need create a prefix with the format yyyy/MM/dd #45

Open ghost opened 11 months ago

ghost commented 11 months ago

I need to create a prefix that allows me to have the year, month and day as a variable so that my logstash process does not analyze the millions of records I have in my storage but only uses the folder of the current day.

i tried with this but doesnt work

        prefix => "%{YEAR}/%{MONTHNUM}/%{MONTHDAY}/"
janmg commented 10 months ago

Apart from prefix which is the starting directory of what you want to process there is also path_filters which does an fnmatch using a glob. I don't know the details, but I image something like /202[4-9]/ will look at new files and stores the size in the registry just in case the file grows. https://www.rubydoc.info/stdlib/core/File.fnmatch

Because logstash is giving me problems with ruby dependancies, I have started rewriting the plugin in golang https://github.com/janmg/blob-to-queue/ There I'm pondering if keeping a list of all the files and their sizes in a registry, just like done in the original plugin https://github.com/Azure/azure-diagnostics-tools/tree/master/Logstash/logstash-input-azureblob Perhaps using a directory counter based on year month day hour makes more sense also for nsgflowlogs