search

janmg / logstash-input-azure_blob_storage

This is a plugin for Logstash to fetch files from Azure Storage Accounts
Other
29 stars 8 forks source link

Azure container with access policy #14

Closed EH1980 closed 3 years ago

EH1980 commented 3 years ago

Hi, I tried to read files from my azure container but I get an error:

"[2021-02-16T07:59:21,432][INFO ][logstash.inputs.azureblobstorage][main][8fc49f3655f2c70ab7e37a2d0379d5e54d919ee1c50b0c7dc64721f8b31a3049] resuming from remote registry data/registry.dat [2021-02-16T07:59:24,069][ERROR][logstash.inputs.azureblobstorage][main][29a4c675ca27f6ed56d4ce2e0aeec3534fcfad468f8f5d7befb17d452685398a] caught: BlobNotFound (404): The specified blob does not exist. RequestId:c40fcf4a-701e-000b-4739-04f6ec000000 " when I put another container name everything work fine. the different between them is that in the container that I got the error I have access policy. so I guess that this is the problem.

How should I access to container with access policy? (the container type is blob) thanks.

janmg commented 3 years ago

Hi ... The default behavior is to write a registry to the container in the path set by registry_path (data/registry.dat). You can change that path in the container or maybe better for you, store the registry on the server that is running logstash in the directory set by registry_local_path. This probably faster, but you have to resume from the same server, so for migrations to another server you have to copy the resume file.

If you don't care about resuming you can set the registry_create_policy to either start_over to start processing all the files that are in the storage account or start_fresh that will start with only files that were created or have grown since the start of the pipeline.

EH1980 commented 3 years ago

Hi, thanks for the quick response. I wrote this: azure_blob_storage { storageaccount => {account_name} access_key => {account_key} container => {continer_name} addfilename => true registry_create_policy => "start_over" registry_local_path=> "/usr/share/logstash/plugin" } and know it doesn't throw any error but it still doesn't work. It doesn't create any registry.dat file in my local folder and don't load data as well. I think that logstash doesn't have any permissions to this container without sas_token

janmg commented 3 years ago

ah, make sure the /usr/share/logstash/plugin directory is owned by logstash and is writable. But you may be right that your problem is that the listing the container doesn't work without SAS token. You can configure the plugin to use sas_token, the storageaccount / access_key is only one of three access methods.

  1. storageaccount / sas_token
  2. connection_string
  3. storageaccount / access_key

To see what the plugin is doing at the startup, you can set debug_until => 5000 so that the plugin shows a bit more information for the first 5000 events (maybe already in one file). This will show you the listing of the files. Another option is to set debug_timer => true to get timing information on sleeping and listing.

EH1980 commented 3 years ago

It works now thanks "storageaccount / sas_token" thanks!