Sending Data from azure blob storage to ES using logstash ?

[Private-SO]

• Version: Logstash 2.4.0

• Operating System: WIN 10

• Config File (if you have sensitive info, please remove it):

  input
  {
  azureblob
  {
      storage_account_name => "mystorageaccount"
      storage_access_key => "VGhpcyBpcyBhIGZha2Uga2V5Lg=="
      container => "mycontainer"
  }
  }
  output {
  elasticsearch { hosts => ["localhost:9200"] 
  }
  }

• Sample Data: [2017-04-26 20:39:54,633][TRACE][index.search.slowlog.query] [data-0] [content-apr-2017][0] took[82.1micros], took_millis[0], types[publishedarticle], stats[], search_type[QUERY_THEN_FETCH], total_shards[5], source[{"query":{"bool":{"must":[{"terms":{"List.Id":[316271282]}}]}}}], extra_source[], [2017-04-26 20:39:54,633][TRACE][index.search.slowlog.query] [data-0] [content-apr-2017][0] took[82.1micros], took_millis[0], types[publishedarticle], stats[], search_type[QUERY_THEN_FETCH], total_shards[5], source[{"query":{"bool":{"must":[{"terms":{"List.Id":[316271292]}}]}}}], extra_source[], [2017-04-26 20:39:54,633][TRACE][index.search.slowlog.query] [data-0] [content-apr-2017][0] took[82.1micros], took_millis[0], types[publishedarticle], stats[], search_type[QUERY_THEN_FETCH], total_shards[5], source[{"query":{"bool":{"must":[{"terms":{"List.Id":[316271382]}}]}}}], extra_source[],

• Steps to Reproduce: 1) Create a storage account in azure and copied raw data .txt file to a container into that account. 2) With logstash azureblob plugin sending it to local ES cluster using above logstash config file.

I am getting error like this in logstash

Caught exception while locking {:exception=>#<Azure::Core::Http::HTTPError:15752 @status_code: 400, @http_response: #<Azure::Core::Http::HttpResponse:0x70b417e1 @http_response=#<Faraday::Response:0x5dbac6c4 @on_complete_callbacks=[], @env=#<Faraday::Env @method=:put @body="\xEF\xBB\xBF<?xml version=\"1.0\" encoding=\"utf-8\"?><Error><Code>BlobTypeNotSupported</Code><Message>Page blob is not supported for this account type.\nRequestId:a1be676a-0001-0050-3fb9-002df1000000\nTime:2017-07-19T18:06:18.1830830Z</Message></Error>" @url=#<URI::HTTPS:0x37eca579 URL:https://azureexplorerblob.blob.core.windows.net/eslogs/picaso.txt.lock> @request=#<Faraday::RequestOptions open_timeout=60> @request_headers={"User-Agent"=>"Azure-Core/0.1.7", "x-ms-date"=>"Wed, 19 Jul 2017 18:06:17 GMT", "x-ms-version"=>"2014-02-14", "DataServiceVersion"=>"1.0;NetFx", "MaxDataServiceVersion"=>"2.0;NetFx", "Content-Type"=>"application/atom+xml; charset=utf-8", "x-ms-blob-type"=>"PageBlob", "Content-Length"=>"0", "x-ms-blob-content-length"=>"512", "x-ms-sequence-number"=>"0", "Authorization"=>"SharedKey azureexplorerblob:FXkteDw28TnQ7LuQRdRy+DmmLxzQZtHrce9E2lthLoI="} @ssl=#<Faraday::SSLOptions verify=true> @response=#<Faraday::Response:0x5dbac6c4 ...> @response_headers={"content-length"=>"238", "content-type"=>"application/xml", "server"=>"Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0", "x-ms-request-id"=>"a1be676a-0001-0050-3fb9-002df1000000", "x-ms-version"=>"2014-02-14", "date"=>"Wed, 19 Jul 2017 18:06:17 GMT", "connection"=>"close"} @status=400>>, @uri=#<URI::HTTPS:0x37eca579 URL:https://azureexplorerblob.blob.core.windows.net/eslogs/picaso.txt.lock>>, @uri: #<URI::HTTPS:0x37eca579 URL:https://azureexplorerblob.blob.core.windows.net/eslogs/picaso.txt.lock>, @description: "Page blob is not supported for this account type.\nRequestId:a1be676a-0001-0050-3fb9-002df1000000\nTime:2017-07-19T18:06:18.1830830Z", @type: "BlobTypeNotSupported">, :level=>:error}

But when i changed the storage account type from blob to general storage when i executed the first time , i can see the output send to ES . When i did the same again it is not sending any output to ES and also didn't showing any error in logstash too(i.e.Even i kept stdout { codec => rubydebug } in output ).

How can i resolve this and send data to ES ?

[prov4]

@Yaswanth-C have you been able to resolve this one? I'm looking into similar thing, blob storage to logstash or data lake to log stash.

[Private-SO]

Sorry i didn't resolved this one yet ,rather i am sending data directly from ES snapshot to Azure blob storage. BTW if you got this one resolved please share your info .

[prov4]

Looking into it right now, will do if/when I get to a solution or a workaround.

On Mon, Apr 9, 2018 at 10:51 AM, Yaswanth-C notifications@github.com wrote:

Sorry i didn't resolved this one rather we are sending data directly from ES snapshot to Azure blob storage. BTW if you got this one resolved please share your info .

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/elastic/logstash/issues/7748#issuecomment-379681071, or mute the thread https://github.com/notifications/unsubscribe-auth/AAmsq1Z5rrMuPVyxY2WIbYnVDrh6Np6yks5tmyEVgaJpZM4OdxA4 .

[syuukawa]

How can i do to solve it?