elastic / connectors

Source code for all Elastic connectors, developed by the Search team at Elastic, and home of our Python connector development framework
https://www.elastic.co/guide/en/enterprise-search/master/index.html
Other
58 stars 116 forks source link

[Azure Blob Storage] Incorrect padding #2616

Open artem-shelkovnikov opened 3 weeks ago

artem-shelkovnikov commented 3 weeks ago

Bug Description

Issue from telemetry, see stack trace and error:

File "/usr/share/enterprise-search/lib/python3.10/site-packages/connectors/sync_job_runner.py", line 148, in execute
    await self.data_provider.ping()
  File "/usr/share/enterprise-search/lib/python3.10/site-packages/connectors/sources/azure_blob_storage.py", line 133, in ping
    await azure_base_client.get_account_information()
  File "/usr/share/enterprise-search/lib/python3.10/site-packages/azure/core/tracing/decorator_async.py", line 77, in wrapper_use_tracer
    return await func(*args, **kwargs)
  File "/usr/share/enterprise-search/lib/python3.10/site-packages/azure/storage/blob/aio/_blob_service_client_async.py", line 183, in get_account_information
    process_storage_error(error)
  File "/usr/share/enterprise-search/lib/python3.10/site-packages/azure/storage/blob/_shared/response_handlers.py", line 89, in process_storage_error
    raise storage_error
  File "/usr/share/enterprise-search/lib/python3.10/site-packages/azure/storage/blob/aio/_blob_service_client_async.py", line 181, in get_account_information
    return await self._client.service.get_account_info(cls=return_response_headers, **kwargs) # type: ignore
  File "/usr/share/enterprise-search/lib/python3.10/site-packages/azure/core/tracing/decorator_async.py", line 77, in wrapper_use_tracer
    return await func(*args, **kwargs)
  File "/usr/share/enterprise-search/lib/python3.10/site-packages/azure/storage/blob/_generated/aio/operations/_service_operations.py", line 544, in get_account_info
    pipeline_response = await self._client._pipeline.run(  # type: ignore # pylint: disable=protected-access
  File "/usr/share/enterprise-search/lib/python3.10/site-packages/azure/core/pipeline/_base_async.py", line 221, in run
    return await first_node.send(pipeline_request)
  File "/usr/share/enterprise-search/lib/python3.10/site-packages/azure/core/pipeline/_base_async.py", line 69, in send
    response = await self.next.send(request)
  File "/usr/share/enterprise-search/lib/python3.10/site-packages/azure/core/pipeline/_base_async.py", line 69, in send
    response = await self.next.send(request)
  File "/usr/share/enterprise-search/lib/python3.10/site-packages/azure/core/pipeline/_base_async.py", line 69, in send
    response = await self.next.send(request)
  [Previous line repeated 3 more times]
  File "/usr/share/enterprise-search/lib/python3.10/site-packages/azure/core/pipeline/_base_async.py", line 66, in send
    await _await_result(self._policy.on_request, request)
  File "/usr/share/enterprise-search/lib/python3.10/site-packages/azure/core/pipeline/_tools_async.py", line 56, in await_result
    result = func(*args, **kwargs)
  File "/usr/share/enterprise-search/lib/python3.10/site-packages/azure/storage/blob/_shared/authentication.py", line 142, in on_request
    self._add_authorization_header(request, string_to_sign)
  File "/usr/share/enterprise-search/lib/python3.10/site-packages/azure/storage/blob/_shared/authentication.py", line 125, in _add_authorization_header
    raise _wrap_exception(ex, AzureSigningError)

Looking at this stack overflow answer it's a proble with invalid service account json:

Accroding to the error information Error: Incorrect padding ... it seems to be not BASE64 encoding. Some changes for the key like missing the last = symbol or adding more = symbol will cause the error. And the length of correct account key of Azure Storage is 88.

We need to catch this error and display human-readable error to the user.

To Reproduce

Steps to reproduce the behavior:

  1. Create an ABS connector
  2. Enter invalid account key (just remove last symbol)
  3. See "Incorrect padding" error

Expected behavior

Error is human-actionable and states that account key is invalid + some steps on how to ensure it's valid (like the fact that its length should be 88, should be BASE64 decodable and such)