Closed IronJayx closed 2 months ago
@IronJayx - At first I thought this was due the connector being in Java (it's not). Looking more closely, I think what is happening is that the connector is not able to discover any streams. I believe you need to also specify a streams
collection in the config.
https://docs.airbyte.com/integrations/sources/azure-blob-storage#reference
Thanks @aaronsteers that was correct ! The .check() works now.
Now I want to disable parser to process unstructured files (images/ videos) but I guess that is for another issue.
import os
import airbyte as ab
from dotenv import load_dotenv
load_dotenv()
# Configure and read from the source
source = ab.get_source(
"source-azure-blob-storage",
install_if_missing=True,
config={
# "authentification": ["airbytehq/pyAirbyte"],
"credentials": {
"auth_type": "storage_account_key",
"azure_blob_storage_account_key": os.environ.get('AZURE_STORAGE_ACCOUNT_KEY')
},
"azure_blob_storage_account_name": os.environ.get('AZURE_STORAGE_ACCOUNT_NAME'),
"azure_blob_storage_container_name": os.environ.get('AZURE_STORAGE_CONTAINER_NAME'),
"streams": [{
"name": "all",
"format": {
"filetype": "unstructured",
"skip_unprocessable_files": False,
},
"globs": ["**"]
}]
},
)
source.check()
Hi,
I am getting a catalag error while trying to connect/ read from Azure storage.
Any idea on why ?
All help would be much appreciated, thanks !
Here is my code:
Here is the error: