Updated luigi.contrib.azureblob to 12.x.y series of azure.storage.blob - Githubissues

spotify / luigi

Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.

Apache License 2.0

17.71k stars 2.39k forks source link

Updated luigi.contrib.azureblob to 12.x.y series of azure.storage.blob #3290

Closed bcamel closed 2 months ago

bcamel commented 3 months ago

Description

Updated luigi.contrib.azureblob to adhere to API changes in the Azure Storage Blobs client library for Python.

Related changes:

Updated the CI script to start an Azurite service via Docker. Uses the official image mcr.microsoft.com/azure-storage/azurite
Updated the related environment variables in the unit test runner to be able to connect to the local Azurite instance.

Motivation and Context

This PR addresses issue #2965.

Have you tested this? If so, how?

Ran the GitHub Actions including the matrix items pyXYZ-azureblob.
Extended test case to cover the mkdir function of AzureBlobClient