search

slingdata-io / sling-cli

Sling is a CLI tool that extracts data from a source storage/database and loads it in a target storage/database.
https://docs.slingdata.io
GNU General Public License v3.0
299 stars 16 forks source link

Supporting Azurite to test pipelines locally #321

Closed pcc-cimasc closed 2 days ago

pcc-cimasc commented 2 weeks ago

The Azure SDK supports the ability to utilize a custom Blob endpoint. Similar to the following:

DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=...;BlobEndpoint=http://azurite:10000/devstoreaccount1

This is useful when utilizing a tool such as Azurite to perform integration tests of data pipelines locally, from within a dockerized environment. Unfortunately, the base Azure url is hard coded in a bunch of different locations. There's a bit of a challenge trying to implement this feature due to the different locations where the Azure type is evaluated.

The approach feels a bit hacky - if a BlobEndpoint is defined in a connection string, it will implement a prefix to the custom Blob Endpoint's URI scheme called "azurecustom-". Ideally, the Prefix code should probably detect the type of the URI, and adjust accordingly, but this was the simplest way for me to get this functionality working to allow me to connect to local Azurite container.

pcc-cimasc commented 2 days ago

fixes within 1.2.12, enable the proper container location within azure blob storage, and properly writes to azurite locally now. The delete functionality still doesn't work with azurite, but it didn't work with this PR either

flarco commented 2 days ago

Ah ok, cool. Thanks for opening the PR, I was planning to take a closer look prior to merging 1.2.12. So that works out.