ml-tooling / contaxy

MIT License
10 stars 8 forks source link

Add support for azure blob storage as object store. #22

Closed JNKielmann closed 3 years ago

JNKielmann commented 3 years ago

This PR adds the option to use Azure Blob Storage as the object store for Contaxy. The AzureBlobFileManager is implemented as an alternative to the MinioFileManager. It is automatically activated when the AZURE_BLOB_CONNECTION_STRING environment variable is set. The access key can either be included in the connection string or it can be specified in the AZURE_BLOB_TOKEN variable.

Metadata handling

For storing file metadata, the Contaxy PostgresDB is used (similarly to the MinioFileManager). Therefore, the two implementations share a lot of code. It would be possible to extract that logic to reduce the code duplication.

Issues

The Azure Blob Storage does not allow the deletion of the currently active file version. Therefore, I had to remove a test case that was trying to only delete the current version. It will still work with the MinioFileManager but the AzureBlobFileManager will throw an error. Is that behavior ok?

lgtm-com[bot] commented 3 years ago

This pull request introduces 1 alert when merging ab7be6a0abcedc0dfab97105a1dcc78f0f60951a into 48682a59ac32f90692c715994a625115a95853db - view on LGTM.com

new alerts:

lgtm-com[bot] commented 3 years ago

This pull request introduces 1 alert when merging 21198801f1e92a5b4f4e40466eab6c2c407664e0 into d9b2aad05076653ffaccb263d97eff084e7d3f85 - view on LGTM.com

new alerts: