ClearML - Auto-Magical CI/CD to streamline your AI workload. Experiment Management, Data Management, Pipeline, Orchestration, Scheduling & Serving in one MLOps/LLMOps solution
Hi, I tried to migrate our storage from S3 to Azure. First, I tried to upload artifacts with Azure as the storage. The upload was successful, but it failed to download via the UI.
Upload Artifacts
import os
from time import sleep
from clearml import Task
from clearml.backend_config.bucket_config import AzureContainerConfig
from clearml.storage.helper import StorageHelper
def get_storage_uri() -> str:
# Set up logging to Azure
bucket_config = AzureContainerConfig(
account_name=os.environ["AZURE_STORAGE_ACCOUNT"], # type: ignore
account_key=os.environ["AZURE_STORAGE_KEY"], # type: ignore
container_name="zzzzzzzz", # type: ignore
)
StorageHelper.add_azure_configuration(bucket_config)
return StorageHelper.get_azure_storage_uri_from_config(bucket_config)
storage_uri = get_storage_uri()
print(storage_uri)
task1: Task = Task.init(
project_name="playground/boy",
task_name="Create artifact Example",
task_type=Task.TaskTypes.testing,
output_uri=storage_uri + "/testing",
# upload data file to the initialized task, inputting a name and file location
task1.upload_artifact(
name="data file",
artifact_object="assets/image.jpg",
metadata={
"key": "hoi",
},
wait_on_upload=True,
)
# close the task, to be able to initialize a new task
task1.close()
There should be a '?' between the Azure Blob URL and the SAS token, so: URL_AZURE_BLOB + '?' + SAS_TOKEN, i think?
..../artifacts/data%20file/image.jpgsp=r&st=2024.....
^^^ here should be there a "?"
# so it will be:
..../artifacts/data%20file/image.jpg?sp=r&st=2024.....
^
Hi, I tried to migrate our storage from S3 to Azure. First, I tried to upload artifacts with Azure as the storage. The upload was successful, but it failed to download via the UI.
Upload Artifacts
it success upload 🎉
Failed to download Via UI
But when trying to download the file via the UI, the blob is not found. i guess its produce not correct link: https://xxxxxxxxx.blob.core.windows.net/zzzzzzzz/testing/playground/boy/Create%20artifact%20Example.fb504e00f3524295ab8342c5ea1adb24/artifacts/data%20file/image.jpgsp=r&st=2024-06-30T17:38:17Z&se=2024-07-01T01:38:17Z&spr=https&sv=2022-11-02&sr=c&sig=XXXXXXXXXXXXX
There should be a '?' between the Azure Blob URL and the SAS token, so: URL_AZURE_BLOB + '?' + SAS_TOKEN, i think?
ClearML Version
WebApp: 1.15.1-478 • Server: 1.15.1-478 • API: 2.29 Thanks in Advance!