allegroai / clearml-server

ClearML - Auto-Magical CI/CD to streamline your AI workload. Experiment Management, Data Management, Pipeline, Orchestration, Scheduling & Serving in one MLOps/LLMOps solution
https://clear.ml/docs
Other
364 stars 131 forks source link

Error Download via UI for Azure Blob Storage #247

Open muhammadAgfian96 opened 2 days ago

muhammadAgfian96 commented 2 days ago

Hi, I tried to migrate our storage from S3 to Azure. First, I tried to upload artifacts with Azure as the storage. The upload was successful, but it failed to download via the UI.

Upload Artifacts

import os
from time import sleep

from clearml import Task
from clearml.backend_config.bucket_config import AzureContainerConfig
from clearml.storage.helper import StorageHelper

def get_storage_uri() -> str:
    # Set up logging to Azure
    bucket_config = AzureContainerConfig(
        account_name=os.environ["AZURE_STORAGE_ACCOUNT"],  # type: ignore
        account_key=os.environ["AZURE_STORAGE_KEY"],  # type: ignore
        container_name="zzzzzzzz",  # type: ignore
    )
    StorageHelper.add_azure_configuration(bucket_config)
    return StorageHelper.get_azure_storage_uri_from_config(bucket_config)

storage_uri = get_storage_uri()
print(storage_uri)
task1: Task = Task.init(
    project_name="playground/boy",
    task_name="Create artifact Example",
    task_type=Task.TaskTypes.testing,
    output_uri=storage_uri + "/testing",

# upload data file to the initialized task, inputting a name and file location
task1.upload_artifact(
    name="data file",
    artifact_object="assets/image.jpg",
    metadata={
        "key": "hoi",
    },
    wait_on_upload=True,
)

# close the task, to be able to initialize a new task
task1.close()

it success upload 🎉 image

Failed to download Via UI

But when trying to download the file via the UI, the blob is not found. image i guess its produce not correct link: https://xxxxxxxxx.blob.core.windows.net/zzzzzzzz/testing/playground/boy/Create%20artifact%20Example.fb504e00f3524295ab8342c5ea1adb24/artifacts/data%20file/image.jpgsp=r&st=2024-06-30T17:38:17Z&se=2024-07-01T01:38:17Z&spr=https&sv=2022-11-02&sr=c&sig=XXXXXXXXXXXXX

There should be a '?' between the Azure Blob URL and the SAS token, so: URL_AZURE_BLOB + '?' + SAS_TOKEN, i think?

..../artifacts/data%20file/image.jpgsp=r&st=2024.....
                                   ^^^ here should be there a "?"

# so it will be:
..../artifacts/data%20file/image.jpg?sp=r&st=2024.....
                                    ^

ClearML Version

WebApp: 1.15.1-478 • Server: 1.15.1-478 • API: 2.29 Thanks in Advance!

ainoam commented 2 days ago

Thanks for reporting @muhammadAgfian96 - Hope to be able to provide a fix soon.