Board Review: Introducing azure-ai-mlfs: fsspec implementation to support azure machine learning defined uris

[QianqianNie]

Thank you for starting the process for approval of the client library for your Azure service. Thorough review of your client library ensures that your APIs are consistent with the guidelines and the consumers of your client library have a consistently good experience when using Azure.

The Architecture Board reviews Track 2 libraries only. If your library does not meet this requirement, please reach out to Architecture Board before creating the issue.

Please reference our review process guidelines to understand what is being asked for in the issue template.

Before submitting, ensure you adjust the title of the issue appropriately.

Note that the required material must be included before a meeting can be scheduled.

Contacts and Timeline

• Service team responsible for the client library: Azure Machine Learning: Data Team
• Main contacts: PM(xunwan), Dev(qianie)
• Expected stable release date for this library: ASAP

About the Service

• Link to documentation introducing/describing the service:
• Link to the service REST APIs:
• Is the goal to release a Public Preview, Private Preview, or GA?

About the client library

• Name of client library: azure-ai-mlfs
• Link to library reference documentation: https://apiview.dev/Assemblies/Review/ba35f34c826a47d5934824d33e9d2881# https://learn.microsoft.com/en-us/python/api/azure-ai-mlfs/?view=azure-ml-py
• Is there an existing SDK library? If yes, provide link: https://pypi.org/project/azure-ai-mlfs/#history

Step 1: Champion Scenarios

Ultimately the library should be easy to use for common scenarios that developers want. Consider the following questions when thinking about champion scenarios:

1. What is the app the developer is building that uses your client library?
2. Who is the end-user of the application (the developer's customer)?
3. What features of the API need to be explained in the sample so that someone could use this API in real app?
4. How does the authentication workflow look?

See Champion Scenario section here.

Code is appreciated but optional. Pseudocode is fine.

• Champion scenario 1
  • Link to library’s sample folder:
• Champion scenario 2
  • Link to library’s sample folder:
• …
• Champion scenario n
  • Link to library’s sample folder:

Step 2: Quickstart Samples (Optional)

Include samples demonstrating how to consume the client library if available:

• Create a new resource
• Read the resource
• Modify the resource
• Delete the resource
• Error handling

• Handling race conditions/concurrency issues

      #open file to read through the file system api
      from azure.ai.mlfs import AzureMachineLearningFileSystem
      datastore_uri_fs = AzureMachineLearningFileSystem(uri='azureml://subscriptions/b1fff005-d722-4d97-99ac-7c6e9ef020aa/resourcegroups/azureml-core-e2e/workspaces/mlcore-e2e/datastores/workspaceblobstore/paths/')
  
       with datastore_uri_fs.open('workspaceblobstore/015f2183-97d6-458f-a07f-f379ef0d623b/39879821-eb9f-4f94-8119-16287c972cff/output_0.txt') as f:
           print(f.read().decode())
  
      # seamless integration with open source library
      import pandas
      df = pandas.read_parquet('azureml://subscriptions/{sub_id}/resourcegroups/{rs_group}/workspaces/{ws}
                                /datastores/workspaceblobstore/paths/myfolder/mydata.parquet')
  
      # load csv file to pandas
      import pandas
      df = pandas.read_csv('azureml://subscriptions/{sub_id}/resourcegroups/{rs_group}/workspaces/{ws}
                                /datastores/workspaceblobstore/paths/myfolder/mydata.csv')
  
      # load parquet file to dask
      import dask.dataframe as dd
      df = dd.read_parquet('azureml://subscriptions/{sub_id}/resourcegroups/{rs_group}/workspaces/{ws}
                                /datastores/workspaceblobstore/paths/myfolder/mydata.parquet')
  
      # load csv file to dask
      import dask.dataframe as dd
      df = dd.read_csv('azureml://subscriptions/{sub_id}/resourcegroups/{rs_group}/workspaces/{ws}
                                /datastores/workspaceblobstore/paths/myfolder/mydata.csv')

Thank you for your submission!

[ronniegeraghty]

Drafted

[ronniegeraghty]

Scheduled for Nov 21st from 2:05PM - 4PM PST.

[tg-msft]

Recording (MS INTERNAL ONLY)

[yeraylopez]

Hey did this ever get implemented or is the recommended package to use with SDKv2 azureml-fsspec? I am migrating my codebase to SDKv2 and I am concerned that I will have dependency conflicts since azureml-fsspec has a whole bunch of SDKv1 dependencies.

Should I be concerned?

[tawan0109]

@yeraylopez the package name mentioned above has been renamed to azureml-fsspec (https://pypi.org/project/azureml-fsspec/#description), azureml-fsspec is recommended for data access usage during interactive development (https://learn.microsoft.com/en-us/azure/machine-learning/how-to-access-data-interactive?view=azureml-api-2&tabs=adls).

For the shared dependencies owned by azure ml they will be upgraded both in v1 and v2 if new versions are available, and there are validations to ensure that any changes of dependencies won't break existing functionalities in v1 and v2, for more details about the migration please refer to https://learn.microsoft.com/en-us/azure/machine-learning/how-to-migrate-from-v1?view=azureml-api-2