search

Azure / azure-sdk-for-python

This repository is for active development of the Azure SDK for Python. For consumers of the SDK we recommend visiting our public developer docs at https://learn.microsoft.com/python/azure/ or our versioned developer docs at https://azure.github.io/azure-sdk-for-python.
MIT License
4.52k stars 2.76k forks source link

Python SDKv2 as well as CLI do not archive environment #36527

Open BennoWingender opened 1 month ago

BennoWingender commented 1 month ago

Describe the bug The Python SDKv2 and the CLI both do not archive a given environment. https://learn.microsoft.com/en-us/azure/machine-learning/how-to-manage-environments-v2?view=azureml-api-2&tabs=cli

To Reproduce Steps to reproduce the behavior:

  1. CLI: az ml environment archive --name [name] --resource-group [grp_name] --workspace-name [ws_name]
  2. SDK: from azure.ai.ml import MLClient ml_client = MLClient(DefaultAzureCredential(), subscription_id, res_grp, workspace_name) ml_client.environments.archive(name=env_name, version=1) #can use other versions

Expected behavior We expect the environment to be archived.

Additional context We tried to use the function on different versions. We have sufficient rights to archive environments (We can archive environments over the graphical overlay) Code does not return an error.

github-actions[bot] commented 1 month ago

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @Azure/azure-ml-sdk @azureml-github.

achauhan-scc commented 1 month ago

@BennoWingender - I just tried to archive an environment using CLI and archived env got removed from list command az ml environment archive -g --workspace-name -n az ml environment list -g --workspace-name -o table

image

But I can fetch the environment using show command az ml environment show -g --workspace-name -n --version

Also after archive, I am not seeing it on ML studio UI, Can you please help me what do you mean not working. Are they continue to appear on list command or UI? If yes, can you capture the debug logs for both archive command and list command and share it? please ensure to remove the sensitive information from the logs.

BennoWingender commented 1 month ago

Hello, thank you for your fast response!

When I use the archive command I still see the environment in the output of the list command as well as in ML studio UI. These are the debug-logs from calling the method:

az ml environment archive -g --workspace-name -n test -v 5 --debug cli.knack.cli: Command arguments: ['ml', 'environment', 'archive', '-g', '', '--workspace-name', '', '-n', 'test', '-v', '5', '--debug'] cli.knack.cli: init debug log: Enable color in terminal. cli.knack.cli: Event: Cli.PreExecute [] cli.knack.cli: Event: CommandParser.OnGlobalArgumentsCreate [<function CLILogging.on_global_arguments at 0x7f1037f64040>, <function OutputProducer.on_global_arguments at 0x7f1037f12200>, <function CLIQuery.on_global_arguments at 0x7f1037f4fce0>] cli.knack.cli: Event: CommandInvoker.OnPreCommandTableCreate [] cli.azure.cli.core: Modules found from index for 'ml': ['azext_mlv2'] cli.azure.cli.core: Loading command modules: cli.azure.cli.core: Name Load Time Groups Commands cli.azure.cli.core: Total (0) 0.000 0 0 cli.azure.cli.core: These extensions are not installed and will be skipped: ['azext_ai_examples', 'azext_next'] cli.azure.cli.core: Loading extensions: cli.azure.cli.core: Name Load Time Groups Commands Directory azext_mlv2.manual.vendored_curated_sdk.azure.ai.ml._azure_environments: Using the default cloud configuration: 'AzureCloud'. azext_mlv2.manual.vendored_curated_sdk.azure.ai.ml._azure_environments: Using the default cloud configuration: 'AzureCloud'. azext_mlv2.manual.vendored_curated_sdk.azure.ai.ml._azure_environments: Using the default cloud configuration: 'AzureCloud'. cli.azure.cli.core: ml 2.694 24 160 /opt/az/extensions/ml cli.azure.cli.core: Total (1) 2.694 24 160
cli.azure.cli.core: Loaded 24 groups, 160 commands. cli.azure.cli.core: Found a match in the command table. cli.azure.cli.core: Raw command : ml environment archive cli.azure.cli.core: Command table: ml environment archive cli.knack.cli: Event: CommandInvoker.OnPreCommandTableTruncate [<function AzCliLogging.init_command_file_logging at 0x7f1036fd4e00>] cli.azure.cli.core.azlogging: metadata file logging enabled - writing logs to '/home/azureuser/.azure/commands/2024-07-24.09-51-51.ml_environment_archive.37074.log'. az_command_data_logger: command args: ml environment archive -g {} --workspace-name {} -n {} -v {} --debug cli.knack.cli: Event: CommandInvoker.OnPreArgumentLoad [<function register_global_subscription_argument..add_subscription_parameter at 0x7f1037025ee0>] cli.knack.commands: Configured default '' for arg resource_group_name cli.knack.commands: Configured default '' for arg workspace_name cli.knack.cli: Event: CommandInvoker.OnPostArgumentLoad [] cli.knack.cli: Event: CommandInvoker.OnPostCommandTableCreate [<function register_ids_argument..add_ids_arguments at 0x7f103703d120>, <function register_cache_arguments..add_cache_arguments at 0x7f103703d260>] cli.knack.cli: Event: CommandInvoker.OnCommandTableLoaded [] cli.knack.cli: Event: CommandInvoker.OnPreParseArgs [] cli.knack.cli: Event: CommandInvoker.OnPostParseArgs [<function OutputProducer.handle_output_argument at 0x7f1037f122a0>, <function CLIQuery.handle_query_parameter at 0x7f1037f4fd80>, <function register_ids_argument..parse_ids_arguments at 0x7f103703d1c0>] az_command_data_logger: extension name: ml az_command_data_logger: extension version: 2.27.0 cli.azure.cli.core.commands.client_factory: Getting management service client client_type=AzureMachineLearningWorkspaces urllib3.connectionpool: Starting new HTTP connection (1): 127.0.0.1:46808 urllib3.connectionpool: http://127.0.0.1:46808 "GET /MSI/auth/?resource=https://management.core.windows.net/&api-version=2017-09-01 HTTP/1.1" 200 None msrestazure.azure_active_directory: MSI: Retrieving a token from http://127.0.0.1:46808/MSI/auth/?resource=https://management.core.windows.net/&api-version=2017-09-01 msrestazure.azure_active_directory: MSI: token retrieved cli.azure.cli.core.commands.client_factory: Getting management service client client_type=_ml_client_cli urllib3.connectionpool: Starting new HTTP connection (1): 127.0.0.1:46808 urllib3.connectionpool: http://127.0.0.1:46808 "GET /MSI/auth/?resource=https://management.core.windows.net/&api-version=2017-09-01 HTTP/1.1" 200 None msrestazure.azure_active_directory: MSI: Retrieving a token from http://127.0.0.1:46808/MSI/auth/?resource=https://management.core.windows.net/&api-version=2017-09-01 msrestazure.azure_active_directory: MSI: token retrieved cli.azure.cli.core.auth.adal_authentication: MSIAuthenticationWrapper.get_token: scopes=('https://management.azure.com/.default',), kwargs={} urllib3.connectionpool: Starting new HTTP connection (1): 127.0.0.1:46808 urllib3.connectionpool: http://127.0.0.1:46808 "GET /MSI/auth/?resource=https://management.azure.com&api-version=2017-09-01 HTTP/1.1" 200 None msrestazure.azure_active_directory: MSI: Retrieving a token from http://127.0.0.1:46808/MSI/auth/?resource=https://management.azure.com&api-version=2017-09-01 msrestazure.azure_active_directory: MSI: token retrieved cli.azure.cli.core.auth.adal_authentication: Normalize expires_on: 1721815432 -> 1721815432 azure.core.pipeline.policies._universal: Request URL: 'https://management.azure.com/subscriptions/***/resourceGroups/***/providers/Microsoft.MachineLearningServices/workspaces/***/environments/test/versions/5?api-version=2023-04-01-preview' Request method: 'GET' Request headers: 'Accept': 'application/json' 'x-ms-client-request-id': '' 'User-Agent': 'azureml-cli-v2/2.27.0 azure-ai-ml/1.17.0 azsdk-python-mgmt-machinelearningservices/0.1.0 Python/3.11.8 (Linux-5.15.0-1040-azure-x86_64-with-glibc2.31)' 'Authorization': 'Bearer ' Request body: This request has no body urllib3.connectionpool: Starting new HTTPS connection (1): management.azure.com:443 urllib3.connectionpool: https://management.azure.com:443 "GET /subscriptions//resourceGroups//providers/Microsoft.MachineLearningServices/workspaces//environments/test/versions/5?api-version=2023-04-01-preview HTTP/1.1" 200 1367 azure.core.pipeline.policies._universal: Response status: '200' Response headers: 'Cache-Control': 'no-cache' 'Pragma': 'no-cache' 'Content-Length': '1367' 'Content-Type': 'application/json; charset=utf-8' 'Expires': '-1' 'Vary': 'Accept-Encoding' 'x-ms-ratelimit-remaining-subscription-global-reads': '3749' 'x-ms-ratelimit-remaining-subscription-reads': '249' 'Request-Context': 'appId=cid-v1:' 'x-ms-response-type': 'standard' 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains' 'X-Content-Type-Options': 'nosniff' 'x-aml-cluster': 'vienna-westeurope-02' 'x-request-time': '0.054' 'x-ms-request-id': '' 'x-ms-correlation-request-id': '' 'x-ms-routing-request-id': 'WESTEUROPE:20240724T095152Z:' 'X-Cache': 'CONFIG_NOCACHE' 'X-MSEdge-Ref': 'Ref A: Ref B: Ref C: 2024-07-24T09:51:51Z' 'Date': 'Wed, 24 Jul 2024 09:51:51 GMT' Response content: { "id": "/subscriptions//resourceGroups//providers/Microsoft.MachineLearningServices/workspaces//environments/test/versions/5", "name": "5", "type": "Microsoft.MachineLearningServices/workspaces/environments/versions", "properties": { "description": "Python automated test creation", "tags": {}, "properties": { "azureml.labels": "" }, "isArchived": false, "isAnonymous": false, "autoDeleteSetting": null, "environmentType": "UserCreated", "image": "mcr.microsoft.com/azureml/openmpi4.1.0-ubuntu20.04:latest", "condaFile": "{\n \"channels\": [\n \"anaconda\",\n \"conda-forge\"\n ],\n \"dependencies\": [\n \"python=3.8.16\",\n \"scikit-learn\",\n {\n \"pip\": [\n \"azureml-defaults\",\n \"numpy\"\n ]\n }\n ],\n \"name\": \"project_environment\"\n}", "osType": "Linux", "autoRebuild": "Disabled", "provisioningState": "Succeeded", "stage": "Development", "intellectualProperty": null }, "systemData": { "createdAt": "2024-07-12T10:53:38.2190888+00:00", "createdBy": "", "createdByType": "User", "lastModifiedAt": "2024-07-12T10:53:38.2190888+00:00", "lastModifiedBy": "", "lastModifiedByType": "User" } } azure.core.pipeline.policies._universal: Request URL: 'https://management.azure.com/subscriptions/***/resourceGroups/***/providers/Microsoft.MachineLearningServices/workspaces/***/environments/test/versions/5?api-version=2023-04-01-preview' Request method: 'PUT' Request headers: 'Content-Type': 'application/json' 'Content-Length': '587' 'Accept': 'application/json' 'x-ms-client-request-id': '' 'User-Agent': 'azureml-cli-v2/2.27.0 azure-ai-ml/1.17.0 azsdk-python-mgmt-machinelearningservices/0.1.0 Python/3.11.8 (Linux-5.15.0-1040-azure-x86_64-with-glibc2.31)' 'Authorization': 'Bearer ' Request body: {"properties": {"description": "Python automated test creation", "properties": {"azureml.labels": ""}, "tags": {}, "isAnonymous": false, "isArchived": true, "autoRebuild": "Disabled", "condaFile": "{\n \"channels\": [\n \"anaconda\",\n \"conda-forge\"\n ],\n \"dependencies\": [\n \"python=3.8.16\",\n \"scikit-learn\",\n {\n \"pip\": [\n \"azureml-defaults\",\n \"numpy\"\n ]\n }\n ],\n \"name\": \"project_environment\"\n}", "image": "mcr.microsoft.com/azureml/openmpi4.1.0-ubuntu20.04:latest", "osType": "Linux", "stage": "Development"}} urllib3.connectionpool: https://management.azure.com:443 "PUT /subscriptions//resourceGroups//providers/Microsoft.MachineLearningServices/workspaces//environments/test/versions/5?api-version=2023-04-01-preview HTTP/1.1" 201 1367 azure.core.pipeline.policies._universal: Response status: '201' Response headers: 'Cache-Control': 'no-cache' 'Pragma': 'no-cache' 'Content-Length': '1367' 'Content-Type': 'application/json; charset=utf-8' 'Expires': '-1' 'Location': 'https://management.azure.com/subscriptions//resourceGroups//providers/Microsoft.MachineLearningServices/workspaces//environments/test/versions/5?api-version=2023-04-01-preview' 'x-ms-ratelimit-remaining-subscription-writes': '199' 'Request-Context': 'appId=cid-v1:' 'x-ms-response-type': 'standard' 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains' 'X-Content-Type-Options': 'nosniff' 'x-aml-cluster': 'vienna-westeurope-02' 'x-request-time': '1.205' 'x-ms-ratelimit-remaining-subscription-global-writes': '2999' 'x-ms-request-id': '**' 'x-ms-correlation-request-id': '' 'x-ms-routing-request-id': 'WESTEUROPE:20240724T095153Z:' 'X-Cache': 'CONFIG_NOCACHE' 'X-MSEdge-Ref': 'Ref A: Ref B: Ref C: 2024-07-24T09:51:52Z' 'Date': 'Wed, 24 Jul 2024 09:51:53 GMT' Response content: { "id": "/subscriptions//resourceGroups//providers/Microsoft.MachineLearningServices/workspaces//environments/test/versions/5", "name": "5", "type": "Microsoft.MachineLearningServices/workspaces/environments/versions", "properties": { "description": "Python automated test creation", "tags": {}, "properties": { "azureml.labels": "" }, "isArchived": false, "isAnonymous": false, "autoDeleteSetting": null, "environmentType": "UserCreated", "image": "mcr.microsoft.com/azureml/openmpi4.1.0-ubuntu20.04:latest", "condaFile": "{\n \"channels\": [\n \"anaconda\",\n \"conda-forge\"\n ],\n \"dependencies\": [\n \"python=3.8.16\",\n \"scikit-learn\",\n {\n \"pip\": [\n \"azureml-defaults\",\n \"numpy\"\n ]\n }\n ],\n \"name\": \"project_environment\"\n}", "osType": "Linux", "autoRebuild": "Disabled", "provisioningState": "Succeeded", "stage": "Development", "intellectualProperty": null }, "systemData": { "createdAt": "2024-07-12T10:53:38.2190888+00:00", "createdBy": "", "createdByType": "User", "lastModifiedAt": "2024-07-12T10:53:38.2190888+00:00", "lastModifiedBy": "", "lastModifiedByType": "User" } } cli.knack.cli: Event: CommandInvoker.OnTransformResult [<function _resource_group_transform at 0x7f1037026340>, <function _x509_from_base64_to_hex_transform at 0x7f10370263e0>] cli.knack.cli: Event: CommandInvoker.OnFilterResult [] cli.knack.cli: Event: Cli.SuccessfulExecute [] cli.knack.cli: Event: Cli.PostExecute [<function AzCliLogging.deinit_cmd_metadata_logging at 0x7f1036fd5080>] az_command_data_logger: exit code: 0 cli.main: Command ran in 4.629 seconds (init: 0.246, invoke: 4.383) telemetry.main: Begin splitting cli events and extra events, total events: 1 telemetry.client: Accumulated 0 events. Flush the clients. telemetry.main: Finish splitting cli events and extra events, cli events: 1 telemetry.save: Save telemetry record of length 3779 in cache telemetry.main: Begin creating telemetry upload process. telemetry.process: Creating upload process: "/opt/az/bin/python3 /opt/az/lib/python3.11/site-packages/azure/cli/telemetry/init.py /home/azureuser/.azure" telemetry.process: Return from creating process telemetry.main: Finish creating telemetry upload process.

BennoWingender commented 1 month ago

Hi @achauhan-scc

I hope you're doing well. I wanted to check in and see if there are any updates regarding the archiving logs. It's been about two weeks since my last message, and I just wanted to make sure it didn't get overlooked.

Thank you in advance for your response.