search

microsoft / vscode-jupyter

VS Code Jupyter extension
https://marketplace.visualstudio.com/items?itemName=ms-toolsai.jupyter
MIT License
1.28k stars 286 forks source link

Azure ML Widgets not working when connecting to Azure ML Notebooks from VS Code #5900

Closed ansaxena1 closed 3 years ago

ansaxena1 commented 3 years ago

Environment data

Expected behaviour

Widget should be displayed.

Actual behaviour

Output: No renderer could be found for MIME type: application/aml.mini.widget.v1.

Steps to reproduce:

Run the following, replacing source directory and compute target as necessary:

from azureml.widgets import RunDetails

ws = Workspace.from_config()
experiment = Experiment(workspace=ws, name='day1-experiment-hello')

config = ScriptRunConfig(source_directory='./code/hello', script='hello.py', compute_target='TestComputeForDon')

run = experiment.submit(config)
RunDetails(run).show()
claudiaregio commented 3 years ago

Hi @ansaxena1, Could you please update your VS Code version, Python extension version, and your jupyter extension version and let me know if this issue persists?

DonJayamanne commented 3 years ago

A little confused here. @ansaxena1 you confident m confirmed this was working a few days ago here https://github.com/microsoft/vscode-jupyter/issues/5704#issuecomment-830360433

ansaxena1 commented 3 years ago

It was working at that point. I seem to have some intermittent issues where the widget only shows up occasionally for me. Not sure if there is something misconfigured with my environment?

DonJayamanne commented 3 years ago

Please update vscode, Jupyter and your portion packages to the latest version

rchiodo commented 3 years ago

Please reopen if the issue is continuing to happen

ansaxena1 commented 3 years ago

I updated vs code to version 1.56.2 and I am still seeing this error

ansaxena1 commented 3 years ago

Logs from debug console:

electron/js2c/renderer_init.js:61 The vm module of Node.js is deprecated in the renderer process and will be removed. t._load @ electron/js2c/renderer_init.js:61 console.ts:137 [Extension Host] (node:20736) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.(Use Code --trace-deprecation ... to show where the warning was created) S @ console.ts:137 2console.ts:137 [Extension Host] Terminal not found: 2 S @ console.ts:137 TMScopeRegistry.ts:45 Overwriting grammar scope name to file mapping for scope source.yaml. Old grammar file: file:///c%3A/Users/ansaxena/AppData/Local/Programs/Microsoft%20VS%20Code/resources/app/extensions/yaml/syntaxes/yaml.tmLanguage.json. New grammar file: vscode-remote://amlext%2Bfilepath%3D%2Btelemetrytrackingid%3D0600e709-ef8b-466e-a84a-63629a83527c%2Bprivatelink%3Dfalse%2Bsessionid%3D2raxr%2Burl%3Dansaxenatestcompute.eastus2.instances.azureml.ms/home/azureuser/.vscode-server/extensions/ms-toolsai.vscode-ai-0.6.22/syntaxes/yaml.tmLanguage.json register @ TMScopeRegistry.ts:45 console.ts:137 [Extension Host] (node:7995) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.(Use node --trace-deprecation ... to show where the warning was created) S @ console.ts:137 console.ts:137 [Extension Host] [2021-05-19T17:50:40.917Z] Information: WebSocket connected to wss://vienna-eastus2.service.signalr.net/client/?hub=signalrhub&asrs.op=%2Fwebsocket%2Fv1.0%2Fhub&asrs_request_id=WxrghQ0iAAA%3D&id=Hfw-ee6sBtPHN0I_Co6S6wf128ee3d1&access_token=eyJhbGciOiJIUzI1NiIsImtpZCI6IjE2ODQ4Mjc4NjMiLCJ0eXAiOiJKV1QifQ.eyJhc3JzLnMudWlkIjoiUF9xSE04R0o0RjFSQVAydE0wd1ZQcWJDd1dNQm4zQlhEcHlOOF9VS3Z6RSIsImFzcnMucy5hdXQiOiJBdXRoZW50aWNhdGlvblR5cGVzLkZlZGVyYXRpb24iLCJhc3JzLnUuYXVkIjoiaHR0cHM6Ly9tYW5hZ2VtZW50LmNvcmUud2luZG93cy5uZXQvIiwiaXNzIjoiaHR0cHM6Ly9zdHMud2luZG93cy5uZXQvNzJmOTg4YmYtODZmMS00MWFmLTkxYWItMmQ3Y2QwMTFkYjQ3LyIsImFzcnMudS5pYXQiOiIxNjIxNDQ2MzM2IiwiYXNycy51Lm5iZiI6IjE2MjE0NDYzMzYiLCJhc3JzLnUuZXhwIjoiMTYyMTQ1MDIzNiIsIl9jbGFpbV9uYW1lcyI6eyJncm91cHMiOiJzcmMxIn0sIl9jbGFpbV9zb3VyY2VzIjp7InNyYzEiOnsiZW5kcG9pbnQiOiJodHRwczovL2dyYXBoLndpbmRvd3MubmV0LzcyZjk4OGJmLTg2ZjEtNDFhZi05MWFiLTJkN2NkMDExZGI0Ny91c2Vycy81NzE5ZWQyNS0wMTY3LTQ2MzktYTNlNC1kN2RkMzBlZjI3YjQvZ2V0TWVtYmVyT2JqZWN0cyJ9fSwiYWlvIjoiQVZRQXEvOFRBQUFBQ3B5eUdGbmNGNVlRZHVCT2orTWdnUUUrRjBLeXdZVGNvb0tXZUZZVG1Wbnl1UFJmblhMc1F5V3QzbVFKTVdSZmU2bnRjM09ydm9qMmo3bnp3SFIxUmdDcCt5NGxvU1VtMWR1TmhhcHlnSnc9IiwiYXBwaWQiOiJhZWJjNjQ0My05OTZkLTQ1YzItOTBmMC0zODhmZjk2ZmFhNTYiLCJhcHBpZGFjciI6IjAiLCJmYW1pbHlfbmFtZSI6IlNheGVuYSIsImdpdmVuX25hbWUiOiJBbm91c2hrYSIsImlwYWRkciI6IjUyLjE3Ny4xNDkuNDkiLCJuYW1lIjoiQW5vdXNoa2EgU2F4ZW5hIiwib2lkIjoiNTcxOWVkMjUtMDE2Ny00NjM5LWEzZTQtZDdkZDMwZWYyN2I0Iiwib25wcmVtX3NpZCI6IlMtMS01LTIxLTIxMjc1MjExODQtMTYwNDAxMjkyMC0xODg3OTI3NTI3LTM3NTcxMjA2IiwicHVpZCI6IjEwMDMyMDAwNjFCOTlGNTAiLCJyaCI6IjAuQVJvQXY0ajVjdkdHcjBHUnF5MTgwQkhiUjBOa3ZLNXRtY0pGa1BBNGpfbHZxbFlhQUhRLiIsInRpZCI6IjcyZjk4OGJmLTg2ZjEtNDFhZi05MWFiLTJkN2NkMDExZGI0NyIsInV0aSI6IkpyMWdPTmxNckVHdXZ4YUZrbXBIQUEiLCJ2ZXIiOiIxLjAiLCJ3aWRzIjoiYjc5ZmJmNGQtM2VmOS00Njg5LTgxNDMtNzZiMTk0ZTg1NTA5IiwieG1zX3RjZHQiOjEyODkyNDE1NDcsIm5iZiI6MTYyMTQ0NjY0MCwiZXhwIjoxNjIxNDUwMjQwLCJpYXQiOjE2MjE0NDY2NDAsImF1ZCI6Imh0dHBzOi8vdmllbm5hLWVhc3R1czIuc2VydmljZS5zaWduYWxyLm5ldC9jbGllbnQvP2h1Yj1zaWduYWxyaHViIn0.FOoQPptOYJ5kO9amdX1i1DejsovbsQPZVT_N67DoSVY. VM15:3 Rendering widgets on notebook open is not supported. r @ VM15:3 console.ts:137 [Extension Host] Starting WebSocket: wss://ansaxenatestcompute.eastus2.instances.azureml.ms/api/kernels/1461ba77-ffb2-413b-9828-f87f6e47dbdb console.ts:137 [Extension Host] Kernel: connected (1461ba77-ffb2-413b-9828-f87f6e47dbdb) console.ts:137 [Extension Host] Starting WebSocket: wss://ansaxenatestcompute.eastus2.instances.azureml.ms/api/kernels/08e31804-48aa-4401-8fd6-8db7235a4752 console.ts:137 [Extension Host] Kernel: connected (08e31804-48aa-4401-8fd6-8db7235a4752) console.ts:137 [Extension Host] Starting WebSocket: wss://ansaxenatestcompute.eastus2.instances.azureml.ms/api/kernels/05c5f2fb-e973-4b9f-8275-23b4b3edd55a VM22:4 IPyWidget kernel initializing... VM22:4 UI PostOffice Sent IPyWidgets_Ready VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received updateSelectedMarkdownPreviews VM22:4 Received some pending message updateSelectedMarkdownPreviews VM22:4 UI PostOffice Received updateSelectedMarkdownPreviews VM22:4 Received some pending message updateSelectedMarkdownPreviews console.ts:137 [Extension Host] Kernel: connected (05c5f2fb-e973-4b9f-8275-23b4b3edd55a) VM22:4 UI PostOffice Received updateSelectedMarkdownPreviews VM22:4 Received some pending message updateSelectedMarkdownPreviews VM22:4 UI PostOffice Received updateSelectedMarkdownPreviews VM22:4 Received some pending message updateSelectedMarkdownPreviews VM22:4 UI PostOffice Received updateSelectedMarkdownPreviews VM22:4 Received some pending message updateSelectedMarkdownPreviews VM22:4 UI PostOffice Received updateSelectedMarkdownPreviews VM22:4 Received some pending message updateSelectedMarkdownPreviews VM22:4 UI PostOffice Received updateSelectedMarkdownPreviews VM22:4 Received some pending message updateSelectedMarkdownPreviews log.ts:296 ERR Error: [File Watcher (chokidar)] Inotify limit reached (ENOSPC) VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received updateSelectedMarkdownPreviews VM22:4 Received some pending message updateSelectedMarkdownPreviews VM22:4 UI PostOffice Received updateSelectedMarkdownPreviews VM22:4 Received some pending message updateSelectedMarkdownPreviews VM22:4 UI PostOffice Received updateSelectedMarkdownPreviews VM22:4 Received some pending message updateSelectedMarkdownPreviews VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received html VM22:4 Received some pending message html VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received ack-dimension VM22:4 Received some pending message ack-dimension VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received updateSelectedMarkdownPreviews VM22:4 Received some pending message updateSelectedMarkdownPreviews VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received updateSelectedMarkdownPreviews VM22:4 Received some pending message updateSelectedMarkdownPreviews resourceLoading.ts:112 Error: Unable to read file 'vscode-remote://amlext+filepath=+telemetrytrackingid=0600e709-ef8b-466e-a84a-63629a83527c+privatelink=false+sessionid=2raxr+url=ansaxenatestcompute.eastus2.instances.azureml.ms/vscode-resource?{"requestResourcePath":"/home/azureuser/.vscode-server/extensions/ms-toolsai.jupyter-2021.6.832593372/out/datascience-ui/viewers/node_modules/font-awesome/css/font-awesome.min.css"}' (Error: Unable to resolve non-existing file 'vscode-remote://amlext+filepath=+telemetrytrackingid=0600e709-ef8b-466e-a84a-63629a83527c+privatelink=false+sessionid=2raxr+url=ansaxenatestcompute.eastus2.instances.azureml.ms/vscode-resource?{"requestResourcePath":"/home/azureuser/.vscode-server/extensions/ms-toolsai.jupyter-2021.6.832593372/out/datascience-ui/viewers/node_modules/font-awesome/css/font-awesome.min.css"}') at o.doReadFileStream (fileService.ts:529) at async w (resourceLoading.ts:98) at async p.loadResource (baseWebviewElement.ts:487) VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received updateSelectedMarkdownPreviews VM22:4 Received some pending message updateSelectedMarkdownPreviews VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received updateSelectedMarkdownPreviews VM22:4 Received some pending message updateSelectedMarkdownPreviews VM22:4 UI PostOffice Received updateSelectedMarkdownPreviews VM22:4 Received some pending message updateSelectedMarkdownPreviews DevTools failed to load SourceMap: Could not parse content for vscode-webview://06a813a1-9e9f-40c6-8643-9beb1486f92c/ipywidgetsRenderer.js.map: Unexpected end of JSON input DevTools failed to load SourceMap: Could not parse content for vscode-webview://06a813a1-9e9f-40c6-8643-9beb1486f92c/ipywidgets.js.map: Unexpected end of JSON input DevTools failed to load SourceMap: Could not parse content for vscode-webview://06a813a1-9e9f-40c6-8643-9beb1486f92c/ipywidgetsKernel.js.map: Unexpected end of JSON input VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received ack-dimension VM22:4 Received some pending message ack-dimension VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll VM22:4 UI PostOffice Received view-scroll VM22:4 Received some pending message view-scroll

ansaxena1 commented 3 years ago

Output from the cell with the widget: image

DonJayamanne commented 3 years ago

Can you confirm this works in jupyter, when using the exact same python environment? Is it conda (you haven't provided that information, you've given xxx)

ansaxena1 commented 3 years ago

Yes this works in Jupyter. I am not using conda? Not sure what details to provide here

DonJayamanne commented 3 years ago

could you upload the ipynb after running it in Jupyter (I'd like to see the raw outputs in the ipynb )

ansaxena1 commented 3 years ago

Unable to upload a .ipynb file, so here is the notebook content:

Details { "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "Copyright (c) Microsoft Corporation. All rights reserved.\n", "\n", "Licensed under the MIT License." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/NotebookVM/tutorials/get-started-day1/day1-part2-hello-world.png)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Tutorial: \"Hello World\" (Part 2 of 4)\n", "\n", "---\n", "## Introduction\n", "In **part 2 of this get started series**, you will submit a trivial \"hello world\" python script to the cloud by:\n", "\n", "- Running Python code in the cloud with Azure Machine Learning SDK\n", "- Switching between debugging locally on a compute instance.\n", "- Submitting remote runs in the cloud\n", "- Monitoring and recording runs in the Azure Machine Learning studio\n", "\n", "This notebook follows the steps provided on the [Python (day 1) - \"hello world\" documentation page](https://aka.ms/day1aml). This tutorial is part of a **four-part tutorial series** in which you learn the fundamentals of Azure Machine Learning and complete simple jobs-based machine learning tasks in the Azure cloud. It builds off the work you completed in [Tutorial part 1: set up an Azure Machine Learning compute cluster](day1-part1-setup.ipynb).\n", "\n", "## Pre-requisites\n", "\n", "- Complete [Tutorial part 1: set up an Azure Machine Learning compute cluster](day1-part1-setup.ipynb) if you don't already have an Azure Machine Learning compute cluster.\n", "- Familiarity with Python and Machine Learning concepts.\n", "- If you are using a compute instance in Azure Machine Learning to run this notebook series, you are all set. Otherwise, please follow the [Configure a development environment for Azure Machine Learning](https://docs.microsoft.com/azure/machine-learning/how-to-configure-environment)\n", "---" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Your code\n", "\n", "In the `code/hello` subdirectory you will find a trivial python script [hello.py](code/hello/hello.py) that has the following code:\n", "\n", "```Python\n", "# code/hello/hello.py\n", "print(\"hello world!\")\n", "```\n", "\n", "In this tutorial you are going to submit this trivial python script to an Azure Machine Learning Compute Cluster." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Test in your development environment\n", "\n", "You can test your code works on a compute instance or locally (for example, a laptop), which has the benefit of interactive debugging of code:" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Submit your code to Azure Machine Learning\n", "\n", "Below you create a __*control script*__ this is where you specify _how_ your code is submitted to Azure Machine Learning. The code you submit to Azure Machine Learning (in this case `hello.py`) does not need anything specific to Azure Machine Learning - it can be any valid Python code. It is only the control script that is Azure Machine Learning specific.\n", "\n", "The code below will show a Jupyter widget that tracks the progress of your run, and displays logs.\n", "\n", "> ! NOTE
\n", "> The very first run will take 5-10minutes to complete. This is because in the background a docker image is built in the cloud, the compute cluster is resized from 0 to 1 node, and the docker image is downloaded to the compute. Subsequent runs are much quicker (~15 seconds) as the docker image is cached on the compute - you can test this by resubmitting the code below after the first run has completed." ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "gather": { "logged": 1619814177604 }, "tags": [ "remote run", "batchai", "configure run", "use notebook widget" ] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Performing interactive authentication. Please follow the instructions on the terminal.\n", "To sign in, use a web browser to open the page https://microsoft.com/devicelogin and enter the code EESPMNF8T to authenticate.\n", "You have logged in. Now let us find all the subscriptions to which you have access...\n", "Interactive authentication successfully completed.\n" ] }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "c0fec76bc0474cab8864dcc9174fe0eb", "version_major": 2, "version_minor": 0 }, "text/plain": [ "_UserRunWidget(widget_settings={'childWidgetDisplay': 'popup', 'send_telemetry': False, 'log_level': 'INFO', '…" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "application/aml.mini.widget.v1": "{\"status\": \"Completed\", \"workbench_run_details_uri\": \"https://ml.azure.com/experiments/day1-experiment-hello/runs/day1-experiment-hello_1621536376_585b456b?wsid=/subscriptions/e9b2ec51-5c94-4fa8-809a-dc1e695e4896/resourcegroups/DefaultResourceGroup-EUS2/workspaces/ansaxena\", \"run_id\": \"day1-experiment-hello_1621536376_585b456b\", \"run_properties\": {\"run_id\": \"day1-experiment-hello_1621536376_585b456b\", \"created_utc\": \"2021-05-20T18:46:17.964514Z\", \"properties\": {\"_azureml.ComputeTargetType\": \"amlcompute\", \"ContentSnapshotId\": \"161d1596-4313-4f80-832b-36ba5f6a5228\", \"ProcessInfoFile\": \"azureml-logs/process_info.json\", \"ProcessStatusFile\": \"azureml-logs/process_status.json\"}, \"tags\": {\"_aml_system_ComputeTargetStatus\": \"{\\\"AllocationState\\\":\\\"steady\\\",\\\"PreparingNodeCount\\\":0,\\\"RunningNodeCount\\\":0,\\\"CurrentNodeCount\\\":1}\"}, \"script_name\": null, \"arguments\": null, \"end_time_utc\": \"2021-05-20T18:46:56.891339Z\", \"status\": \"Completed\", \"log_files\": {\"azureml-logs/55_azureml-execution-tvmps_1e9a1b3c4303ec6155cde09e296d667844eff0b04589f34083c7a8629a3be419_d.txt\": \"https://ansaxena9430492851.blob.core.windows.net/azureml/ExperimentRun/dcid.day1-experiment-hello_1621536376_585b456b/azureml-logs/55_azureml-execution-tvmps_1e9a1b3c4303ec6155cde09e296d667844eff0b04589f34083c7a8629a3be419_d.txt?sv=2019-02-02&sr=b&sig=QgNcXah9eagAXzp1KSbtv63qv5e%2BrzbgB74i%2Bw8L%2Fpk%3D&st=2021-05-20T18%3A36%3A49Z&se=2021-05-21T02%3A46%3A49Z&sp=r\", \"azureml-logs/65_job_prep-tvmps_1e9a1b3c4303ec6155cde09e296d667844eff0b04589f34083c7a8629a3be419_d.txt\": \"https://ansaxena9430492851.blob.core.windows.net/azureml/ExperimentRun/dcid.day1-experiment-hello_1621536376_585b456b/azureml-logs/65_job_prep-tvmps_1e9a1b3c4303ec6155cde09e296d667844eff0b04589f34083c7a8629a3be419_d.txt?sv=2019-02-02&sr=b&sig=tcLQXdcTUtDfwgBpWpj6PjxtNzezcZCBz6bixFf5qh4%3D&st=2021-05-20T18%3A36%3A49Z&se=2021-05-21T02%3A46%3A49Z&sp=r\", \"azureml-logs/70_driver_log.txt\": \"https://ansaxena9430492851.blob.core.windows.net/azureml/ExperimentRun/dcid.day1-experiment-hello_1621536376_585b456b/azureml-logs/70_driver_log.txt?sv=2019-02-02&sr=b&sig=jIw1jx4FUIXUQ3sVqEEoX2feIH%2FPk79oJoN%2BZZp3Leg%3D&st=2021-05-20T18%3A36%3A49Z&se=2021-05-21T02%3A46%3A49Z&sp=r\", \"azureml-logs/75_job_post-tvmps_1e9a1b3c4303ec6155cde09e296d667844eff0b04589f34083c7a8629a3be419_d.txt\": \"https://ansaxena9430492851.blob.core.windows.net/azureml/ExperimentRun/dcid.day1-experiment-hello_1621536376_585b456b/azureml-logs/75_job_post-tvmps_1e9a1b3c4303ec6155cde09e296d667844eff0b04589f34083c7a8629a3be419_d.txt?sv=2019-02-02&sr=b&sig=AzUFiAem3e6n%2FWEBQ2tGfiBOFLMyCuG0fAMzsr%2BJK3w%3D&st=2021-05-20T18%3A36%3A49Z&se=2021-05-21T02%3A46%3A49Z&sp=r\", \"azureml-logs/process_info.json\": \"https://ansaxena9430492851.blob.core.windows.net/azureml/ExperimentRun/dcid.day1-experiment-hello_1621536376_585b456b/azureml-logs/process_info.json?sv=2019-02-02&sr=b&sig=%2B9fOnzHyG70M6Fe%2FDwLo%2Ba%2F53Xqv9kb3zCQKSNp9Vp4%3D&st=2021-05-20T18%3A36%3A49Z&se=2021-05-21T02%3A46%3A49Z&sp=r\", \"azureml-logs/process_status.json\": \"https://ansaxena9430492851.blob.core.windows.net/azureml/ExperimentRun/dcid.day1-experiment-hello_1621536376_585b456b/azureml-logs/process_status.json?sv=2019-02-02&sr=b&sig=UxfnTuiP9ZkHBB7tYHLn03TumS0N%2F2dJu%2BBm8o7thTw%3D&st=2021-05-20T18%3A36%3A49Z&se=2021-05-21T02%3A46%3A49Z&sp=r\", \"logs/azureml/103_azureml.log\": \"https://ansaxena9430492851.blob.core.windows.net/azureml/ExperimentRun/dcid.day1-experiment-hello_1621536376_585b456b/logs/azureml/103_azureml.log?sv=2019-02-02&sr=b&sig=3tW1ubbkIve2GcUnKnTv5gLgC4m5JcBiN7jtJuA0ytc%3D&st=2021-05-20T18%3A36%3A49Z&se=2021-05-21T02%3A46%3A49Z&sp=r\", \"logs/azureml/job_prep_azureml.log\": \"https://ansaxena9430492851.blob.core.windows.net/azureml/ExperimentRun/dcid.day1-experiment-hello_1621536376_585b456b/logs/azureml/job_prep_azureml.log?sv=2019-02-02&sr=b&sig=NwsOOEuEuvbPgua6uIYCfz77sqqm8T%2BK02rAzG0MlcI%3D&st=2021-05-20T18%3A36%3A49Z&se=2021-05-21T02%3A46%3A49Z&sp=r\", \"logs/azureml/job_release_azureml.log\": \"https://ansaxena9430492851.blob.core.windows.net/azureml/ExperimentRun/dcid.day1-experiment-hello_1621536376_585b456b/logs/azureml/job_release_azureml.log?sv=2019-02-02&sr=b&sig=l57iXd6Z%2BxoNU9Ab%2Fq4kBLYXFti3fy2HRUortJ%2BldrI%3D&st=2021-05-20T18%3A36%3A49Z&se=2021-05-21T02%3A46%3A49Z&sp=r\"}, \"log_groups\": [[\"azureml-logs/process_info.json\", \"azureml-logs/process_status.json\", \"logs/azureml/job_prep_azureml.log\", \"logs/azureml/job_release_azureml.log\"], [\"azureml-logs/55_azureml-execution-tvmps_1e9a1b3c4303ec6155cde09e296d667844eff0b04589f34083c7a8629a3be419_d.txt\"], [\"azureml-logs/65_job_prep-tvmps_1e9a1b3c4303ec6155cde09e296d667844eff0b04589f34083c7a8629a3be419_d.txt\"], [\"azureml-logs/70_driver_log.txt\"], [\"azureml-logs/75_job_post-tvmps_1e9a1b3c4303ec6155cde09e296d667844eff0b04589f34083c7a8629a3be419_d.txt\"], [\"logs/azureml/103_azureml.log\"]], \"run_duration\": \"0:00:38\"}, \"child_runs\": [], \"children_metrics\": {}, \"run_metrics\": [], \"run_logs\": \"2021-05-20 18:46:40,555|azureml|DEBUG|Inputs:: kwargs: {'OutputCollection': True, 'EnableMLflowTracking': True, 'snapshotProject': True, 'only_in_process_features': True, 'skip_track_logs_dir': True}, track_folders: None, deny_list: None, directories_to_watch: []\\n2021-05-20 18:46:40,556|azureml.history._tracking.PythonWorkingDirectory|DEBUG|Execution target type: batchai\\n2021-05-20 18:46:40,573|azureml.history._tracking.PythonWorkingDirectory|DEBUG|Failed to import pyspark with error: No module named 'pyspark'\\n2021-05-20 18:46:40,573|azureml.history._tracking.PythonWorkingDirectory.workingdir|DEBUG|Pinning working directory for filesystems: ['pyfs']\\n2021-05-20 18:46:41,090|azureml.core.run|DEBUG|Adding new factory for run source azureml.scriptrun\\n2021-05-20 18:46:41,091|azureml.core.run|DEBUG|Identity in use is not set. Falling back to using AMLToken\\n2021-05-20 18:46:41,091|azureml.core.run|DEBUG|Using AMLToken auth for remote run\\n2021-05-20 18:46:41,094|azureml.core.authentication.TokenRefresherDaemon|DEBUG|Starting daemon and triggering first instance\\n2021-05-20 18:46:41,103|azureml._restclient.clientbase|INFO|Created a worker pool for first use\\n2021-05-20 18:46:41,104|azureml.core.authentication|DEBUG|Time to expire 1814375.895916 seconds\\n2021-05-20 18:46:41,104|azureml._restclient.service_context|DEBUG|Created a static thread pool for ServiceContext class\\n2021-05-20 18:46:41,104|azureml._restclient.clientbase|DEBUG|ClientBase: Calling get with url None\\n2021-05-20 18:46:41,141|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://eastus2.api.azureml.ms.\\n2021-05-20 18:46:41,142|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://eastus2.api.azureml.ms.\\n2021-05-20 18:46:41,142|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://eastus2.api.azureml.ms.\\n2021-05-20 18:46:41,142|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://eastus2.api.azureml.ms.\\n2021-05-20 18:46:41,143|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://eastus2.api.azureml.ms.\\n2021-05-20 18:46:41,143|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://eastus2.api.azureml.ms.\\n2021-05-20 18:46:41,143|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://eastus2.api.azureml.ms.\\n2021-05-20 18:46:41,184|azureml._SubmittedRun#day1-experiment-hello_1621536376_585b456b.RunHistoryFacade.RunClient.get_by_exp_id-async:False|DEBUG|[START]\\n2021-05-20 18:46:41,184|azureml._SubmittedRun#day1-experiment-hello_1621536376_585b456b.RunHistoryFacade.RunClient|DEBUG|ClientBase: Calling get_by_exp_id with url /history/v1.0/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.MachineLearningServices/workspaces/{workspaceName}/experimentids/{experimentId}/runs/{runId}\\n2021-05-20 18:46:41,244|azureml._SubmittedRun#day1-experiment-hello_1621536376_585b456b.RunHistoryFacade.RunClient.get_by_exp_id-async:False|DEBUG|[STOP]\\n2021-05-20 18:46:41,245|azureml._SubmittedRun#day1-experiment-hello_1621536376_585b456b|DEBUG|Constructing run from dto. type: azureml.scriptrun, source: None, props: {'_azureml.ComputeTargetType': 'amlcompute', 'ContentSnapshotId': '161d1596-4313-4f80-832b-36ba5f6a5228', 'ProcessInfoFile': 'azureml-logs/process_info.json', 'ProcessStatusFile': 'azureml-logs/process_status.json'}\\n2021-05-20 18:46:41,245|azureml._SubmittedRun#day1-experiment-hello_1621536376_585b456b.RunContextManager|DEBUG|Valid logs dir, setting up content loader\\n2021-05-20 18:46:41,245|azureml|WARNING|Could not import azureml.mlflow or azureml.contrib.mlflow mlflow APIs will not run against AzureML services. Add azureml-mlflow as a conda dependency for the run if this behavior is desired\\n2021-05-20 18:46:41,245|azureml.WorkerPool|DEBUG|[START]\\n2021-05-20 18:46:41,246|azureml.SendRunKillSignal|DEBUG|[START]\\n2021-05-20 18:46:41,246|azureml.RunStatusContext|DEBUG|[START]\\n2021-05-20 18:46:41,246|azureml._SubmittedRun#day1-experiment-hello_1621536376_585b456b.RunContextManager.RunStatusContext|DEBUG|[START]\\n2021-05-20 18:46:41,246|azureml.MetricsClient|DEBUG|[START]\\n2021-05-20 18:46:41,246|azureml._SubmittedRun#day1-experiment-hello_1621536376_585b456b.RunHistoryFacade.MetricsClient|DEBUG|[START]\\n2021-05-20 18:46:41,246|azureml.WorkingDirectoryCM|DEBUG|[START]\\n2021-05-20 18:46:41,246|azureml.history._tracking.PythonWorkingDirectory.workingdir|DEBUG|[START]\\n2021-05-20 18:46:41,246|azureml.history._tracking.PythonWorkingDirectory|INFO|Current working dir: /mnt/batch/tasks/shared/LS_root/jobs/ansaxena/azureml/day1-experiment-hello_1621536376_585b456b/mounts/workspaceblobstore/azureml/day1-experiment-hello_1621536376_585b456b\\n2021-05-20 18:46:41,246|azureml.history._tracking.PythonWorkingDirectory.workingdir|DEBUG|Calling pyfs\\n2021-05-20 18:46:41,246|azureml.history._tracking.PythonWorkingDirectory.workingdir|DEBUG|Storing working dir for pyfs as /mnt/batch/tasks/shared/LS_root/jobs/ansaxena/azureml/day1-experiment-hello_1621536376_585b456b/mounts/workspaceblobstore/azureml/day1-experiment-hello_1621536376_585b456b\\n2021-05-20 18:46:41,316|azureml.history._tracking.PythonWorkingDirectory.workingdir|DEBUG|Calling pyfs\\n2021-05-20 18:46:41,316|azureml.history._tracking.PythonWorkingDirectory|INFO|Current working dir: /mnt/batch/tasks/shared/LS_root/jobs/ansaxena/azureml/day1-experiment-hello_1621536376_585b456b/mounts/workspaceblobstore/azureml/day1-experiment-hello_1621536376_585b456b\\n2021-05-20 18:46:41,316|azureml.history._tracking.PythonWorkingDirectory.workingdir|DEBUG|Reverting working dir from /mnt/batch/tasks/shared/LS_root/jobs/ansaxena/azureml/day1-experiment-hello_1621536376_585b456b/mounts/workspaceblobstore/azureml/day1-experiment-hello_1621536376_585b456b to /mnt/batch/tasks/shared/LS_root/jobs/ansaxena/azureml/day1-experiment-hello_1621536376_585b456b/mounts/workspaceblobstore/azureml/day1-experiment-hello_1621536376_585b456b\\n2021-05-20 18:46:41,316|azureml.history._tracking.PythonWorkingDirectory|INFO|Working dir is already updated /mnt/batch/tasks/shared/LS_root/jobs/ansaxena/azureml/day1-experiment-hello_1621536376_585b456b/mounts/workspaceblobstore/azureml/day1-experiment-hello_1621536376_585b456b\\n2021-05-20 18:46:41,316|azureml.history._tracking.PythonWorkingDirectory.workingdir|DEBUG|[STOP]\\n2021-05-20 18:46:41,316|azureml.WorkingDirectoryCM|DEBUG|[STOP]\\n2021-05-20 18:46:41,316|azureml._SubmittedRun#day1-experiment-hello_1621536376_585b456b.RunHistoryFacade.MetricsClient.FlushingMetricsClient|DEBUG|[START]\\n2021-05-20 18:46:41,317|azureml._SubmittedRun#day1-experiment-hello_1621536376_585b456b.RunHistoryFacade.MetricsClient|DEBUG|Overrides: Max batch size: 50, batch cushion: 5, Interval: 1.\\n2021-05-20 18:46:41,317|azureml._SubmittedRun#day1-experiment-hello_1621536376_585b456b.RunHistoryFacade.MetricsClient.PostMetricsBatch.PostMetricsBatchDaemon|DEBUG|Starting daemon and triggering first instance\\n2021-05-20 18:46:41,317|azureml._SubmittedRun#day1-experiment-hello_1621536376_585b456b.RunHistoryFacade.MetricsClient|DEBUG|Used for use_batch=True.\\n2021-05-20 18:46:41,317|azureml._SubmittedRun#day1-experiment-hello_1621536376_585b456b.RunHistoryFacade.MetricsClient.PostMetricsBatch.WaitFlushSource:MetricsClient|DEBUG|[START]\\n2021-05-20 18:46:41,317|azureml._SubmittedRun#day1-experiment-hello_1621536376_585b456b.RunHistoryFacade.MetricsClient.PostMetricsBatch.WaitFlushSource:MetricsClient|DEBUG|flush timeout 120 is different from task queue timeout 120, using flush timeout\\n2021-05-20 18:46:41,318|azureml._SubmittedRun#day1-experiment-hello_1621536376_585b456b.RunHistoryFacade.MetricsClient.PostMetricsBatch.WaitFlushSource:MetricsClient|DEBUG|Waiting 120 seconds on tasks: [].\\n2021-05-20 18:46:41,318|azureml._SubmittedRun#day1-experiment-hello_1621536376_585b456b.RunHistoryFacade.MetricsClient.PostMetricsBatch|DEBUG|\\n2021-05-20 18:46:41,318|azureml._SubmittedRun#day1-experiment-hello_1621536376_585b456b.RunHistoryFacade.MetricsClient.PostMetricsBatch.WaitFlushSource:MetricsClient|DEBUG|[STOP]\\n2021-05-20 18:46:41,318|azureml._SubmittedRun#day1-experiment-hello_1621536376_585b456b.RunHistoryFacade.MetricsClient|DEBUG|Overrides: Max batch size: 50, batch cushion: 5, Interval: 1.\\n2021-05-20 18:46:41,318|azureml._SubmittedRun#day1-experiment-hello_1621536376_585b456b.RunHistoryFacade.MetricsClient.PostMetricsBatchV2.PostMetricsBatchV2Daemon|DEBUG|Starting daemon and triggering first instance\\n2021-05-20 18:46:41,318|azureml._SubmittedRun#day1-experiment-hello_1621536376_585b456b.RunHistoryFacade.MetricsClient|DEBUG|Used for use_batch=True.\\n2021-05-20 18:46:41,318|azureml._SubmittedRun#day1-experiment-hello_1621536376_585b456b.RunHistoryFacade.MetricsClient.PostMetricsBatchV2.WaitFlushSource:MetricsClient|DEBUG|[START]\\n2021-05-20 18:46:41,318|azureml._SubmittedRun#day1-experiment-hello_1621536376_585b456b.RunHistoryFacade.MetricsClient.PostMetricsBatchV2.WaitFlushSource:MetricsClient|DEBUG|flush timeout 120 is different from task queue timeout 120, using flush timeout\\n2021-05-20 18:46:41,318|azureml._SubmittedRun#day1-experiment-hello_1621536376_585b456b.RunHistoryFacade.MetricsClient.PostMetricsBatchV2.WaitFlushSource:MetricsClient|DEBUG|Waiting 120 seconds on tasks: [].\\n2021-05-20 18:46:41,319|azureml._SubmittedRun#day1-experiment-hello_1621536376_585b456b.RunHistoryFacade.MetricsClient.PostMetricsBatchV2|DEBUG|\\n2021-05-20 18:46:41,319|azureml._SubmittedRun#day1-experiment-hello_1621536376_585b456b.RunHistoryFacade.MetricsClient.PostMetricsBatchV2.WaitFlushSource:MetricsClient|DEBUG|[STOP]\\n2021-05-20 18:46:41,319|azureml._SubmittedRun#day1-experiment-hello_1621536376_585b456b.RunHistoryFacade.MetricsClient.FlushingMetricsClient|DEBUG|[STOP]\\n2021-05-20 18:46:41,319|azureml._SubmittedRun#day1-experiment-hello_1621536376_585b456b.RunHistoryFacade.MetricsClient.wait_on_ingest-async:False|DEBUG|[START]\\n2021-05-20 18:46:41,319|azureml._SubmittedRun#day1-experiment-hello_1621536376_585b456b.RunHistoryFacade.MetricsClient|DEBUG|ClientBase: Calling wait_on_ingest with url /history/v1.0/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.MachineLearningServices/workspaces/{workspaceName}/experiments/{experimentName}/runs/{runId}/metricsingest/wait\\n2021-05-20 18:46:41,377|azureml._SubmittedRun#day1-experiment-hello_1621536376_585b456b.RunHistoryFacade.MetricsClient.wait_on_ingest-async:False|DEBUG|[STOP]\\n2021-05-20 18:46:41,378|azureml._SubmittedRun#day1-experiment-hello_1621536376_585b456b.RunHistoryFacade.MetricsClient|DEBUG|[STOP]\\n2021-05-20 18:46:41,378|azureml.MetricsClient|DEBUG|[STOP]\\n2021-05-20 18:46:41,379|azureml._SubmittedRun#day1-experiment-hello_1621536376_585b456b.RunHistoryFacade.MetricsClient.FlushingMetricsClient|DEBUG|[START]\\n2021-05-20 18:46:41,379|azureml._SubmittedRun#day1-experiment-hello_1621536376_585b456b.RunHistoryFacade.MetricsClient.PostMetricsBatch.WaitFlushSource:MetricsClient|DEBUG|[START]\\n2021-05-20 18:46:41,379|azureml._SubmittedRun#day1-experiment-hello_1621536376_585b456b.RunHistoryFacade.MetricsClient.PostMetricsBatch.WaitFlushSource:MetricsClient|DEBUG|flush timeout 300 is different from task queue timeout 120, using flush timeout\\n2021-05-20 18:46:41,379|azureml._SubmittedRun#day1-experiment-hello_1621536376_585b456b.RunHistoryFacade.MetricsClient.PostMetricsBatch.WaitFlushSource:MetricsClient|DEBUG|Waiting 300 seconds on tasks: [].\\n2021-05-20 18:46:41,379|azureml._SubmittedRun#day1-experiment-hello_1621536376_585b456b.RunHistoryFacade.MetricsClient.PostMetricsBatch|DEBUG|\\n2021-05-20 18:46:41,379|azureml._SubmittedRun#day1-experiment-hello_1621536376_585b456b.RunHistoryFacade.MetricsClient.PostMetricsBatch.WaitFlushSource:MetricsClient|DEBUG|[STOP]\\n2021-05-20 18:46:41,379|azureml._SubmittedRun#day1-experiment-hello_1621536376_585b456b.RunHistoryFacade.MetricsClient.PostMetricsBatchV2.WaitFlushSource:MetricsClient|DEBUG|[START]\\n2021-05-20 18:46:41,379|azureml._SubmittedRun#day1-experiment-hello_1621536376_585b456b.RunHistoryFacade.MetricsClient.PostMetricsBatchV2.WaitFlushSource:MetricsClient|DEBUG|flush timeout 300 is different from task queue timeout 120, using flush timeout\\n2021-05-20 18:46:41,380|azureml._SubmittedRun#day1-experiment-hello_1621536376_585b456b.RunHistoryFacade.MetricsClient.PostMetricsBatchV2.WaitFlushSource:MetricsClient|DEBUG|Waiting 300 seconds on tasks: [].\\n2021-05-20 18:46:41,380|azureml._SubmittedRun#day1-experiment-hello_1621536376_585b456b.RunHistoryFacade.MetricsClient.PostMetricsBatchV2|DEBUG|\\n2021-05-20 18:46:41,380|azureml._SubmittedRun#day1-experiment-hello_1621536376_585b456b.RunHistoryFacade.MetricsClient.PostMetricsBatchV2.WaitFlushSource:MetricsClient|DEBUG|[STOP]\\n2021-05-20 18:46:41,380|azureml._SubmittedRun#day1-experiment-hello_1621536376_585b456b.RunHistoryFacade.MetricsClient.FlushingMetricsClient|DEBUG|[STOP]\\n2021-05-20 18:46:41,380|azureml._SubmittedRun#day1-experiment-hello_1621536376_585b456b.RunHistoryFacade.MetricsClient.wait_on_ingest-async:False|DEBUG|[START]\\n2021-05-20 18:46:41,380|azureml._SubmittedRun#day1-experiment-hello_1621536376_585b456b.RunHistoryFacade.MetricsClient|DEBUG|ClientBase: Calling wait_on_ingest with url /history/v1.0/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.MachineLearningServices/workspaces/{workspaceName}/experiments/{experimentName}/runs/{runId}/metricsingest/wait\\n2021-05-20 18:46:41,451|azureml._SubmittedRun#day1-experiment-hello_1621536376_585b456b.RunHistoryFacade.MetricsClient.wait_on_ingest-async:False|DEBUG|[STOP]\\n2021-05-20 18:46:41,452|azureml.RunStatusContext|DEBUG|[STOP]\\n2021-05-20 18:46:41,452|azureml._SubmittedRun#day1-experiment-hello_1621536376_585b456b.RunHistoryFacade.MetricsClient.FlushingMetricsClient|DEBUG|[START]\\n2021-05-20 18:46:41,452|azureml._SubmittedRun#day1-experiment-hello_1621536376_585b456b.RunHistoryFacade.MetricsClient.PostMetricsBatch.WaitFlushSource:MetricsClient|DEBUG|[START]\\n2021-05-20 18:46:41,452|azureml._SubmittedRun#day1-experiment-hello_1621536376_585b456b.RunHistoryFacade.MetricsClient.PostMetricsBatch.WaitFlushSource:MetricsClient|DEBUG|flush timeout 900.0 is different from task queue timeout 120, using flush timeout\\n2021-05-20 18:46:41,452|azureml._SubmittedRun#day1-experiment-hello_1621536376_585b456b.RunHistoryFacade.MetricsClient.PostMetricsBatch.WaitFlushSource:MetricsClient|DEBUG|Waiting 900.0 seconds on tasks: [].\\n2021-05-20 18:46:41,452|azureml._SubmittedRun#day1-experiment-hello_1621536376_585b456b.RunHistoryFacade.MetricsClient.PostMetricsBatch|DEBUG|\\n2021-05-20 18:46:41,453|azureml._SubmittedRun#day1-experiment-hello_1621536376_585b456b.RunHistoryFacade.MetricsClient.PostMetricsBatch.WaitFlushSource:MetricsClient|DEBUG|[STOP]\\n2021-05-20 18:46:41,453|azureml._SubmittedRun#day1-experiment-hello_1621536376_585b456b.RunHistoryFacade.MetricsClient.PostMetricsBatchV2.WaitFlushSource:MetricsClient|DEBUG|[START]\\n2021-05-20 18:46:41,453|azureml._SubmittedRun#day1-experiment-hello_1621536376_585b456b.RunHistoryFacade.MetricsClient.PostMetricsBatchV2.WaitFlushSource:MetricsClient|DEBUG|flush timeout 900.0 is different from task queue timeout 120, using flush timeout\\n2021-05-20 18:46:41,453|azureml._SubmittedRun#day1-experiment-hello_1621536376_585b456b.RunHistoryFacade.MetricsClient.PostMetricsBatchV2.WaitFlushSource:MetricsClient|DEBUG|Waiting 900.0 seconds on tasks: [].\\n2021-05-20 18:46:41,453|azureml._SubmittedRun#day1-experiment-hello_1621536376_585b456b.RunHistoryFacade.MetricsClient.PostMetricsBatchV2|DEBUG|\\n2021-05-20 18:46:41,453|azureml._SubmittedRun#day1-experiment-hello_1621536376_585b456b.RunHistoryFacade.MetricsClient.PostMetricsBatchV2.WaitFlushSource:MetricsClient|DEBUG|[STOP]\\n2021-05-20 18:46:41,453|azureml._SubmittedRun#day1-experiment-hello_1621536376_585b456b.RunHistoryFacade.MetricsClient.FlushingMetricsClient|DEBUG|[STOP]\\n2021-05-20 18:46:41,453|azureml._SubmittedRun#day1-experiment-hello_1621536376_585b456b.RunHistoryFacade.MetricsClient.wait_on_ingest-async:False|DEBUG|[START]\\n2021-05-20 18:46:41,453|azureml._SubmittedRun#day1-experiment-hello_1621536376_585b456b.RunHistoryFacade.MetricsClient|DEBUG|ClientBase: Calling wait_on_ingest with url /history/v1.0/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.MachineLearningServices/workspaces/{workspaceName}/experiments/{experimentName}/runs/{runId}/metricsingest/wait\\n2021-05-20 18:46:41,511|azureml._SubmittedRun#day1-experiment-hello_1621536376_585b456b.RunHistoryFacade.MetricsClient.wait_on_ingest-async:False|DEBUG|[STOP]\\n2021-05-20 18:46:41,511|azureml.SendRunKillSignal|DEBUG|[STOP]\\n2021-05-20 18:46:41,511|azureml.HistoryTrackingWorkerPool.WorkerPoolShutdown|DEBUG|[START]\\n2021-05-20 18:46:41,511|azureml.HistoryTrackingWorkerPool.WorkerPoolShutdown|DEBUG|[STOP]\\n2021-05-20 18:46:41,512|azureml.WorkerPool|DEBUG|[STOP]\\n\\nRun is completed.\", \"graph\": {}, \"widget_settings\": {\"childWidgetDisplay\": \"popup\", \"send_telemetry\": false, \"log_level\": \"INFO\", \"sdk_version\": \"1.19.0\"}, \"loading\": false}" }, "metadata": {}, "output_type": "display_data" }, { "name": "stderr", "output_type": "stream", "text": [ "Class SynapseCompute: This is an experimental class, and may change at any time.
For more information, see https://aka.ms/azuremlexperimental.\n", "Class SynapseCompute: This is an experimental class, and may change at any time.
For more information, see https://aka.ms/azuremlexperimental.\n", "Class SynapseCompute: This is an experimental class, and may change at any time.
For more information, see https://aka.ms/azuremlexperimental.\n", "Class SynapseCompute: This is an experimental class, and may change at any time.
For more information, see https://aka.ms/azuremlexperimental.\n", "Class SynapseCompute: This is an experimental class, and may change at any time.
For more information, see https://aka.ms/azuremlexperimental.\n", "Class SynapseCompute: This is an experimental class, and may change at any time.
For more information, see https://aka.ms/azuremlexperimental.\n", "Class SynapseCompute: This is an experimental class, and may change at any time.
For more information, see https://aka.ms/azuremlexperimental.\n", "Class SynapseCompute: This is an experimental class, and may change at any time.
For more information, see https://aka.ms/azuremlexperimental.\n", "Class SynapseCompute: This is an experimental class, and may change at any time.
For more information, see https://aka.ms/azuremlexperimental.\n" ] } ], "source": [ "from azureml.core import Workspace, Experiment, ScriptRunConfig\n", "from azureml.widgets import RunDetails\n", "\n", "ws = Workspace.from_config()\n", "experiment = Experiment(workspace=ws, name='day1-experiment-hello')\n", "\n", "config = ScriptRunConfig(source_directory='./code/hello', script='hello.py', compute_target='ansaxenatestcompute')\n", "\n", "run = experiment.submit(config)\n", "RunDetails(run).show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Understanding the control code\n", "\n", "| Code |Description | \n", "|---|---|\n", "| `ws = Workspace.from_config()` | [Workspace](https://docs.microsoft.com/python/api/azureml-core/azureml.core.workspace.workspace?view=azure-ml-py&preserve-view=true) connects to your Azure Machine Learning workspace, so that you can communicate with your Azure Machine Learning resources. |\n", "| `experiment = Experiment( ... )` | [Experiment](https://docs.microsoft.com/python/api/azureml-core/azureml.core.experiment.experiment?view=azure-ml-py&preserve-view=true) provides a simple way to organize multiple runs under a single name.
Later you can see how experiments make it easy to compare metrics between dozens of runs. |\n", "| `config = ScriptRunConfig( ... )` | [ScriptRunConfig](https://docs.microsoft.com/python/api/azureml-core/azureml.core.scriptrunconfig?view=azure-ml-py&preserve-view=true) wraps your `hello.py` code and passes it to your workspace.
As the name suggests, you can use this class to _configure_ how you want your _script_ to _run_ in Azure Machine Learning.
Also specifies what compute target the script will run on.
In this code, the target is the compute cluster you created in the [setup tutorial](tutorial-1st-experiment-sdk-setup-local.md). |\n", "| `run = experiment.submit(config)` | Submits your script. This submission is called a [Run](https://docs.microsoft.com/python/api/azureml-core/azureml.core.run(class)?view=azure-ml-py&preserve-view=true).
A run encapsulates a single execution of your code. Use a run to monitor the script progress, capture the output,
analyze the results, visualize metrics and more. |\n", "| `aml_url = run.get_portal_url()` | The `run` object provides a handle on the execution of your code. Monitor its progress from
the Azure Machine Learning Studio with the URL that is printed from the python script. |\n", "|`RunDetails(run).show()` | There is an Azure Machine Learning widget that shows the progress of your job along with streaming the log files.\n", "\n", "## View the logs\n", "\n", "The widget has a dropdown box titled **Output logs** select `70_driver_log.txt`, which shows the following standard output: \n", "\n", "```\n", " 1: [2020-08-04T22:15:44.407305] Entering context manager injector.\n", " 2: [context_manager_injector.py] Command line Options: Namespace(inject=['ProjectPythonPath:context_managers.ProjectPythonPath', 'RunHistory:context_managers.RunHistory', 'TrackUserError:context_managers.TrackUserError', 'UserExceptions:context_managers.UserExceptions'], invocation=['hello.py'])\n", " 3: Starting the daemon thread to refresh tokens in background for process with pid = 31263\n", " 4: Entering Run History Context Manager.\n", " 5: Preparing to call script [ hello.py ] with arguments: []\n", " 6: After variable expansion, calling script [ hello.py ] with arguments: []\n", " 7:\n", " 8: Hello world!\n", " 9: Starting the daemon thread to refresh tokens in background for process with pid = 31263\n", "10:\n", "11:\n", "12: The experiment completed successfully. Finalizing run...\n", "13: Logging experiment finalizing status in history service.\n", "14: [2020-08-04T22:15:46.541334] TimeoutHandler __init__\n", "15: [2020-08-04T22:15:46.541396] TimeoutHandler __enter__\n", "16: Cleaning up all outstanding Run operations, waiting 300.0 seconds\n", "17: 1 items cleaning up...\n", "18: Cleanup took 0.1812913417816162 seconds\n", "19: [2020-08-04T22:15:47.040203] TimeoutHandler __exit__\n", "```\n", "\n", "On line 8 above, you see the \"Hello world!\" output. The 70_driver_log.txt file contains the standard output from run and can be useful when debugging remote runs in the cloud. You can also view the run by clicking on the **Click here to see the run in Azure Machine Learning studio** link in the wdiget.\n", "\n", "## Next steps\n", "\n", "In this tutorial, you took a simple \"hello world\" script and ran it on Azure. You saw how to connect to your Azure Machine Learning workspace, create an Experiment, and submit your `hello.py` code to the cloud.\n", "\n", "In the [next tutorial](day1-part3-train-model.ipynb), you build on these learnings by running something more interesting than `print(\"Hello world!\")`.\n" ] } ], "metadata": { "authors": [ { "name": "samkemp" } ], "categories": [ "tutorials", "get-started-day1" ], "celltoolbar": "Edit Metadata", "kernel_info": { "name": "python3-azureml" }, "kernelspec": { "display_name": "Python 3.6 - AzureML", "language": "python", "name": "python3-azureml" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.9" }, "microsoft": { "host": { "AzureML": { "notebookHasBeenCompleted": true } } }, "notice": "Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT License.", "nteract": { "version": "nteract-front-end@1.0.0" } }, "nbformat": 4, "nbformat_minor": 4 }
DonJayamanne commented 3 years ago

@ansaxena1 Please could you confirm you ran this locally in classic Jupyter. Looks like you ran this in Azure Notebooks. Please could you confirm, I'd like the output when you run this notebook in Jupyter Notebooks (classic) or Jupyter Labs, & please confirm it works in there (not VS Code, & not Azure Notebooks or other front end).

ansaxena1 commented 3 years ago

I did run this notebook in classic Jupyter (accessed through AML to use my compute instance) and this is the output after running there. Is there something else you'd like me to try running?

DonJayamanne commented 3 years ago

@ansaxena1

Also, is it possible to provide some instructions on how to setup the the Python environment to test this? I'm looking for steps like:

FYI - I'll be looking into this issue & will try to get this resolved for the next release. Thanks

ansaxena1 commented 3 years ago
DonJayamanne commented 3 years ago

I've updated the title, this works when testing locally heres the evidence. So it seems its only a problem when using remote Jupyter.

Screen Shot 2021-07-21 at 11 50 59
DonJayamanne commented 3 years ago

Works when I launch VS Code from remote Azure ML Compute as well

here's the evidence: Screen Shot 2021-07-21 at 12 40 46

DonJayamanne commented 3 years ago

Instructions to setup & test AML widgets locally/remote

https://github.com/microsoft/vscode-jupyter/wiki/How-to-test-Azure-AML-Widgets

DonJayamanne commented 3 years ago

Problem was, the output renderer selected was text/plain, vs code rememnbers this, hence widgets were never rendered.