Azure / MachineLearningNotebooks

Python notebooks with ML and deep learning examples with Azure Machine Learning Python SDK | Microsoft
https://docs.microsoft.com/azure/machine-learning/service/
MIT License
4.07k stars 2.52k forks source link

Pipelines created through code are not appearing in AML jobs UI. #1927

Closed DanielPerezJensen closed 1 year ago

DanielPerezJensen commented 1 year ago

I created some pipelines using the code provided in https://github.com/Azure/MachineLearningNotebooks/blob/master/how-to-use-azureml/machine-learning-pipelines/intro-to-pipelines/aml-pipelines-setup-schedule-for-a-published-pipeline.ipynb.

What I did was basically the following code and it ran without issue when connected to AML:

import azureml.core
from azureml.core import Workspace
from azureml.pipeline.core import Pipeline, PublishedPipeline
from azureml.core.experiment import Experiment
from azure.ai.ml import MLClient
from azureml.core import Dataset, Datastore, Model, Run

from demo_aml.deployment.aml_helper import (
    create_registered_env,
    get_credential,
    register_to_endpoint,
)
from demo_aml.deployment.pipeline_components import (
    build_promoting_pipeline,
    build_scoring_pipeline,
    build_training_pipeline,
)

from azureml.pipeline.core.schedule import ScheduleRecurrence, Schedule

from pathlib import Path

from demo_aml.deployment.config import Config

config = Config.from_yaml(
    str(Path(__file__).parent / "deployment/configs/dta.yml")
)  # Hardcoded as in both dta and prod yaml files these constants are the same.

ws = Workspace(
    config.SUBSCRIPTION_ID,
    config.RESOURCE_GROUP_NAME,
    config.WORKSPACE_NAME,
)

# Get all published pipeline objects in the workspace
all_pub_pipelines = PublishedPipeline.list(ws)

# We will iterate through the list of published pipelines and
# use the last ID in the list for Schelue operations:
print("Published pipelines found in the workspace:")
for pub_pipeline in all_pub_pipelines:
    print(pub_pipeline.name, pub_pipeline.id)
    pub_pipeline_id = pub_pipeline.id

print("Published pipeline id to be used for Schedule operations: {}".format(pub_pipeline_id))

recurrence = ScheduleRecurrence(
    frequency="Day", interval=2, hours=[22], minutes=[30]
)  # Runs every other day at 10:30pm

schedule = Schedule.create(
    workspace=ws,
    name="My_Schedule",
    pipeline_id=pub_pipeline_id,
    experiment_name="Schedule-run-sample",
    recurrence=recurrence,
    wait_for_provisioning=True,
    description="Schedule Run",
)

# You may want to make sure that the schedule is provisioned properly
# before making any further changes to the schedule

print("Created schedule with id: {}".format(schedule.id))

schedules = Schedule.list(ws, pipeline_id=pub_pipeline_id)

# We will iterate through the list of schedules and
# use the last recurrence schedule in the list for further operations:
print("Found these schedules for the pipeline id {}:".format(pub_pipeline_id))
for schedule in schedules:
    print(schedule.id)
    if schedule.recurrence is not None:
        schedule_id = schedule.id

print("Schedule id to be used for schedule operations: {}".format(schedule_id))

# Use active_only=False to get all schedules including disabled schedules
schedules = Schedule.list(ws, active_only=True)
print("Your workspace has the following schedules set up:")
for schedule in schedules:
    print(schedule.name)
    print(schedule.pipeline_id)

It runs without issue and after I get the output:

Your workspace has the following schedules set up:
My_Schedule
2bfc207e-6cef-416e-bd11-d7caca3d31c9
MyRecurringSchedule
4dd73fc7-fbcc-41e8-b70c-5147fe843d10

However, if I take a screenshot of the schedules in AML GUI I see the following: image

I only see the schedule I created manually through GUI earlier, this one also does not appear when connecting to code.

I am certain that I am connected to the right workspace, as I use the same config options when creating pipelines. And these do appear in AML gui. Any help would be appreciated :)

DanielPerezJensen commented 1 year ago

I figured it out myself, apparently my AML workspace is in Azure ML v2 Python SDK, and the code I was using was v1 SDK. Perhaps good that this would be indicated more clearly?