2 Orchestration with Prefect

[K0nkere]

Under the conda mlops-project-env pip install prefect

Copy notebook full-pipeline-movel.ipynb to the new folder 2-orchestration-prefect as prefect-model.ipynb and it starts to be our main file.

[K0nkere]

Launching Prefect

under the conda environment pip install prefect

Locally

prefect orion start

Launching the server

On the server its need to allowed connection on 0.0.0.0 in security group policy after that under conda environment prefect config set PREFECT_ORION_UI_API_URL="http://<external-ip>:4200/api" On the local machine, configure the API: prefect config set PREFECT_API_URL="http://<external-ip>:4200/api" Checking the config prefect config view and starting server prefect orion start --host 0.0.0.0

So we can get access to Prefect UI with public_server_address:4200

[K0nkere]

Deployment of Prefect flow

Build a deployment¶

To create a deployment from an existing flow script, there are just a few steps:

Use the prefect deployment build Prefect CLI command to build a deployment manifest and a deployment YAML file. These describe the files and settings needed to create your deployment. If you needed to add customizations to the manifest or the deployment YAML file, you would do so now.

Use the prefect deployment apply Prefect CLI command to create the deployment on the API based on the settings in the deployment manifest and deployment YAML.

What you need for the first step, building the deployment artifacts, is:

• The path and filename of the flow script
• The name of the flow function that is the entrypoint to the flow
• A name for the deployment

So to build deployment files for leo_flow.py, we'll use the command: prefect deployment build prefect-model.py:main -n project-test -q test format prefect deployment build <script_filename.py>:<entrypoint_flow> -n <deployment_name> -q <queue_name - for prefect agent> -n project-test name of the deployment in Prefect UI -q test name of work queue

Note that our flow needs a name parameter, but we didn't specify one when building the deployment files. Open the main-deployment.yaml file and add the parameter as parameters: {'name':project-test-depl'} !!! So we can pass parameters for a flow !!!

prefect deployment apply <entrypoint_flow>-deployment.yaml

Deployment in code

from prefect.orion.schemas.schedules import CronSchedule

deployment = Deployment.build_from_flow(
    flow=script_flow,
    name="<deployment_name>",
    schedule=CronSchedule(cron="0 6 1 * *", timezone="<optional>"),
    work_queue_name="<queue_name>",
)

After that can be used, in block section as example

if __name__ == "__main__":
   deployment.apply()

To demonstrate that your deployment exists, list all of the current deployments: prefect deployment ls To display details for a specific deployment prefect deployment inspect MLOps-project-test/project-test

[K0nkere]

We are still need two more items to run orchestrated deployments: an agent and a work queue In the Prefect UI, you can create a work queue by selecting the Work Queues page, then creating a new work queue. However, you don't need to manually create a work queue because it was created automatically when you created your deployment. If you hadn't created your deployment yet, it would be created when you start your agent.

In your terminal specifying queue name prefect agent start -q queue_name

[K0nkere]

Run a deployment from the UI

In the Prefect UI, select the Deployments page. You'll see a list of all deployments that have been created in this Prefect Orion instance. Select MLOps-project-test / project-test and press RUN button

Run from CLI

prefect deployment run MLOps-project-test/project-test

[K0nkere]

Deployment build options Schedules for the deployment

.YAML config

Interval schedules

schedule:
  interval: 600
  timezone: America/Chicago 

Cron schedules

schedule:
  cron: 0 0 * * *
  timezone: America/Chicago

 ┌───────────── minute (0 - 59)
 │ ┌───────────── hour (0 - 23)
 │ │ ┌───────────── day of the month (1 - 31)
 │ │ │ ┌───────────── month (1 - 12)
 │ │ │ │ ┌───────────── day of the week (0 - 6) (Sunday to Saturday;
 │ │ │ │ │                                   7 is also Sunday on some systems)
 *  *  * *  * <command to execute>

CLI schedules

When you build a deployment from the CLI you can use --cron, --interval --rrule flags to set a schedule: prefect deployment build <script_name.py>:<entrypoint_flow> -n <deployment_name> --cron "0 0 * * *"

[K0nkere]

Dockerized Prefect Orion

docker run -it --rm \
    -p 4200:4200 \
    prefecthq/prefect:2.0b5-python3.8 \
        prefect orion start --host=0.0.0.0