Open K0nkere opened 2 years ago
under the conda environment
pip install prefect
prefect orion start
On the server its need to allowed connection on 0.0.0.0 in security group policy after that under conda environment
prefect config set PREFECT_ORION_UI_API_URL="http://<external-ip>:4200/api"
On the local machine, configure the API:
prefect config set PREFECT_API_URL="http://<external-ip>:4200/api"
Checking the config
prefect config view
and starting server
prefect orion start --host 0.0.0.0
So we can get access to Prefect UI with
public_server_address:4200
Build a deployment¶
To create a deployment from an existing flow script, there are just a few steps:
Use the prefect deployment build
Prefect CLI command to build a deployment manifest and a deployment YAML file.
These describe the files and settings needed to create your deployment.
If you needed to add customizations to the manifest or the deployment YAML file, you would do so now.
Use the prefect deployment apply
Prefect CLI command to create the deployment on the API based on the settings in the deployment manifest and deployment YAML.
What you need for the first step, building the deployment artifacts, is:
So to build deployment files for leo_flow.py, we'll use the command:
prefect deployment build prefect-model.py:main -n project-test -q test
format
prefect deployment build <script_filename.py>:<entrypoint_flow> -n <deployment_name> -q <queue_name - for prefect agent>
-n project-test name of the deployment in Prefect UI
-q test name of work queue
Note that our flow needs a name parameter, but we didn't specify one when building the deployment files. Open the main-deployment.yaml file and add the parameter as parameters: {'name':project-test-depl'} !!! So we can pass parameters for a flow !!!
prefect deployment apply <entrypoint_flow>-deployment.yaml
from prefect.orion.schemas.schedules import CronSchedule
deployment = Deployment.build_from_flow(
flow=script_flow,
name="<deployment_name>",
schedule=CronSchedule(cron="0 6 1 * *", timezone="<optional>"),
work_queue_name="<queue_name>",
)
After that can be used, in block section as example
if __name__ == "__main__":
deployment.apply()
To demonstrate that your deployment exists, list all of the current deployments:
prefect deployment ls
To display details for a specific deployment
prefect deployment inspect MLOps-project-test/project-test
We are still need two more items to run orchestrated deployments: an agent and a work queue In the Prefect UI, you can create a work queue by selecting the Work Queues page, then creating a new work queue. However, you don't need to manually create a work queue because it was created automatically when you created your deployment. If you hadn't created your deployment yet, it would be created when you start your agent.
In your terminal specifying queue name
prefect agent start -q queue_name
In the Prefect UI, select the Deployments page. You'll see a list of all deployments that have been created in this Prefect Orion instance. Select MLOps-project-test / project-test and press RUN button
prefect deployment run MLOps-project-test/project-test
Deployment build options Schedules for the deployment
schedule:
interval: 600
timezone: America/Chicago
schedule:
cron: 0 0 * * *
timezone: America/Chicago
┌───────────── minute (0 - 59)
│ ┌───────────── hour (0 - 23)
│ │ ┌───────────── day of the month (1 - 31)
│ │ │ ┌───────────── month (1 - 12)
│ │ │ │ ┌───────────── day of the week (0 - 6) (Sunday to Saturday;
│ │ │ │ │ 7 is also Sunday on some systems)
* * * * * <command to execute>
When you build a deployment from the CLI you can use
--cron,
--interval
--rrule flags to set a schedule:
prefect deployment build <script_name.py>:<entrypoint_flow> -n <deployment_name> --cron "0 0 * * *"
docker run -it --rm \
-p 4200:4200 \
prefecthq/prefect:2.0b5-python3.8 \
prefect orion start --host=0.0.0.0
Under the conda mlops-project-env
pip install prefect
Copy notebook full-pipeline-movel.ipynb to the new folder 2-orchestration-prefect as prefect-model.ipynb and it starts to be our main file.