Dockerize Prefect Server, Agent and Proxy

d-jeph commented 1 month ago

Feature: Dockerize Prefect Server, Agent and Proxy

Issue

In the current setup it's not possible to run prefect, prefect agent and the proxy as docker containers. This PR adds the necessary docker files to enable running these services as containers.

Changes

Added the following files and folders:

A Docker folder with a docker_compose.yml file and env.template file. The docker_compose file has three services: prefect_proxy,server, and agent
A Dockerfile for the proxy python application
The env.template file contains the necessary environment variables for the docker_compose file
A docker_compose.dev.yml for running the services locally during development and testing
A docker.yml that contains a Github actions workflow for pushing the docker images to docker hub

codecov-commenter commented 1 month ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 77.00%. Comparing base (b246e24) to head (b6b5190).

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #128 +/- ## ========================================== - Coverage 77.05% 77.00% -0.05% ========================================== Files 2 2 Lines 924 922 -2 ========================================== - Hits 712 710 -2 Misses 212 212 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Ishankoradia commented 1 month ago

Hi @d-jeph , thank you submitting this PR. Its super helpful.

For prefect, we have upgraded from agents to workers. This issue has the details Can we incorporate this in the docker-compose.yml ?

Also will we be able to start mutliple workers in the current setup ? On production we use 3 - 2 workers execute work from ddp queue and 1 from manual-dbt queue.

d-jeph commented 1 month ago

Hi @d-jeph , thank you submitting this PR. Its super helpful.

For prefect, we have upgraded from agents to workers. This issue has the details Can we incorporate this in the docker-compose.yml ?

Also will we be able to start mutliple workers in the current setup ? On production we use 3 - 2 workers execute work from ddp queue and 1 from manual-dbt queue.

@Ishankoradia Thanks. I will update the code to use workers and also the multiple workers

d-jeph commented 1 month ago

Hi @d-jeph , thank you submitting this PR. Its super helpful. For prefect, we have upgraded from agents to workers. This issue has the details Can we incorporate this in the docker-compose.yml ? Also will we be able to start mutliple workers in the current setup ? On production we use 3 - 2 workers execute work from ddp queue and 1 from manual-dbt queue.

@Ishankoradia Thanks. I will update the code to use workers and also the multiple workers

@Ishankoradia I have pushed the updated docker-compose.yml. I could not find a straight forward way for making the number of workers dynamic so for now I have three different services, one for each worker

d-jeph commented 1 month ago

@fatchat I added a github actions workflow change to this PR as well.

Ishankoradia commented 1 month ago

Thanks @d-jeph i will take a look at this and merge it before the weekend.

Ishankoradia commented 1 month ago

Hi @d-jeph , the docker compose works very nicely. I am able to run proxy + prefect on my current local database via docker.

One thing i noticed, when i run an existing prefect deployment (from its UI, where prefect is running via docker). It throws this error. Although the flow is getting submitted to the container worker but looks like it is still trying to find the code (to be run) from the host. Wanted to check if you have faced this before, looks we might have to update the existing deployment to read from prefect-proxy's storage instead of host somehow.

d-jeph commented 4 weeks ago

@Ishankoradia Thanks for flagging this. Yes we experienced the same issue when we tested this on GCP. I have updated the docker compose files to add a shared volume between the proxy service(where the flow code lives) and the prefect worker containers. This should help prefect get access to the flow code. Please test and let me know if you get any errors.

Ishankoradia commented 3 weeks ago

Hey @d-jeph thanks for making the changes. This is what i found in testing experiment

I ran docker compose up and connected it to my local prefect that has whole lot of deployments setup
I tried running one of them , it gave the same error as above. To make it work i had to go into prefect's database, navigate to table depolyment and update the column path to /app. This is absolutely fine, we could run a script that does this for our current deployments if/when we move to dockerize production deployment.
After making the above change, i again ran the same deployment & this time i got the below error

What i suspect the problem is, we are using a new prefect image (a different env) for each worker and the prefect itself. The way we have setup in our deployment currently is - we run all workers & the prefect from the same environment as proxy (requirements.txt). We need to somehow make sure that the env for workers & prefect_server & proxy are same or atleast have same dependencies

d-jeph commented 3 weeks ago

@Ishankoradia Thanks for sharing this. I have now setup a new Docker file Dockerfile.prefect_server that creates a prefect server image that includes the same dependencies as the proxy service - this is the image used now in the docker compose for the prefect server and worker service. When I used the proxy requirements.txt I however got an error cannot import name 'safe _get_annotation' from 'griffe.agents-nodes' so I had to upgrade the griffe package to the newest version 0.47.0. Please let me know if you've encountered the error

I have also noticed the team upgraded prefect to a new version 2.18.3 so I upgraded the image used as well.

Please test and let me know if you experience any errors on your end.

DalgoT4D / prefect-proxy