search

PrefectHQ / prefect

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
https://prefect.io
Apache License 2.0
16.01k stars 1.57k forks source link

Improve exception message when deployment flow run is not found and API URl is unset #7518

Open kielnino opened 1 year ago

kielnino commented 1 year ago

First check

Prefect Version

2.x

Describe the current behavior

I've followed the Running flows with Docker tutorial, but missed the part with setting prefect config set PREFECT_API_URL=http://127.0.0.1:4200/api. This results in errors from the docker container like 

16:44:19.165 | ERROR   | prefect.engine - Engine execution of flow run 'effb09ca-2302-4476-b72c-1943107eaccb' exited with unexpected exception

After setting the PREFECT_API_URL it works flawlessly.

Describe the proposed behavior

I would like to get a more verbose error message like "You didn't set the PREFECT_API_URL, do you want to connect to the cloud or have you startet your orion server locally?".

Example Use

No response

Additional context

No response

zanieb commented 1 year ago

Hi! When PREFECT_API_URL is None the server is run locally in memory. The error I presume you're getting is that the flow run id cannot be found, since the in-memory server does not know about the flow run created by another server.

A user could be connecting to a database somewhere else and null API URL is a totally valid way to run flows where you have multiple ephemeral APIs talking to a single database. This makes adding a specific error message here difficult, but we can definitely add some sort of hint.

kielnino commented 1 year ago

@madkinsz Thank you for the quick reply. Then maybe I just didn't understand the whole concept. Following the tutorial I started a server (prefect orion start) in one terminal, an agent (prefect agent start -q test) in a second and then the flow (prefect deployment run log-flow/log-flow-docker) in a third terminal.

With the error message mentioned I had no idea what went wrong. So if I misunderstood the concept or the tutorial simply misled me, I'm sorry, but to better pick up beginners, more meaningful error messages would be very helpful.

By the way, I find prefect really a great project and would like to thank you for your commitment.

zanieb commented 1 year ago

👍 yeah so this is a little tricky: each of those ran a separate server since you hadn't set the API URL. Since they were all running on the same file system, they shared a SQLite database so everything looked like it was using a single server. When you started the flow run in a container, it also ran a copy of the server but it was using a separate file system so it used a new database and the flow run was missing.

There are a couple things we can do:

  1. We can detect that we are creating a new database and improve the missing flow run error message. Flow runs from deployments should always use an existing database.
  2. We should improve the clarity of how these ephemeral API servers work

Holistically, 2. is more important. We can tell if you have an API server running on localhost and should probably auto-set the API URL for you in some cases. It's a bit tricker to do well though, 1. will help in the meantime.

By the way, I find prefect really a great project and would like to thank you for your commitment.

Thanks! We appreciate hearing it :)