ValueError: `PREFECT_API_URL` must be set to start a Worker.

Ulthran commented 2 weeks ago

Bug summary

I'm trying to have prefect server and a local worker pool startup automatically with a RHEL server. I'm using supervisord to start both:

[program:prefect_server]
command=prefect server start
directory=/var/www/prefect
autostart=true
redirect_stderr=true          ; redirect proc stderr to stdout (default false)
stdout_logfile=/var/www/prefect/prefect.log        ; stdout log path, NONE for none; default AUTO
stdout_logfile_maxbytes=1MB   ; max # logfile bytes b4 rotation (default 50MB)
stdout_logfile_backups=10     ; # of stdout logfile backups (0 means none, default 10)
stdout_capture_maxbytes=1MB   ; number of bytes in 'capturemode' (default 0)
stdout_events_enabled=false   ; emit events on stdout writes (default false)
stdout_syslog=true

[program:prefect_local_worker_pool]
command=prefect config set PREFECT_API_URL="http://127.0.0.1/api" && prefect worker start --pool "LOCAL"
directory=/var/www/prefect
autostart=true
redirect_stderr=true          ; redirect proc stderr to stdout (default false)
stdout_logfile=/var/www/prefect/LOCAL.log        ; stdout log path, NONE for none; default AUTO
stdout_logfile_maxbytes=1MB   ; max # logfile bytes b4 rotation (default 50MB)
stdout_logfile_backups=10     ; # of stdout logfile backups (0 means none, default 10)
stdout_capture_maxbytes=1MB   ; number of bytes in 'capturemode' (default 0)
stdout_events_enabled=false   ; emit events on stdout writes (default false)
stdout_syslog=true
environment=PREFECT_API_URL="http://127.0.0.1/api"

This snippet from /etc/supervisord.conf shows the program definitions for both. The prefect server starts up as expected but then the worker pool keeps raising the below ValueError:

09:18:32.773 | INFO    | prefect - Starting server on http://127.0.0.1:8805
Discovered type 'process' for work pool 'LOCAL'.
Traceback (most recent call last):
  File "/home/bushmanc1/miniconda3/lib/python3.12/site-packages/prefect/cli/_utilities.py", line 42, in wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/bushmanc1/miniconda3/lib/python3.12/site-packages/prefect/cli/_types.py", line 153, in sync_fn
    return asyncio.run(async_fn(*args, **kwargs))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bushmanc1/miniconda3/lib/python3.12/asyncio/runners.py", line 194, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/home/bushmanc1/miniconda3/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bushmanc1/miniconda3/lib/python3.12/asyncio/base_events.py", line 687, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/home/bushmanc1/miniconda3/lib/python3.12/site-packages/prefect/cli/worker.py", line 167, in start
    await worker.start(
  File "/home/bushmanc1/miniconda3/lib/python3.12/site-packages/prefect/workers/process.py", line 176, in start
    async with self as worker:
  File "/home/bushmanc1/miniconda3/lib/python3.12/site-packages/prefect/workers/base.py", line 1135, in __aenter__
    await self.setup()
  File "/home/bushmanc1/miniconda3/lib/python3.12/site-packages/prefect/workers/base.py", line 606, in setup
    raise ValueError("`PREFECT_API_URL` must be set to start a Worker.")
ValueError: `PREFECT_API_URL` must be set to start a Worker.
An exception occurred.
09:18:42.087 | INFO    | prefect - Stopping server on http://127.0.0.1:8805

I've tried restarting it manually through supervisorctl (sudo supervisorctl restart prefect_local_worker_pool) and the same error occurs on each restart attempt. When I run prefect worker start --pool "LOCAL" in the terminal it works as expected. I've also tried other variations on the program definition shown above including 1) not defining PREFECT_API_URL at all, 2) defining it only with environment=..., and 3) exporting it directly export PREFECT_API_URL=... && prefect start ....

Any help is greatly appreciated!

Version info (`prefect version` output)

Version:             3.0.1
API version:         0.8.4
Python version:      3.12.4
Git commit:          c6b2ffe1
Built:               Fri, Sep 6, 2024 10:05 AM
OS/Arch:             linux/x86_64
Profile:             local
Server type:         server
Pydantic version:    2.9.0
Integrations:
  prefect-github:    0.3.0

Additional context

No response

desertaxle commented 2 weeks ago

Thanks for the bug report @Ulthran! I'd recommend using only the environment setting and make sure you include the port (4200 by default):

environment=PREFECT_API_URL="http://127.0.0.1:4200/api"

If that doesn't work, can you get the config out of your supervisord process with prefect config view and share it?

Ulthran commented 2 weeks ago

Thanks @desertaxle that worked! I do think I might've been messing up the supervisord updates also so it's possible some of the earlier combinations I mentioned would work too. First time with prefect and with supervisord.

Thanks for the quick reply and great software though!

PrefectHQ / prefect