PrefectHQ / prefect

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
https://prefect.io
Apache License 2.0
15.8k stars 1.55k forks source link

Injectable services #7064

Open thomafred opened 1 year ago

thomafred commented 1 year ago

First check

Prefect Version

2.x

Describe the current behavior

Server Loopservices are currently hardcoded for when the FastAPI starts: https://github.com/PrefectHQ/prefect/blob/main/src/prefect/orion/api/server.py#L361

Describe the proposed behavior

User should be able to create custom services and inject them using a setting/ENV-variable.

Create a new setting PREFECT_ORION_SERVICES_EXTRA which is used when when initializing the services with the Orion server. The setting should hold a comma-separated list of class-references for classes based on LoopService.

Example Use

export PREFECT_ORION_SERVICES_EXTRA="some.path.MyCustomService,some.other.path.MyOtherService"
prefect orion start
# Services are initialized on app init

Example service:

class MyCustomService(LoopService):
  """A super custom service that prints a message with every loop"""

  loop_seconds: int = 60

  async def run_once(self) -> None:
    print("This is a very custom service")

Additional context

Custom services are very useful when users have specific needs. They can also be used to create and distribute optional services, similar to how blocks and and task-runners are distributed.

Exampel services could be:

zanieb commented 1 year ago

👍 Makes sense to me! We are unlikely to work on this in the near future but I'd review a contribution.