search

PrefectHQ / prefect

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
https://prefect.io
Apache License 2.0
15.82k stars 1.55k forks source link

config in project directory overrides config in user home directory #15044

Open LarsParTec opened 3 weeks ago

LarsParTec commented 3 weeks ago

Describe the current behavior

Currently, the Prefect configuration is stored in the user directory ~/.prefect/profiles.toml.

Describe the proposed behavior

Configurations in a project directory ~/myproject/.prefect/profiles.toml should automatically (!) override configs in ~/.prefect/profiles.toml when running a project flow ~/myproject/myflow.py. No PREFECT_HOME is required to be set.

Example Use

.
├── .prefect
│   └── profile.toml
└── myproject
    ├── .prefect
    │   └── profile.toml
    └── myflow.py

Additional context

~/myproject/.prefect/profiles.toml can be stored in a global project git repo, while personalisation remains possible in ~/.prefect/profiles.toml.

cleung1996 commented 3 weeks ago

Have this issue too. I have multiple prefect instances in one machine and would prefer having one .toml AND one prefect.db per virtual env (prefect instance), so that my flow runs per instance are not mixed up in one prefect.db, and I dont need to switch between .toml profiles everytime I switch prefect instances.

LarsParTec commented 2 weeks ago

@cleung1996 Good point. That is a further argument for nested Prefect configs.