PrefectHQ / prefect

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
https://prefect.io
Apache License 2.0
16.26k stars 1.58k forks source link

run_shell_process should support passing the current working directory or environment #15816

Open Jared314 opened 3 hours ago

Jared314 commented 3 hours ago

Describe the current behavior

The run_shell_process flow currently assumes it is running from within the current working directory and with the default environment variables. When the run_shell_process flow is either run outside the target working directory, or as a child flow, those values cannot be passed as arguments.

Describe the proposed behavior

The run_shell_process flow should accept additional optional arguments for cwd and env that are passed to the subprocess.Popen call.

@flow
def run_shell_process(
    command: str,
    log_output: bool = True,
    stream_stdout: bool = False,
    log_stderr: bool = False,
    cwd: str = None,
    env: Dict[str, str] = None,
):

Example Use

run_shell_process(
    command="touch test.txt",
    cwd="/tmp/test",
    env=os.environ.copy(),
    stream_stdout=True,
    return_state=True,
)

Additional context

run_shell_process is being run as a child flow within a Prefect (v3) flow.

zzstoatzz commented 2 hours ago

hi @Jared314 - thanks for the issue! the PR linked above should allow this - let me know if you have any concerns!