argoproj-labs / hera

Hera makes Python code easy to orchestrate on Argo Workflows through native Python integrations. It lets you construct and submit your Workflows entirely in Python. ⭐️ Remember to star!
https://hera.rtfd.io
Apache License 2.0
560 stars 105 forks source link

Add runner debug mode for extra logging #1164

Closed elliotgunton closed 3 weeks ago

elliotgunton commented 3 weeks ago

Is your feature request related to a problem? Please describe. I've run into a problem when trying to load an artifact from S3 and using the runner, but the Artifact path comes out as None:

FileNotFoundError: [Errno 2] No such file or directory: '/tmp/hera-inputs/artifacts/None'
@script(constructor="runner")
def load_and_split_dataset(
    dataset_path: Annotated[
        Path,
        S3Artifact(
            endpoint="minio:9000",
            bucket="my-bucket",
            key="dataset.csv",
            access_key_secret=m.SecretKeySelector(
                name="my-minio-cred",
                key="accesskey",
            ),
            secret_key_secret=m.SecretKeySelector(
                name="my-minio-cred",
                key="secretkey",
            ),
            insecure=True,
            loader=None,
        ),
    ],
):
    print(dataset_path)
    data = pd.read_csv(dataset_path)

Describe the solution you'd like A clear and concise description of what you want to happen.

I'd like some debug logs from within the runner loading stage which can be turned on via an environment variable on the script.

Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered. printing debug logs myself but they are after the fact when the runner has already tried to load the artifact.

Additional context Add any other context or screenshots about the feature request here.

The code works with an inline script. When using the runner the Artifact is retrieved as an input correctly, but seemingly not loaded by the runner:

image

elliotgunton commented 3 weeks ago

Dupe of https://github.com/argoproj-labs/hera/issues/972