Missing error logs in flow run timeline from `KubernetesWorker`

[DGolubets]

Bug summary

My deployment executing in K8s fails with an error (inspected with kubectl logs):

ModuleNotFoundError: No module named 'polars'

But the flow logs in the UI don't have anything except:

Process for flow run 'banana-bug' exited with status code: 1

Version info (prefect version output)

Version:             3.0.4
API version:         0.8.4
Python version:      3.12.3
Git commit:          c068d7e2
Built:               Tue, Oct 1, 2024 11:54 AM
OS/Arch:             linux/x86_64
Profile:             ephemeral
Server type:         server
Pydantic version:    2.9.2
Integrations:
  prefect-kubernetes: 0.5.0
  prefect-aws:       0.5.0
  prefect-dask:      0.3.1

Additional context

No response

[zzstoatzz]

hi @DGolubets - on some level this is expected to me, i.e. your deployment's runtime is likely missing polars as a dependency, so your code failed at import time before a flow run logger could be setup.

is your expectation that this line

ModuleNotFoundError: No module named 'polars'

is surfaced by the worker in the flow run logs? I think that's a very reasonable enhancement request

[DGolubets]

Hi @zzstoatzz

Thanks for quick reply. I wrongly assumed that Prefect collects the logs from pods stdout. If it send them from Python, then this behavior is understandable indeed.

Though UX would definitely be better if Prefect could handle this case.

[zzstoatzz]

Though UX would definitely be better if Prefect could handle this case.

Agreed! I will mark this as an enhancement