search

dagster-io / dagster

An orchestration platform for the development, production, and observation of data assets.
https://dagster.io
Apache License 2.0
11.17k stars 1.4k forks source link

Stream k8s_job_op logs to Dagit UI while job is running #9439

Open williamho123 opened 2 years ago

williamho123 commented 2 years ago

What's the use case?

Using the k8s_job_op, logs streamed from the job are only available in the compute logs section of Dagit after the entire job has finished running. Users only see this blank screen during job execution:

Screen Shot 2022-08-19 at 10 42 30 AM

For long-running jobs, it would be nice to see the progress/status of the job live rather than waiting until completion.

Ideas of implementation

When logs from the child job of the k8s_job_op are streamed, we could utilize the op context to send event logs instead.

try:
    log_entry = next(log_stream)
    context.log.info(log_entry)
except StopIteration:
    break

With this approach though, I'm not sure there's an easy way to distinguish between error, info, debug, etc. logs.

Thoughts on this or an alternative that might be more suitable? Or if there's a way to see the raw compute logs live during execution that I might have missed?

Additional information

No response

Message from the maintainers

Impacted by this issue? Give it a 👍! We factor engagement into prioritization.

dpeng817 commented 2 years ago

The idea of retrieving the logs as event logs is interesting. Given that the job completion information is a critical part of the execution of the op, representing in structured event logs doesn't seem crazy to me. The volume might be too large though to be tenable long term. cc @prha for thoughts?