search

kubeflow / pipelines

Machine Learning Pipelines for Kubeflow
https://www.kubeflow.org/docs/components/pipelines/
Apache License 2.0
3.63k stars 1.63k forks source link

[feature] display main.log as artifact for each step #10036

Open Linchin opened 1 year ago

Linchin commented 1 year ago

Feature Area

/area frontend /area backend

What feature would you like to see?

A feature similar to v1 behavior. In the details page of each step, the log appears as one of the artifacts that users can view: image

In v2, this main log artifact is no longer displayed. It would be great if we could add a similar section to show the logs, or use the log artifact as a source for the "logs" panel after the pod is deleted, instead of directly pulling from the (already deleted) pod.

What is the use case or pain point?

Because all the logs of completed pods are auto deleted by Kubernetes after 24 hours, the users can no longer access the logs of earlier pipeline runs.

Is there a workaround currently?

No workaround.

/cc @kromanow94

Love this idea? Give it a 👍.

droctothorpe commented 3 weeks ago

I started working on turning the outputs into proper artifacts. There are a lot of complex design obstacles. There's no simple way to do it that preserves backend agnosticism. We have to decide if we want to rely on AWF to archive the logs and just record the relationship between the execution and the log files in MLMD, which is extremely backend-specific, or just have the launcher capture and publish logs directly, which is kind of reinventing something that AWF already does, and also doesn't account for driver logs. Bandwidth permitting, I may author a design doc. Maybe we can discuss it in the community call.

kromanow94 commented 3 weeks ago

Amazing, thank you so much for your time and effort!