search

tektoncd / results

Long term storage of execution results.
Apache License 2.0
77 stars 74 forks source link

[integrations] Tekton Dashboard #82

Open wlynch opened 3 years ago

wlynch commented 3 years ago

Tracking issue for integrating with the Tekton Dashboard.

We'd like to surface Results/Records in the Tekton UI so that users can visualize the logic groupings / extra data Results can store. A really rough version of this can be found here: https://github.com/tektoncd/results/tree/main/tools/simpleui

I am not sure how we can do this in a minimally invasive way. Open to ideas!

Fabian-K commented 3 years ago

Unfortunately not an idea of how to go forward but want to say that this is really exciting. Due to relying on the Kubernetes APIs the dashboard really struggles when there are lots of resources. Implementing proper pagination is difficult and also filtering possibilities are limited. As a workaround, we try to keep the number of resources low (e.g. using https://github.com/tektoncd/experimental/tree/master/pipeline/cleanup#cleanup-old-taskruns-and-pipelineruns) but ideally we should be able to keep as much history for e.g. all executions of a pipeline as we want and not as technically limited ;)

Would be awesome if results is the solution!

wlynch commented 3 years ago

@Fabian-K Sorry I missed this! But yes, that's the goal for this project. :)

This issue is scoped purely for integrating with the Tekton Dashboard, but you'll probably find https://github.com/tektoncd/results/issues/59 useful to your use case!

wlynch commented 3 years ago

https://github.com/tektoncd/dashboard/blob/master/docs/extensions.md looks like a promising way to hook into the Dashboard.

AlanGreene commented 3 years ago

@wlynch I'd love to talk more about the desired user experience for viewing / interacting with the Results data. There are plenty of options to explore depending on the desired outcome.

At a high level I see 2 main options for the user:

There are some planned changes to the Dashboard's API / data layer that may make integration / customisation easier in future, and some other challenges such as auth that we'll need to address too.

ghost commented 3 years ago
  • alternatively/additionally, provide better integration into the existing UI, so for example we might populate the existing PipelineRun details page with data from the Results API if the resources are no longer available on the cluster

This seems like the ideal route to me. Results providing the historical record / source of truth for anything that's reached a Completed ( / deleted) state.

@wlynch do you think that's fair assessment?

Are the dashboard's data layer plans related to the redux removal issue or documented elsewhere?

AlanGreene commented 3 years ago

Are the dashboard's data layer plans related to the redux removal issue or documented elsewhere?

Yes that's the first part of it, a major simplification of the data layer (and corresponding reduction in boilerplate) so we can more easily plugin additional / alternative sources while solving some of the performance and caching issues we have today. I need to flesh out the plan a bit more and document it so we can start to make progress. I have my hands full for the next 1 - 2 weeks but will try to get something on paper asap for others to review.

wlynch commented 3 years ago
  • alternatively/additionally, provide better integration into the existing UI, so for example we might populate the existing PipelineRun details page with data from the Results API if the resources are no longer available on the cluster

This seems like the ideal route to me. Results providing the historical record / source of truth for anything that's reached a Completed ( / deleted) state.

@wlynch do you think that's fair assessment?

Yup! The Result API also captures data for unique instances of Runs with the same name (e.g. if you create->delete->create) which would probably also be useful for the dashboard.

Modeling the dashboard on the Result API would also let us display related data (e.g. PipelineRun + TaskRuns) in groups.

@AlanGreene I added a discussion item for the March 8th Dashboard WG so we can talk about questions/ideas!

tekton-robot commented 3 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale with a justification. Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with /close with a justification. If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle stale

Send feedback to tektoncd/plumbing.

pietervincken commented 3 years ago

Is this issue still actively being worked on? (I came here from the Dashboard project due to the slowness of the dashboard)

wlynch commented 3 years ago

@AlanGreene it's been awhile since we've touched base on this. Any updates on the dashboard side for how to get started experimenting?

tekton-robot commented 3 years ago

Stale issues rot after 30d of inactivity. Mark the issue as fresh with /remove-lifecycle rotten with a justification. Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with /close with a justification. If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle rotten

Send feedback to tektoncd/plumbing.

tekton-robot commented 3 years ago

Rotten issues close after 30d of inactivity. Reopen the issue with /reopen with a justification. Mark the issue as fresh with /remove-lifecycle rotten with a justification. If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/close

Send feedback to tektoncd/plumbing.

tekton-robot commented 3 years ago

@tekton-robot: Closing this issue.

In response to [this](https://github.com/tektoncd/results/issues/82#issuecomment-943856798): >Rotten issues close after 30d of inactivity. >Reopen the issue with `/reopen` with a justification. >Mark the issue as fresh with `/remove-lifecycle rotten` with a justification. >If this issue should be exempted, mark the issue as frozen with `/lifecycle frozen` with a justification. > >/close > >Send feedback to [tektoncd/plumbing](https://github.com/tektoncd/plumbing). Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
tekton-robot commented 3 years ago

@pietervincken: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to [this](https://github.com/tektoncd/results/issues/82#issuecomment-945806311): >/reopen > >This feature is still missing. IIRC this was the endgoal of this subproject? Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
pietervincken commented 3 years ago

Can we re-open this issue? We're running Tekton on a larger scale (2k+ pipelineruns, 20k+ taskruns active in the cluster) so we'd be very interested in having the ability to not rely on the Kubernetes API for looking into results of a pipeline. Currently we need to very aggressively clean the runs in order for the dashboard to stay performant.

wlynch commented 3 years ago

/lifecycle frozen

This is something we still want to get to, but need help from the dashboard team 🙏

drriguz commented 2 years ago

Hi, guys, any progress on the integration?

Currently it's not possible to view the build status of a project via dashboard like jenkins, and it's also not directly supported through the pipeline metrics(use grafana + prometheus). So I'm wondering if this could be achived by tekton results?

jb-2020 commented 2 years ago

alternatively/additionally, provide better integration into the existing UI, so for example we might populate the existing PipelineRun details page with data from the Results API if the resources are no longer available on the cluster

+1 to the second approach recommended by @AlanGreene - It would be great to leverage results api to make the PipelineRun link (e.g. namespaces/tekton-pipelines/pipelineruns/foo-123 permanent with minimal overhead in etcd as well as persist the logs as mentioned in: https://github.com/tektoncd/results/issues/198

These features would greatly improve the out of the box experience of Tekton IMO.

vdemeester commented 1 year ago

/area roadmap

xinnjie commented 1 year ago

/assign

xinnjie commented 1 year ago

@AlanGreene @adambkaplan I'd like to propose a TEP about Results visualization in Dashboard in a few day.

The general idea is: In PipelineRun/TaskRun list and detail view, first query Results API, if PipelineRun/TaskRun is not finished, fall back to Kubernetes API.

Benifits:

  1. Resource URL in deashboard will match Results API perfectly.
  2. If first query Kubernetes API and then query Results API, there will be be some pagination problems, because we need to take care of both API pagination.
  3. Compactbility: During development, we could set up 2 new tabs: PipelineRun results and TaskRun results for test. Modification to oringal PipelineRun/TaskRun tab would be as less as possible(Although there would be some refactors to make 2 sides use same impletation). When this feature is stable and Results API is enabled, replace original view with results view. If Results API is not enabled, all things go like noew.
sayan-biswas commented 1 year ago
  1. We need to modify Results Watcher reconsilation. Currently watcher not produce result until PipelineRuns/TaskRuns ended.

@xinnjie I guess watcher does create the PipelineRuns/TaskRuns entries in the database as soon as it's reconciled, irrespective of the status. If that is what you meant.

khrm commented 1 year ago

I created a basic dashboard based on results using Flutter. https://github.com/khrm/bowspirit Lot of things still need to be done on it. Atm, it has a list for Pipelinerun, also view Pipelinerun's Yaml and Log

xinnjie commented 1 year ago
  1. We need to modify Results Watcher reconsilation. Currently watcher not produce result until PipelineRuns/TaskRuns ended.

@xinnjie I guess watcher does create the PipelineRuns/TaskRuns entries in the database as soon as it's reconciled, irrespective of the status. If that is what you meant.

Yes, it is. It's my mistake here.

cmorinupgrade commented 9 months ago

Any progress on enabling the dashboard to fallback to Results API database?

@xinnjie did you end up implementing that TEP you were referring to? Thank you! 🙇

AlanGreene commented 9 months ago

@xinnjie Is this something you're still planning to work on or do you have some progress update you can share? If you're not planning to work on it that's fine, please unassign yourself so someone else can pick it up. Thanks.

avinal commented 9 months ago

It seems Xinnjie is not active on GitHub since October. it should be okay to unassign this issue.

GijsvanDulmen commented 8 months ago

This is really a missing feature in the complete Tekton ecosystem. Would easy up resources on the cluster as well.

xinnjie commented 8 months ago

Sorry guys, I'm not available for now.

Stringls commented 2 months ago

Any updates on that? It would be really helpful to save logs of PipelineRuns in a Tekton Dashboard

AlanGreene commented 2 months ago

@Stringls Can you elaborate on what you're looking to do? If it's just persisting logs and loading those in the Dashboard see https://github.com/tektoncd/dashboard/blob/main/docs/walkthrough/walkthrough-logs.md. It describes a way to achieve this today, using minio for storage as an example, but can be achieved using any storage provider you wish as long as you can provide the required API for the Dashboard to consume.

If you're looking to persist the Tekton resources too, and load both those and the logs in the Dashboard from Tekton Results, that is not yet supported. There are a number of blockers on the Results side that need to resolved first before the Dashboard could add this support, including https://github.com/tektoncd/results/issues/535

Stringls commented 2 months ago

@AlanGreene Sure. So as a user of Tekton I expect to:

As we know, we lose logs when pods of the TaskRun/PipelineRun are deleted, but I would like them to be persistent and not be deleted as Logs is a crucial part of any CI.

Setting up minio as a storage of the logs sounds useful, but not ideal. IMHO there should be a native solution in Tekton Results without any further configuration of other tools (e.g. minio). For example, if a user wants to have a local storage, Tekton Results monitors every TaskRun/PipelineRun and saves logs (stdout, stderr) in PVC/s and once Tekton Pruner deletes TaskRuns/PipelineRun Tekton Results deleted the logs associated with that Tekton resources in its storage.

We don't really look to persist the Tekton resources, currently.