Open deepk2u opened 1 year ago
/assign @gkcalat
Hi @deepk2u! It may be due to insufficient resource provisioning or the lack of etc maintenance (see here). How longs did it take for you to reach these numbers?
It's an eks cluster. We connected with AWS support and they are maintaining the cluster and running defragmentation and doing all kinds of maintenance for etcd.
the oldest object I have on the list is from 14th July 2022.
Can check how large are your pipeline manifests used in these recurring runs?
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
This issue has been automatically closed because it has not had recent activity. Please comment "/reopen" to reopen it.
/reopen
@kuldeepjain: You can't reopen an issue/PR unless you authored it or you are a collaborator.
/reopen
@deepk2u: Reopened this issue.
Closing this issue. No activity for more than a year.
/close
@rimolive: Closing this issue.
/reopen
We found this issue in KFP 2.0.5. We'll work on a pruning mechanism for pipeline run k8s objects.
@rimolive: Reopened this issue.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Environment
Steps to reproduce
We have around 125 recurring runs within a single namespace. After a few months of historical runs, we have started seeing performance issues in the k8s cluster.
After digging deeper, we found that we are seeing timeouts in the calls to etcd. When we checked the etcd database for objects we found that one particular namespace which has 125 recurring runs is taking 996MB of etcd space
some data to look at:
namespace1 has 123 recurring runs namespace2 has 40 recurring runs namespace3 has 63 recurring runs
Expected result
Looks like we are storing a lot of unnecessary information in the ScheduledWorkflow object, which eventually is taking space in etcd database resulting in all the performance issues
Materials and Reference
Impacted by this bug? Give it a 👍.