search

tektoncd / chains

Supply Chain Security in Tekton Pipelines
Apache License 2.0
240 stars 125 forks source link

Chains under the load memory consumption raises #1058

Open jhutar opened 5 months ago

jhutar commented 5 months ago

Expected Behavior

I would expect after some time Chains under the load memory consumption would become constant - it will start freeing memory.

Is this expected, or is this some sort of memory leak?

Actual Behavior

This is a memory graph of a Chains signing 10k very simple TaskRuns with that are just printing "hello world" (Pipeline, PipelineRun)

CHains was started around 15:30 and started signing PRs and TRs:

image

Chains was configured with this:

kubectl patch TektonConfig/config \
            --type merge \
            -p '{"spec":{"chain":{"artifacts.pipelinerun.format": "slsa/v1"}}}'
kubectl patch TektonConfig/config \
            --type merge \
            -p '{"spec":{"chain":{"artifacts.pipelinerun.storage": "tekton"}}}'
kubectl patch TektonConfig/config \
            --type='merge' \
            -p='{"spec":{"chain":{"artifacts.taskrun.format": "slsa/v1"}}}'
kubectl patch TektonConfig/config \
            --type='merge' \
            -p='{"spec":{"chain":{"artifacts.taskrun.storage": "tekton"}}}'

Steps to Reproduce the Problem

  1. Run 10k PipelineRuns and wait for all of them to finish
  2. Then start the Chains and let it sign PRs and TRs
  3. This was automated in this repo with signing-tr-tekton-bigbang scenario

Additional Info

Cluster is gone already, but it was ROSA OpenShift 4.14.11 with 5 compute nodes AWS EC2 m6a.2xlarge

Reported this together with https://github.com/tektoncd/pipeline/issues/7691

concaf commented 5 months ago

@jhutar what's the baseline memory usage of chains in this case when there are no workloads on the cluster?

wlynch commented 5 months ago

https://github.com/tektoncd/plumbing/pull/1840 to enable profiling on our dogfooding instance to help debug this, but if you want to enable this on your cluster and share the pprof output, that might be faster 🙏

Steps:

  1. Add profiling.enable: "true" totekton-chains/tekton-chains-config-observability
  2. kubectl port-forward -n tekton-chains tekton-chains-controller-794dcd9b65-k9f8d 8008 (replace pod name with your own)
  3. wget localhost:8008/debug/pprof/heap
jhutar commented 3 months ago

Oh, thank you Billy! Will try to get a pprof output!

Maybe this is similar to https://github.com/tektoncd/pipeline/issues/7691 : memory used by informer to cache everything.