Open lara-clink opened 1 week ago
Has anyone ever faced an issue like this?
Not me. But have no such a load (10 agents) :)
2.7.1
, next
?pprof
statistic? Is there some guide? I didn't find anything in the WP docs.pprof
info, but this screens from an agent, which allocated 44.36 MB of memory, if I understand correctly. However, Grafana shows memory usage around 1 GB and that is the issue (I suppose). It would be nice if you had pprof
stats from a mentioned agent.WOODPECKER_MAX_WORKFLOWS
and how many do you run simultaneously?
Could you explain the right half of the Grafana chart? Something like:
Hey @zc-devs , we are currently working on our migration project (automated migration from Drone CI to Woodpecker) and I could not collect all of the answers for you yet. By the end of this week I should be able to come back to that.
Component
agent
Describe the bug
I’ve been encountering what appears to be a memory leak issue when running Woodpecker CI on a Kubernetes cluster. After running pipelines over time, I noticed that the memory usage of the Woodpecker agents and server steadily increases, eventually leading to performance degradation and, in some cases, the need for manual intervention to prevent the system from becoming unresponsive.
Steps to reproduce
Deploy Woodpecker CI in a Kubernetes environment. Run multiple pipelines continuously over an extended period. Monitor memory usage of the Woodpecker agents and server, I will attach my grafana memory usage graph. Notice that memory consumption increases over time without being released after pipeline execution.
Expected behavior
Memory usage should stabilize after pipeline executions are completed, and unused memory should be reclaimed properly.
System Info
Additional context
I am using golang profiling to find something about it, this is what I could find so far:
Has anyone ever faced an issue like this?
Validations
next
version already [https://woodpecker-ci.org/faq#which-version-of-woodpecker-should-i-use]