search

woodpecker-ci / woodpecker

Woodpecker is a simple, yet powerful CI/CD engine with great extensibility.
https://woodpecker-ci.org
Apache License 2.0
4.3k stars 371 forks source link

jobs in queue dont have a timeout and can loop infinite #1518

Open 6543 opened 1 year ago

6543 commented 1 year ago

woodpecker server log:


{"level":"debug","time":"2023-01-02T00:34:54Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:335","message":"queue: pending right now: 320496"}
{"level":"debug","time":"2023-01-02T00:34:54Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:335","message":"queue: pending right now: 320497"}
{"level":"debug","time":"2023-01-02T00:34:54Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:335","message":"queue: pending right now: 320505"}
{"level":"debug","time":"2023-01-02T00:34:54Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:335","message":"queue: pending right now: 320506"}
{"level":"debug","time":"2023-01-02T00:34:54Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:335","message":"queue: pending right now: 320514"}
{"level":"debug","time":"2023-01-02T00:34:54Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:335","message":"queue: pending right now: 320515"}
{"level":"debug","time":"2023-01-02T00:34:54Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:335","message":"queue: pending right now: 320523"}
{"level":"debug","time":"2023-01-02T00:34:54Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:335","message":"queue: pending right now: 320524"}
{"level":"debug","time":"2023-01-02T00:34:54Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:335","message":"queue: pending right now: 320532"}
{"level":"debug","time":"2023-01-02T00:34:54Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:335","message":"queue: pending right now: 320533"}
{"level":"debug","time":"2023-01-02T00:34:54Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:335","message":"queue: pending right now: 320541"}
{"level":"debug","time":"2023-01-02T00:34:54Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:335","message":"queue: pending right now: 320542"}
{"level":"debug","time":"2023-01-02T00:34:54Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:335","message":"queue: pending right now: 320550"}
{"level":"debug","time":"2023-01-02T00:34:54Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:335","message":"queue: pending right now: 320551"}
{"level":"debug","time":"2023-01-02T00:34:54Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:335","message":"queue: pending right now: 320559"}
{"level":"debug","time":"2023-01-02T00:34:54Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:335","message":"queue: pending right now: 320560"}
{"level":"debug","time":"2023-01-02T00:34:54Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:335","message":"queue: pending right now: 320568"}
{"level":"debug","time":"2023-01-02T00:34:54Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:335","message":"queue: pending right now: 320569"}
{"level":"debug","time":"2023-01-02T00:34:54Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:335","message":"queue: pending right now: 320577"}
{"level":"debug","time":"2023-01-02T00:34:54Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:335","message":"queue: pending right now: 320578"}
{"level":"debug","time":"2023-01-02T00:34:54Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:335","message":"queue: pending right now: 320586"}
{"level":"debug","time":"2023-01-02T00:34:54Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:335","message":"queue: pending right now: 320587"}
{"level":"debug","time":"2023-01-02T00:34:54Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:335","message":"queue: pending right now: 320595"}
{"level":"debug","time":"2023-01-02T00:34:54Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:335","message":"queue: pending right now: 320596"}
{"level":"debug","time":"2023-01-02T00:34:54Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:335","message":"queue: pending right now: 320604"}
{"level":"debug","time":"2023-01-02T00:34:54Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:335","message":"queue: pending right now: 320605"}
{"level":"debug","time":"2023-01-02T00:34:54Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:335","message":"queue: pending right now: 320613"}
{"level":"debug","time":"2023-01-02T00:34:54Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:335","message":"queue: pending right now: 320614"}
{"level":"debug","time":"2023-01-02T00:34:54Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:335","message":"queue: pending right now: 320622"}
{"level":"debug","time":"2023-01-02T00:34:54Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:335","message":"queue: pending right now: 320623"}
{"level":"debug","time":"2023-01-02T00:34:54Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:335","message":"queue: pending right now: 320631"}
{"level":"debug","time":"2023-01-02T00:34:54Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:335","message":"queue: pending right now: 320632"}
{"level":"debug","time":"2023-01-02T00:34:54Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:335","message":"queue: pending right now: 320640"}
{"level":"debug","time":"2023-01-02T00:34:54Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:335","message":"queue: pending right now: 320641"}
{"level":"debug","time":"2023-01-02T00:34:54Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:335","message":"queue: pending right now: 320649"}
{"level":"debug","time":"2023-01-02T00:34:54Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:335","message":"queue: pending right now: 320650"}
{"level":"debug","time":"2023-01-02T00:34:54Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:335","message":"queue: pending right now: 320658"}
{"level":"debug","time":"2023-01-02T00:34:54Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:335","message":"queue: pending right now: 320659"}
{"level":"debug","time":"2023-01-02T00:34:54Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:335","message":"queue: pending right now: 320667"}
{"level":"debug","time":"2023-01-02T00:34:54Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:335","message":"queue: pending right now: 320668"}
{"level":"debug","time":"2023-01-02T00:34:54Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:335","message":"queue: pending right now: 320676"}
{"level":"debug","time":"2023-01-02T00:34:54Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:335","message":"queue: pending right now: 320677"}
{"level":"debug","time":"2023-01-02T00:34:54Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:335","message":"queue: pending right now: 320685"}
{"level":"debug","time":"2023-01-02T00:34:54Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:335","message":"queue: pending right now: 320686"}
{"level":"debug","time":"2023-01-02T00:34:54Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:335","message":"queue: pending right now: 320694"}
{"level":"debug","time":"2023-01-02T00:34:54Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:335","message":"queue: pending right now: 320695"}
{"level":"debug","time":"2023-01-02T00:34:54Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:335","message":"queue: pending right now: 320703"}
{"level":"debug","time":"2023-01-02T00:34:54Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:335","message":"queue: pending right now: 320704"}
{"level":"debug","time":"2023-01-02T00:34:54Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:335","message":"queue: pending right now: 320712"}
{"level":"debug","time":"2023-01-02T00:34:54Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:335","message":"queue: pending right now: 320713"}
{"level":"debug","time":"2023-01-02T00:34:54Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:335","message":"queue: pending right now: 320721"}
{"level":"debug","time":"2023-01-02T00:34:54Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:335","message":"queue: pending right now: 320722"}
{"level":"debug","time":"2023-01-02T00:34:54Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:335","message":"queue: pending right now: 320730"}
{"level":"debug","time":"2023-01-02T00:34:54Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:335","message":"queue: pending right now: 320731"}
{"level":"debug","time":"2023-01-02T00:34:54Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:335","message":"queue: pending right now: 320739"}
{"level":"debug","time":"2023-01-02T00:34:54Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:335","message":"queue: pending right now: 320740"}
{"level":"debug","time":"2023-01-02T00:34:54Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:335","message":"queue: pending right now: 320748"}
{"level":"debug","time":"2023-01-02T00:34:54Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:335","message":"queue: pending right now: 320749"}
{"level":"debug","time":"2023-01-02T00:34:54Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:335","message":"queue: pending right now: 320757"}
{"level":"debug","time":"2023-01-02T00:34:54Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:335","message":"queue: pending right now: 320758"}
{"level":"debug","time":"2023-01-02T00:34:54Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:335","message":"queue: pending right now: 320766"}
{"level":"debug","time":"2023-01-02T00:34:54Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:335","message":"queue: pending right now: 320767"}
{"level":"debug","time":"2023-01-02T00:34:54Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:335","message":"queue: pending right now: 320775"}
{"level":"debug","time":"2023-01-02T00:34:54Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:335","message":"queue: pending right now: 320776"}
{"level":"debug","time":"2023-01-02T00:34:54Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:335","message":"queue: pending right now: 320784"}
{"level":"debug","time":"2023-01-02T00:34:54Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:335","message":"queue: pending right now: 320785"}
{"level":"debug","time":"2023-01-02T00:34:54Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:335","message":"queue: pending right now: 320793"}
{"level":"debug","time":"2023-01-02T00:34:54Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:335","message":"queue: pending right now: 320794"}
{"level":"debug","time":"2023-01-02T00:34:54Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:335","message":"queue: pending right now: 320802"}
{"level":"debug","time":"2023-01-02T00:34:54Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:335","message":"queue: pending right now: 320803"}
{"level":"debug","time":"2023-01-02T00:34:54Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:335","message":"queue: pending right now: 320811"}
{"level":"debug","time":"2023-01-02T00:34:54Z","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/queue/fifo.go:335","message":"queue: pending right now: 320812"}
6543 commented 1 year ago

related bug/feature: -> admin should be able to reset queue.

lhemala commented 7 months ago

Hi everyone,

we use a complex build config with 30 pipelines per build and a lot inter-dependencies between them. We use two agents with a capacity of 4 and 10. The server is running on the same server as the agent with the capacity of 4. When multiple builds are pending - so there are 50 to 90 pending pipelines in the queue - we sometimes have the issue, that the woodpecker server gets unresponsive and the logs look like shown here. This can on go on for a while and sometimes resolves on its own, but in other cases I had to recover from this state manually (e.g. after 30min). To do that, I stop/kill the agents and then restart the server. Before I start the agents again, I abort all pending/in progress builds and only restart a single one.

How can we help fixing the issue? Do you have something specifically in mind we should look at? Would it help if we provide more logs? We're currently on version 2.4.1 but I think we had this issue for a longer time; although to a smaller extent. I noticed when tailing the docker logs, that the server is emitting more logs per second than docker is able return - noticeable by the timestamp in the logs incrementing slower than the wall clock time. I haven't noticed this issue to occur for our more simple builds with only one or two pipelines.

Thanks