search

woodpecker-ci / woodpecker

Woodpecker is a simple yet powerful CI/CD engine with great extensibility.
https://woodpecker-ci.org
Apache License 2.0
3.95k stars 351 forks source link

woodpecker slowed done #2064

Closed nupplaphil closed 1 month ago

nupplaphil commented 1 year ago

Component

agent

Describe the bug

I upgraded to the latest woodpecker "Next" version about a few weeks ago.

Since then, I notice that woodpecker is slowed down a lot (about 5-6 times slower).

I cannot really say why, but as far as I can see, it's related to the amount of console-output.

The .woodpecker files can be found at https://github.com/friendica/friendica/blob/develop/.woodpecker/.phpunit.yml

2 month ago, it was like this: image

now it's: image

I moved to an exclusive, brand new root-server with 128GB ram, i7 and only docker and woodpecker to have a look if it's an IO/CPU/.. issue --> nope the result stays the same

System Info

woodpecker: "next-c96fc3f2"
docker info:

Client: Docker Engine - Community
 Version:    24.0.5
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.11.2
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.20.2
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 30
  Running: 17
  Paused: 0
  Stopped: 13
 Images: 22
 Server Version: 24.0.5
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local lvm
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: active
  NodeID: 856ujaye5llxyg42a78wfl9s5
  Is Manager: false
  Node Address: 10.0.3.3
  Manager Addresses:
   10.0.3.2:2377
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 3dcXXXXXXXXXXXXXXXXXXXXXXXX
 runc version: v1.1.7-0-g860f061
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  userns
  cgroupns
  no-new-privileges
 Kernel Version: 6.1.0-10-amd64
 Operating System: Debian GNU/Linux 12 (bookworm)
 OSType: linux
 Architecture: x86_64
 CPUs: 12
 Total Memory: 125.7GiB
 Name: auct
 ID: 561aab6d-7adc-44a1-8f70-a7cd157e99ac
 Docker Root Dir: /var/lib/docker/165536.165536
 Debug Mode: false
 Experimental: true
 Insecure Registries:
  127.0.0.0/8

docker-compose (runner-nodes - 128GB RAM, i7, NVMe):
```console
version: '3'

services:
  default-agent:
    image: woodpeckerci/woodpecker-agent:next
    command: agent
    restart: always
    privileged: true
    userns_mode: host
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - woodpecker-default-agent-config:/etc/woodpecker
    environment:
      - WOODPECKER_SERVER=ci.friendi.ca:9000
      - WOODPECKER_AGENT_SECRET=XXXXX
      - WOODPECKER_MAX_PROCS=4
      - WOODPECKER_HOSTNAME=default-agent.opensocial.at
      - WOODPECKER_FILTER_LABELS=type=runner,location=opensocial
      - WOODPECKER_DEBUG_PRETTY=true
      - WOODPECKER_DEBUG_NOCOLOR=false

  specific-agent:
    image: woodpeckerci/woodpecker-agent:next
    command: agent
    restart: always
    privileged: true
    userns_mode: host
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - woodpecker-specific-agent-config:/etc/woodpecker
    environment:
      - WOODPECKER_SERVER=ci.friendi.ca:9000
      - WOODPECKER_AGENT_SECRET=XXXX
      - WOODPECKER_MAX_PROCS=4
      - WOODPECKER_HOSTNAME=XXX
      - WOODPECKER_FILTER_LABELS=type=runner,location=opensocial
      - WOODPECKER_DEBUG_PRETTY=true
      - WOODPECKER_DEBUG_NOCOLOR=false

volumes:
  woodpecker-default-agent-config:
  woodpecker-specific-agent-config:

docker-compose (server - slow node):

version: '3'

services:
  server:
    image: woodpeckerci/woodpecker-server:next
    volumes:
      - woodpecker-vol-1:/var/lib/woodpecker/
    restart: always
    ports:
      - "0.0.0.0:9000:9000"
      - "0.0.0.0:8000:8000"
    extra_hosts:
      - "git.friendi.ca:192.168.173.33"
    environment:
      - WOODPECKER_OPEN=true
      - WOODPECKER_HOST=https://ci.friendi.ca
      - WOODPECKER_GITHUB=true
      - WOODPECKER_GITHUB_CLIENT=XXX
      - WOODPECKER_GITHUB_SECRET=XXX
      - WOODPECKER_AGENT_SECRET=XXXX
      - WOODPECKER_ADMIN=XXX
      - WOODPECKER_REPO_OWNERS=XXX
      - WOODPECKER_DEBUG_PRETTY=true
      - WOODPECKER_DEBUG_NOCOLOR=false
      - "WOODPECKER_DEFAULT_CLONE_IMAGE=docker.io/woodpeckerci/plugin-git:2.1.0"

  runner:
    image: woodpeckerci/woodpecker-agent:next
    restart: always
    privileged: true
    userns_mode: host
    depends_on:
      - server
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    extra_hosts:
      - "git.friendi.ca:192.168.173.33"
    environment:
      - WOODPECKER_SERVER=server:9000
      - WOODPECKER_AGENT_SECRET=XXXX
      - WOODPECKER_MAX_PROCS=2
      - WOODPECKER_HOSTNAME=runner.ci.friendi.ca
      - WOODPECKER_FILTER_LABELS=type=runner,location=friendica
      - WOODPECKER_DEBUG_PRETTY=true
      - WOODPECKER_DEBUG_NOCOLOR=false

  releaser:
    image: woodpeckerci/woodpecker-agent:next
    restart: always
    privileged: true
    hostname: releaser.woodpecker.local
    userns_mode: host
    depends_on:
      - server
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    environment:
      - WOODPECKER_SERVER=server:9000
      - WOODPECKER_AGENT_SECRET=XXX
      - WOODPECKER_MAX_PROCS=2
      - WOODPECKER_HOSTNAME=releaser.ci.friendi.ca
      - WOODPECKER_FILTER_LABELS=type=releaser,location=friendica
      - WOODPECKER_DEBUG_PRETTY=true
      - WOODPECKER_DEBUG_NOCOLOR=false
      - WOODPECKER_ENVIRONMENT=node:releaser

volumes:
  woodpecker-vol-1:
    external: true
    name: woodpecker_data


### Additional context

_No response_

### Validations

- [X] Read the [Contributing Guidelines](https://github.com/woodpecker-ci/woodpecker/blob/master/CONTRIBUTING.md).
- [X] Read the [docs](https://woodpecker-ci.org/docs/intro).
- [X] Check that there isn't [already an issue](https://github.com/woodpecker-ci/woodpecker/issues) that reports the same bug to avoid creating a duplicate.
- [X] Checked that the bug isn't fixed in the `next` version already [https://woodpecker-ci.org/faq#which-version-of-woodpecker-should-i-use]
- [X] Check that this is a concrete bug. For Q&A join our [Discord Chat Server](https://discord.gg/fcMQqSMXJy) or the [Matrix room](https://matrix.to/#/#woodpecker:matrix.org).
6543 commented 1 year ago

My new guess: the new log streaming does not buffer and let the stdout pipe do ioWait untill grpc did ack it got send.

So if we just wrap that pipe into a buffered one it should work

6543 commented 1 year ago

https://github.com/djherbis/nio

https://pkg.go.dev/bufio

zc-devs commented 1 year ago

I had symptoms similar to your, when I built Woodpecker and ephemeral storage was not enough (in Kubernetes though). Containerd downloads images (some of them over 1GB), unpacks them and runs containers/pods/steps. In Kubernetes pods (steps) are cleared when whole pipeline finishes. So, in the end of pipeline there a little storage was left and downloaded images and its unpacked versions were deleted too. Next pipeline starts from scratch: download, unpack, etc.

ateuber commented 1 year ago

We have this problem, too. Not sure with which version it started. We have a repository with a 7GB image which is much slower than another repository with normal size images, so docker image sizes definitely are involved. Strangely it also seems to be depending on the user who pushed. It was tolerable before, but now it takes forever and causes high IO on the VM.

6543 commented 1 year ago

well it it's a pure docker thing, woodpecker can not do much. we also did not change the functionality of the docker backend so I would be surprised if it is caused by us.

But we need more data, just some "feeling" dont help :sweat:

6543 commented 12 months ago

on there are now two images based ontop of next:

witch both have 10Kb buffer for log streaming

would be nice if there performance can be tested (I'll do testing myselfe too)

ateuber commented 12 months ago

@6543 Awesome, thanks! :slightly_smiling_face: woodpeckerci/woodpecker-agent:pull_2072 seems to solve the problem for us. Will do some further tests to make sure and also test woodpeckerci/woodpecker-agent:pull_2074.

6543 commented 12 months ago

pull_2074 is in the pipeline: https://ci.woodpecker-ci.org/repos/3780/pipeline/6932

6543 commented 12 months ago

please tell what's faster :) (if there is a difference)

no diff so stdlib is preferred

6543 commented 12 months ago

ok it will be fixed by #2072

we just need to address how to handle the new input as currently chunks are somewhat hardcoded as "one line" and with that it would mean that there is no "live"-log streaming ... so it get's complicated.

you can use the pull preview image as hotfix, until I figured out how to log stream and use a buffer but time based

ateuber commented 12 months ago

@6543 Awesome, thanks! slightly_smiling_face woodpeckerci/woodpecker-agent:pull_2072 seems to solve the problem for us. Will do some further tests to make sure and also test woodpeckerci/woodpecker-agent:pull_2074.

Unfortunately I can't reproduce our problem with v1.0 any more either. I think it really has to do something with the user who started the build somehow..

6543 commented 12 months ago

ok then this was something else, but there is an issue with "high-volume" log outputing steps