woodpecker-ci / woodpecker

Woodpecker is a simple, yet powerful CI/CD engine with great extensibility.
https://woodpecker-ci.org
Apache License 2.0
4.32k stars 373 forks source link

kubernetes: git clone broken #1955

Closed ymettier closed 1 year ago

ymettier commented 1 year ago

Component

server, agent, web-ui

Describe the bug

Step git clone does not work.

It was working with next-94f2331a4c (July 6th 2023).

It is broken with next-1d6b073ca5 (July 8th 2023 few hours ago).

image

On the right pane, I can read Chargement… that means Loading...

I can switch from one version (it works) to the other version (it does not work) and rollback to the previous version (it works) again and again. I made no change anywhere except the image tag of Woodpecker agent and server.

I compared the logs ( kubectl logs woodpecker-0; kubectl logs woodpecker-agent-xxxxxx). There is no interesting difference.

System Info

Working with next-94f2331a4c (July 6th 2023).
Broken with next-1d6b073ca5 (July 8th 2023 few hours ago).

Kubernetes backend

Additional context

No response

Validations

6543 commented 1 year ago

there are two related pulls since then: #1914 & #1909

cc @zc-devs

6543 commented 1 year ago

thanks for the detailed report @ymettier :+1:

zc-devs commented 1 year ago

Custom next-556607b525 with all my changes which work. The only difference is used env var: I still use CI_SYSTEM_ARCH :)

  containers:
    - name: wp-01h4w19zvmmgb91jgdwcwde9qt-0-step-3
      image: woodpeckerci/plugin-docker-buildx
      workingDir: /woodpecker/src/forge/user/woodpecker
      env:
        - name: CI_PREV_COMMIT_SHA
        - name: CI_REPO_PRIVATE
          value: 'true'
          ...
        - name: DRONE_REPO_BRANCH
          value: master
        - name: CI_SYSTEM_PLATFORM
        - name: CI_SYSTEM_ARCH
          value: linux/amd64
          ...

I reproduce permanent loading if using CI_SYSTEM_PLATFORM.

6543 commented 1 year ago

witch should not have any erffect:

https://github.com/woodpecker-ci/woodpecker/blob/ea895baf836fc7d9c46e02fd883f432f2d0261e9/pipeline/frontend/metadata/environment.go#L109 https://github.com/woodpecker-ci/woodpecker/blob/ea895baf836fc7d9c46e02fd883f432f2d0261e9/pipeline/frontend/metadata/environment.go#L116

6543 commented 1 year ago

hmm we might need the actual logs ?

zc-devs commented 1 year ago

woodpecker-server.log woodpecker-agent.log

Unfortunately, Agent is terse even though in TRACE level.

zc-devs commented 1 year ago

0001-Log-platform.patch woodpecker-agent.log

Agent crashes right after running pipeline.

zc-devs commented 1 year ago

Metadata.System.Platform is always empty https://github.com/woodpecker-ci/woodpecker/blob/de13aeea339bd72d8417b2aea5b51dd1dc540632/pipeline/frontend/metadata.go#L97 Therefore on Server both env vars (CI_SYSTEM_ARCH, CI_SYSTEM_PLATFORM) are empty.

Agent receives empty vars via RPC. Then right before https://github.com/woodpecker-ci/woodpecker/blob/de13aeea339bd72d8417b2aea5b51dd1dc540632/pipeline/pipeline.go#L152 both vars are empty. But right before https://github.com/woodpecker-ci/woodpecker/blob/de13aeea339bd72d8417b2aea5b51dd1dc540632/pipeline/pipeline.go#L180 CI_SYSTEM_ARCH is set.

Between this two lines of code there were two lines in log that mentioned tracer. So, here we are https://github.com/woodpecker-ci/woodpecker/blob/de13aeea339bd72d8417b2aea5b51dd1dc540632/agent/tracer.go#L79

agent-log.txt ^ Log lines would be slightly different.