Gitea tag event: Step does not start

susnux commented 2 years ago

Component

server

Describe the bug

I tried to get my first steps with woodpeckerci, pipelines on a git-push work fine, but when I create a new tag woodpecker shows:

Endless spinning wheel (circular progress)
This step has been canceled. in the build view (.../build/1/4)
Not yet started on the right menu

Trying to click on cancel even does not work (error: An unknown error occured).

I used a slightly modified docker-compose from the docs, and also tried the 0.14.3 release (docker image), but that version did not start at all (see last log).

System Info

/version:

{"source":"https://github.com/woodpecker-ci/woodpecker","version":"next-"}

docker-compose.yml:

version: '3'

services:
  woodpecker-server:
    image: woodpeckerci/woodpecker-server:latest
    restart: always
    ports:
      - 8000:8000
    volumes:
      - woodpecker-server-data:/var/lib/woodpecker/
    environment:
      - WOODPECKER_OPEN=true
      - WOODPECKER_ORGS=Xxx
      - WOODPECKER_ADMIN=xxx,yyy
      - WOODPECKER_REPO_OWNERS=Xxxx
      - WOODPECKER_HOST=${WOODPECKER_HOST}
      - WOODPECKER_GITEA=true
      - WOODPECKER_GITEA_URL=${WOODPECKER_GITEA_URL}
      - WOODPECKER_GITEA_CLIENT=${WOODPECKER_GITEA_CLIENT}
      - WOODPECKER_GITEA_SECRET=${WOODPECKER_GITEA_SECRET}
      - WOODPECKER_AGENT_SECRET=${WOODPECKER_AGENT_SECRET}

  woodpecker-agent:
    image: woodpeckerci/woodpecker-agent:latest
    command: agent
    restart: always
    depends_on:
      - woodpecker-server
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    environment:
      - WOODPECKER_SERVER=woodpecker-server:9000
      - WOODPECKER_MAX_PROCS=2
      - WOODPECKER_AGENT_SECRET=${WOODPECKER_AGENT_SECRET}

volumes:
  woodpecker-server-data:

Docker version 20.10.9-ce, build 79ea9d308018 docker-compose version 1.25.1, build a82fef0

Additional context

latest docker log (before trying to cancel):

woodpecker-server_1  | {"level":"warn","error":"sql: no rows in result set","time":"2021-12-08T17:20:04Z"}
woodpecker-agent_1   | {"level":"error","error":"rpc error: code = Unavailable desc = closing transport due to: connection error: desc = \"error reading from server: EOF\", received prior goaway: code: ENHANCE_YOUR_CALM, debug data: \"too_many_pings\"","time":"2021-12-08T17:20:12Z","message":"grpc error: done(): code: Unavailable: rpc error: code = Unavailable desc = closing transport due to: connection error: desc = \"error reading from server: EOF\", received prior goaway: code: ENHANCE_YOUR_CALM, debug data: \"too_many_pings\""}
woodpecker-agent_1   | {"level":"error","error":"rpc error: code = Unavailable desc = closing transport due to: connection error: desc = \"error reading from server: EOF\", received prior goaway: code: ENHANCE_YOUR_CALM, debug data: \"too_many_pings\"","time":"2021-12-08T17:20:12Z","message":"grpc error: done(): code: Unavailable: rpc error: code = Unavailable desc = closing transport due to: connection error: desc = \"error reading from server: EOF\", received prior goaway: code: ENHANCE_YOUR_CALM, debug data: \"too_many_pings\""}
woodpecker-server_1  | {"level":"error","time":"2021-12-08T17:21:00Z","message":"failure to parse token from hook for XXX/xxx-yyy. signature is invalid"}
woodpecker-server_1  | {"level":"error","error":"Error #01: signature is invalid\n","ip":"172.22.0.1","latency":0.803823,"method":"POST","path":"/hook","status":400,"time":"2021-12-08T17:21:00Z","user-agent":"Go-http-client/1.1","time":"2021-12-08T17:21:00Z"}

Additional when clicking on cancel:

woodpecker-server_1  | {"level":"error","error":"Error #01: queue: task not found\n","ip":"172.22.0.1","latency":1.759874,"method":"DELETE","path":"/api/repos/XXX/xxx-yyy/builds/1/1","status":500,"time":"2021-12-08T17:30:08Z","user-agent":"Mozilla/5.0 (X11; Linux x86_64; rv:94.0) Gecko/20100101 Firefox/94.0","time":"2021-12-08T17:30:08Z"}

Log for 0.14.3 release with docker (UI never starts)

woodpecker-server_1  | time="2021-12-08T17:18:15Z" level=error msg="unable to open database file: no such file or directory"
woodpecker-server_1  | time="2021-12-08T17:18:15Z" level=fatal msg="database ping attempts failed"
woodpecker-agent_1   | 2021/12/08 17:18:15 grpc error: done(): code: Unavailable: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 172.22.0.3:9000: connect: connection refused"

Validations

[X] Read the Contributing Guidelines.
[X] Read the docs.
[X] Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
[X] Check that this is a concrete bug. For Q&A join our Discord Chat Server.

6543 commented 2 years ago

@susnux can you fetch latest image again & test against it?

some fixes wen in just now & send the new response from GET /version ?

6543 commented 2 years ago

switch from latest=next to v0.14.x does not work out of the box, you have to undo migration tasks: https://woodpecker-ci.org/docs/migrations
cancel on latest do currently not work (known issue - working on it - so it's fixed soon^TM)

susnux commented 2 years ago

send the new response from GET /version ?

{"source":"https://github.com/woodpecker-ci/woodpecker","version":"next-e7cfa902"}

Issue is still the same / old task still "not started yet" and at the same time "This step has been canceled."

6543 commented 2 years ago

@susnux can you paste the pipeline config?

susnux commented 2 years ago

I tried 0.14.3 again, but for tags the output is still (endless) Pending.

I noticed this in the 0.14.3 log:

woodpecker-server_1 | time="2021-12-08T19:01:25Z" level=error msg="failure to parse token from hook for XXX/xxx-yyy. signature is invalid" woodpecker-server_1 | time="2021-12-08T19:01:25Z" level=error msg="Error #01: signature is invalid\n" fields.time="2021-12-08T19:01:25Z" ip=141.30.30.142 latency=1.175165ms method=POST path=/hook status=400 user-agent=Go-http-client/1.1

FYI I use gitea 1.15.7.

susnux commented 2 years ago

can you paste the pipeline config?

Just for testing the tag hook, I used:

# .woodpecker.yml
pipeline:
  lint:
    image: node:lts-alpine
    commands:
      - yarn install
      - yarn lint

  publish:
    when:
      event: tag
      tag: v*
    image: node:lts-alpine
    commands:
      - yarn publish --non-interactive
    secrets: [ node_auth_token ]

branches: [main, develop]

6543 commented 2 years ago

@susnux looks like the web-hook secret is not correct, just try to deactivate and activate the specific repo within woodpecker

PS: cancel bugfix on the way: #585

susnux commented 2 years ago

Do you mean "remove repository" and reenable? Did that, but that did not change anything. Same messages: log.txt

I also tried "repair repository", after that the 1 tag event creates 2 pipelines with the same error as above.

6543 commented 2 years ago

ok this is the main issue: {"level":"error","time":"2021-12-08T22:53:24Z","message":"error: org/repo: cannot find in refs/tags/v1.0.0-alpha.5: ConfigFetcher: Fallback did not found config: 404 Not Found"}

so somehow woodpecker can not fetch config via gitea api so we have to look why gitea api report back 404 instead of returning a file content

susnux commented 2 years ago

Oh sorry the log is not relevant, noticed there are some parts from deleting the tags for next try. On a fresh installation there are no special logs (besides the "no rows in result set" lines).

I can see from the webhook, that at least the parsing was successful (response is 200 with JSON object). And even with debug logs enabled there is no further log message.

6543 commented 2 years ago

ugh - I finaly was able to reproduce it :tada: !!

[
  {
    "host": "server",
    "level": "trace",
    "time": "2021-12-10T21:24:23Z",
    "caller": "/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/router/router.go:38",
    "message": "[POST] /hook?access_token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ0ZXh0IjoiNTgxL3Rlc3RDSXNlcnZpY2VzIiwidHlwZSI6Imhvb2sifQ.dXtYKrgc1RqYwBnOQAuoPkqcxv7IQccMFLg_0TrgBlc"
  },
  {
    "host": "server",
    "level": "info",
    "latency": 0.469601,
    "method": "POST",
    "path": "/hook",
    "status": 200,
    "time": "2021-12-10T21:24:23Z",
    "user-agent": "Go-http-client/1.1",
    "caller": "/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/router/middleware/logger.go:45"
  },
  {
    "host": "server",
    "level": "trace",
    "time": "2021-12-10T21:24:23Z",
    "caller": "/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/router/router.go:38",
    "message": "[POST] /hook?access_token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ0ZXh0IjoiNTgxL3Rlc3RDSXNlcnZpY2VzIiwidHlwZSI6Imhvb2sifQ.dXtYKrgc1RqYwBnOQAuoPkqcxv7IQccMFLg_0TrgBlc"
  },
  {
    "host": "server",
    "level": "trace",
    "time": "2021-12-10T21:24:23Z",
    "caller": "/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/shared/configFetcher.go:41",
    "message": "Start Fetching config for '581/testCIservices'"
  },
  {
    "host": "server",
    "level": "trace",
    "time": "2021-12-10T21:24:23Z",
    "caller": "/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/shared/configFetcher.go:85",
    "message": "ConfigFetch[581/testCIservices]: user did not defined own config follow default procedure"
  },
  {
    "host": "server",
    "level": "trace",
    "time": "2021-12-10T21:24:23Z",
    "caller": "/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/shared/configFetcher.go:101",
    "message": "ConfigFetch[581/testCIservices]: found file '.woodpecker.yml'"
  },
  {
    "host": "server",
    "level": "trace",
    "time": "2021-12-10T21:24:23Z",
    "caller": "/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/shared/configFetcher.go:46",
    "message": "0 try failed: <nil>"
  },
  {
    "host": "server",
    "level": "trace",
    "time": "2021-12-10T21:24:23Z",
    "caller": "/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/api/hook.go:301",
    "message": "hook.branchFiltered(): build branch: 'refs/tags/v2' build event: 'tag' config count: 1"
  },
  {
    "host": "server",
    "level": "trace",
    "time": "2021-12-10T21:24:23Z",
    "caller": "/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/api/hook.go:308",
    "message": "config '.woodpecker.yml': &yaml.Config{Cache:types.Stringorslice(nil), Platform:\"\", Branches:yaml.Constraint{Include:[]string{\"main\", \"develop\"}, Exclude:[]string(nil)}, Workspace:yaml.Workspace{Base:\"\", Path:\"\"}, Clone:yaml.Containers{Containers:[]*yaml.Container(nil)}, Pipeline:yaml.Containers{Containers:[]*yaml.Container{(*yaml.Container)(0xc000395680), (*yaml.Container)(0xc000395b00)}}, Services:yaml.Containers{Containers:[]*yaml.Container(nil)}, Networks:yaml.Networks{Networks:[]*yaml.Network(nil)}, Volumes:yaml.Volumes{Volumes:[]*yaml.Volume(nil)}, Labels:types.SliceorMap(nil), DependsOn:[]string(nil), RunsOn:[]string(nil), SkipClone:false}"
  },
  {
    "host": "server",
    "level": "info",
    "latency": 512.372151,
    "method": "POST",
    "path": "/hook",
    "status": 200,
    "time": "2021-12-10T21:24:23Z",
    "user-agent": "Go-http-client/1.1",
    "caller": "/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/router/middleware/logger.go:45"
  },
  {
    "host": "agent1",
    "level": "error",
    "error": "rpc error: code = Unavailable desc = closing transport due to: connection error: desc = \"error reading from server: EOF\", received prior goaway: code: ENHANCE_YOUR_CALM, debug data: \"too_many_pings\"",
    "time": "2021-12-10T21:24:47Z",
    "caller": "/woodpecker/src/github.com/woodpecker-ci/woodpecker/pipeline/rpc/client_grpc.go:49",
    "message": "grpc error: done(): code: Unavailable: rpc error: code = Unavailable desc = closing transport due to: connection error: desc = \"error reading from server: EOF\", received prior goaway: code: ENHANCE_YOUR_CALM, debug data: \"too_many_pings\""
  },
  {
    "host": "agent1",
    "level": "error",
    "error": "rpc error: code = Unavailable desc = closing transport due to: connection error: desc = \"error reading from server: EOF\", received prior goaway: code: ENHANCE_YOUR_CALM, debug data: \"too_many_pings\"",
    "time": "2021-12-10T21:24:47Z",
    "caller": "/woodpecker/src/github.com/woodpecker-ci/woodpecker/pipeline/rpc/client_grpc.go:49",
    "message": "grpc error: done(): code: Unavailable: rpc error: code = Unavailable desc = closing transport due to: connection error: desc = \"error reading from server: EOF\", received prior goaway: code: ENHANCE_YOUR_CALM, debug data: \"too_many_pings\""
  },
  {
    "host": "server",
    "level": "debug",
    "time": "2021-12-10T21:24:48Z",
    "caller": "/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/grpc/rpc.go:63",
    "message": "agent connected: 6a11f8ea22b3: polling"
  },
  {
    "host": "server",
    "level": "debug",
    "time": "2021-12-10T21:24:48Z",
    "caller": "/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/grpc/rpc.go:63",
    "message": "agent connected: 6a11f8ea22b3: polling"
  }
]

6543 commented 2 years ago

@susnux it's the global branch filter!!!

do NOT work:

# .woodpecker.yml
pipeline:
  publish:
    when:
      event: tag
    image: node:lts-alpine
    commands:
      - (cd web; yarn build)

branches: [main, develop]

work:

# .woodpecker.yml
pipeline:
  publish:
    when:
      event: tag
    image: node:lts-alpine
    commands:
      - (cd web; yarn build)

6543 commented 2 years ago

@susnux workaround use branch filter per step where event !tag https://woodpecker-ci.org/docs/usage/conditional-execution#branch

I'll look into fixing this meanwhile

susnux commented 2 years ago

Oh great! As the pipeline started and push-event worked I never thought the filters were the issue. Thank you for investigating.

PS: Maybe add some notes about debugging in the documentation? I tried to track it down, but even the ability to set the logging level to debug using an environment variable was hidden (only found it by searching the repo and finding some commits about it ;) )

susnux commented 2 years ago

I used the workaround and the latest docker image, now it starts, but only the clone step is run (exit without errors, but next step is not executed). Log when pressing restart: log.txt

EDIT It does work when split pipelines into multiple files! Using a single file for different types seems to fail

6543 commented 2 years ago

-> #659

woodpecker-ci / woodpecker