`*` spamming and GPRC error after latest pull of 0.15.0-rc1

nupplaphil commented 2 years ago

Component

server, agent, web-ui

Describe the bug

I pulled the latest woodpeckerci/woodpecker-server and woodpeckerci/woordpecker-agent some hours ago, and there are two unwanted behaviors now:

Log spammed with * see https://ci.friendi.ca/friendica/friendica/build/124/8 The step output is now filled with * before and after each letter in the output, like

********+******** ********g********i********t******** ********i********n********i********t******** ********
********h********i********n********t********:******** ********U********s********i********n********g******** ********'********m********a********s********t********e********r********'******** ********a********s******** ********t********h********e******** ********n********a********m********e******** ********f********o********r******** ********t********h********e******** ********i********n********i********t********i********a********l******** ********b********r********a********n********c********h********.******** ********T********h********i********s******** ********d********e********f********a********u********l********t******** ********b********r********a********n********c********h******** ********n********a********m********e******** ********
********h********i********n********t********:******** ********i********s******** ********s********u********b********j********e********c********t******** ********t********o******** ********c********h********a********n********g********e********.******** ********T********o******** ********c********o********n********f********i********g********u********r********e******** ********t********h********e******** ********i********n********i********t********i********a********l******** ********b********r********a********n********c********h******** ********n********a********m********e******** ********t********o******** ********u********s********e******** ********i********n******** ********a********l********l******** ********
********h********i********n********t********:******** ********o********f******** ********y********o********u********r******** ********n********e********w******** ********r********e********p********o********s********i********t********o********r********i********e********s********,******** ********w********h********i********c********h******** ********w********i********l********l******** ********s********u********p********p********r********e********s********s******** ********t********h********i********s******** ********w********a********r********n********i********n********g********,******** ********c********a********l********l********:******** ********
********h********i********n********t********:******** ******** ********

GPRC errors The log is filled with the same error src/pipeline/rpc/client_grpc.go:287 > grpc error: log(): code: Internal: rpc error: code = Internal desc = grpc: error while marshaling: string field contains invalid UTF-8 error="rpc error: code = Internal desc = grpc: error while marshaling: string field contains invalid UTF-8" all over, like

woodpecker-runner-1    | 10:23PM ERR src/pipeline/rpc/client_grpc.go:287 > grpc error: log(): code: Internal: rpc error: code = Internal desc = grpc: error while marshaling: string field contains invalid UTF-8 error="rpc error: code = Internal desc = grpc: error while marshaling: string field contains invalid UTF-8"
woodpecker-runner-1    | 10:23PM ERR src/pipeline/rpc/client_grpc.go:287 > grpc error: log(): code: Internal: rpc error: code = Internal desc = grpc: error while marshaling: string field contains invalid UTF-8 error="rpc error: code = Internal desc = grpc: error while marshaling: string field contains invalid UTF-8"
woodpecker-runner-1    | 10:23PM ERR src/pipeline/rpc/client_grpc.go:287 > grpc error: log(): code: Internal: rpc error: code = Internal desc = grpc: error while marshaling: string field contains invalid UTF-8 error="rpc error: code = Internal desc = grpc: error while marshaling: string field contains invalid UTF-8"
woodpecker-runner-1    | 10:23PM ERR src/pipeline/rpc/client_grpc.go:287 > grpc error: log(): code: Internal: rpc error: code = Internal desc = grpc: error while marshaling: string field contains invalid UTF-8 error="rpc error: code = Internal desc = grpc: error while marshaling: string field contains invalid UTF-8"
woodpecker-runner-1    | 10:23PM ERR src/pipeline/rpc/client_grpc.go:287 > grpc error: log(): code: Internal: rpc error: code = Internal desc = grpc: error while marshaling: string field contains invalid UTF-8 error="rpc error: code = Internal desc = grpc: error while marshaling: string field contains invalid UTF-8"
woodpecker-runner-1    | 10:23PM ERR src/pipeline/rpc/client_grpc.go:287 > grpc error: log(): code: Internal: rpc error: code = Internal desc = grpc: error while marshaling: string field contains invalid UTF-8 error="rpc error: code = Internal desc = grpc: error while marshaling: string field contains invalid UTF-8"
woodpecker-runner-1    | 10:23PM ERR src/pipeline/rpc/client_grpc.go:287 > grpc error: log(): code: Internal: rpc error: code = Internal desc = grpc: error while marshaling: string field contains invalid UTF-8 error="rpc error: code = Internal desc = grpc: error while marshaling: string field contains invalid UTF-8"

System Info

version "0.15.0-rc1"

version: '3'

services:
  server:
    image: woodpeckerci/woodpecker-server:latest
    volumes:
      - woodpecker-vol-1:/var/lib/woodpecker/
    restart: always
    ports:
      - "0.0.0.0:9000:9000"
      - "0.0.0.0:8000:8000"
    extra_hosts:
      - "git.friendi.ca:192.168.173.33"
    environment:
      - WOODPECKER_OPEN=true
      - WOODPECKER_HOST=https://ci.friendi.ca
      - WOODPECKER_GITHUB=true
      - WOODPECKER_GITHUB_CLIENT=XXXX
      - WOODPECKER_GITHUB_SECRET=XXXX
      - WOODPECKER_AGENT_SECRET=XXXX
      - WOODPECKER_ADMIN=XXXX,XXXX,XXXX,XXXX
      - WOODPECKER_REPO_OWNERS=XXXX,XXXX,XXXX,XXXX
      - WOODPECKER_DEBUG_PRETTY=true
      - WOODPECKER_DEBUG_NOCOLOR=false

  runner:
    image: woodpeckerci/woodpecker-agent:latest
    restart: always
    privileged: true
    userns_mode: host
    depends_on:
      - server
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    extra_hosts:
      - "git.friendi.ca:192.168.173.33"
    environment:
      - WOODPECKER_SERVER=server:9000
      - WOODPECKER_AGENT_SECRET=XXXX
      - WOODPECKER_MAX_PROCS=4
      - WOODPECKER_HOSTNAME=runner.ci.friendi.ca
      - WOODPECKER_DEBUG_PRETTY=true
      - WOODPECKER_DEBUG_NOCOLOR=false

  releaser:
    image: woodpeckerci/woodpecker-agent:latest
    restart: always
    privileged: true
    userns_mode: host
    depends_on:
      - server
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    extra_hosts:
      - "git.friendi.ca:192.168.173.33"
    environment:
      - WOODPECKER_SERVER=server:9000
      - WOODPECKER_AGENT_SECRET=XXXX
      - WOODPECKER_MAX_PROCS=2
      - WOODPECKER_HOSTNAME=releaser.ci.friendi.ca
      - WOODPECKER_PLATFORM=releaser/release 
      - WOODPECKER_DEBUG_PRETTY=true
      - WOODPECKER_DEBUG_NOCOLOR=false
      - WOODPECKER_ENVIRONMENT=node:releaser

volumes:
  woodpecker-vol-1:
    name: woodpecker_data

Additional context

No response

Validations

[X] Read the Contributing Guidelines.
[X] Read the docs.
[X] Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
[X] Check that this is a concrete bug. For Q&A join our Discord Chat Server.

anbraten commented 2 years ago

The spamming of * looks like some problem with the secrets filter which should prevent you from exposing them. We lately added some function to filter secrets on a new-line basis maybe that is the problem here. #671

The second error means there is somehow a problem when sending a log line from the agent to the server, but I am unsure where it comes from. Maybe some problem from the secrets filter as well.

anbraten commented 2 years ago

Could you provide us some information about the different types / formats of your secrets (e.g. ssh-key with new-lines, json format, ...)?

nupplaphil commented 2 years ago

codecov-token - push/pull_request - single

gpg_password - push/pull_request - single

gpg_key - push/pull_request - multi (rsa)

sftp_host - push/pull_request - single

sftp_user - push/pull_request - single

ssh_key - push/pull_request - multi (ec)

nupplaphil commented 2 years ago

jfyi: I deleted the multiline secrets and now it's working again

(Both errors are gone)

6543 commented 2 years ago

@nupplaphil can you tell us related multi line secrets (just replace chars with other chars ... ) so spaces & newline are preserved

6543 commented 2 years ago

PS: https://ci.friendi.ca/ is down

nupplaphil commented 2 years ago

I created a "fake" new key of GPG, having the same properties (pw-strength, rsa4096, ...) & length & export-type:

pgp_password: RVNre4#T>K<-&Gmw47.3JBHPfs/^<7}? gpg_key:

nupplaphil commented 2 years ago

both errors are definitely caused by the gpg_key secret. I tried removing/adding some the secrets.

The ssh_key is working (ed25519 ssh-key), so these multiline secret is working (created a new one as an example):

Maybe the additional newline of the gpg key (\n\n) ?

nupplaphil commented 2 years ago

yes, it's the \n\n!

Try this test at replace_secrets_test.go:

{
        log:     "start log\ndone\nnow\nan\nmulti line secret!! ;)",
        secrets: []string{"-----BEGIN PGP PRIVATE KEY BLOCK-----\n\nlQdGBGHkcwsB"},
        expect:  "start log\ndone\nnow\nan\nmulti line secret!! ;)",
}

Output:

Error Trace:    replace_secrets_test.go:35
            Error:          Not equal: 
                            expected: "start log\ndone\nnow\nan\nmulti line secret!! ;)"
                            actual  : "********s********t********a********r********t******** ********l********o********g********\n********d********o********n********e********\n********n********o********w********\n********a********n********\n********m********u********l********t********i******** ********l********i********n********e******** ********s********e********c********r********e********t********!********!******** ********;********)********"

                            Diff:
                            --- Expected
                            +++ Actual
                            @@ -1,5 +1,5 @@
                            -start log
                            -done
                            -now
                            -an
                            -multi line secret!! ;)
                            +********s********t********a********r********t******** ********l********o********g********
                            +********d********o********n********e********
                            +********n********o********w********
                            +********a********n********
                            +********m********u********l********t********i******** ********l********i********n********e******** ********s********e********c********r********e********t********!********!******** ********;********)********
            Test:           TestNewSecretsReplacer

woodpecker-ci / woodpecker