Build logs sent by syslog drainer contain unredacted secrets

[robtaylorsky]

Summary

When configuring the syslog drainer to send Concourse build logs to a syslog endpoint, and using CONCOURSE_ENABLE_REDACT_SECRETS=true, the build logs that are sent contain unredacted pipeline secrets.

Steps to reproduce

docker-compose.yaml:

version: '3'

services:
  concourse-db:
    image: postgres
    environment:
      POSTGRES_DB: concourse
      POSTGRES_PASSWORD: concourse_pass
      POSTGRES_USER: concourse_user
      PGDATA: /database

  concourse:
    image: concourse/concourse
    platform: linux/amd64
    command: quickstart
    privileged: true
    depends_on: [concourse-db]
    ports: ["8080:8080"]
    environment:
      CONCOURSE_POSTGRES_HOST: concourse-db
      CONCOURSE_POSTGRES_USER: concourse_user
      CONCOURSE_POSTGRES_PASSWORD: concourse_pass
      CONCOURSE_POSTGRES_DATABASE: concourse
      CONCOURSE_EXTERNAL_URL: http://localhost:8080
      CONCOURSE_ADD_LOCAL_USER: test:test
      CONCOURSE_MAIN_TEAM_LOCAL_USER: test
      # If running on a non M1/M2 Mac, change this to overlay
      CONCOURSE_WORKER_BAGGAGECLAIM_DRIVER: naive
      CONCOURSE_WORKER_BAGGAGECLAIM_DISABLE_USER_NAMESPACES: true
      CONCOURSE_CLIENT_SECRET: Y29uY291cnNlLXdlYgo=
      CONCOURSE_TSA_CLIENT_SECRET: Y29uY291cnNlLXdvcmtlcgo=
      CONCOURSE_X_FRAME_OPTIONS: allow
      CONCOURSE_CONTENT_SECURITY_POLICY: "*"
      CONCOURSE_CLUSTER_NAME: tutorial
      # If running on a non M1/M2 Mac, change this to containerd
      CONCOURSE_WORKER_RUNTIME: "houdini"
      # Syslog drainer configuration (default drain interval: 30s)
      CONCOURSE_SYSLOG_ADDRESS: vector:601
      CONCOURSE_SYSLOG_TRANSPORT: "tcp"
      # Credential manager
      CONCOURSE_VAULT_URL: http://vault:8200
      CONCOURSE_VAULT_CLIENT_TOKEN: root
      CONCOURSE_VAULT_PATH_PREFIX: /concourse
      CONCOURSE_VAULT_LOOKUP_TEMPLATES: /{{.Team}}/{{.Secret}}
      CONCOURSE_ENABLE_REDACT_SECRETS: true

  vector:
    image: timberio/vector:0.40.1-debian
    container_name: vector
    volumes:
      - ./vector.yaml:/etc/vector/vector.yaml:ro
      - ./logs/concourse_builds.log:/tmp/concourse_builds.log:rw
    ports:
      - "601:601"

  vault:
    hostname: vault
    container_name: vault
    image: hashicorp/vault:1.17
    environment:
      VAULT_ADDR: "http://0.0.0.0:8200"
      VAULT_API_ADDR: "http://0.0.0.0:8200"
    ports:
      - "8200:8200"
    cap_add:
      - IPC_LOCK
    entrypoint: vault server -dev -dev-listen-address="0.0.0.0:8200" -dev-root-token-id="root"

vector.yaml:

api:
  enabled: true
  address: 0.0.0.0:8686
sources:
  concourse_build_logs:
    type: syslog
    address: 0.0.0.0:601
    mode: tcp
    path: /tmp/test.sock
sinks:
  syslog_file:
    type: file
    inputs:
      - concourse_build_logs
    path: /tmp/concourse_builds.log
    encoding:
      codec: text

Steps:

# Place docker-compose.yaml and vector.yaml in directory
# Setup the vector output logfile
mkdir logs && touch logs/concourse_builds.log

# Startup containers
docker-compose up -d

# Create test secret
docker exec vault vault secrets enable -path /concourse -version=2 kv
docker exec vault vault kv put -mount=concourse main/password value=VerySecret!

# Create and run test pipeline
cat <<EOF > hello-world.yml
---
jobs:
- name: hello-world
  public: true
  plan:
  - task: simple-task
    config:
      platform: linux
      image_resource:
        type: registry-image
        source: { repository: busybox }
      run:
        path: echo
        args: ["Hello world! My password is ((password))"]
EOF

fly -t tutorial login -u test -p test -c http://localhost:8080
fly -t tutorial set-pipeline -p hello-world -c hello-world.yml
fly -t tutorial unpause-pipeline -p hello-world
fly -t tutorial trigger-job -j hello-world/hello-world

# Wait a few seconds (drain interval), then check the build logs sent by the syslog drainer
cat logs/concourse_builds.log
started
task initializing
check initializing image
selected worker: 86638e5e03a8
get initializing
selected worker: 86638e5e03a8
fetching busybox@sha256:82742949a3709938cbeb9cec79f5eaf3e48b255389f2dcedf2de29ef96fd841c
 3d1a87f2317d [==========================================] 2.1MiB/2.1MiB
get {"version": {"digest":"sha256:82742949a3709938cbeb9cec79f5eaf3e48b255389f2dcedf2de29ef96fd841c","tag":"1.36.1"}, "metadata": [{"name":"repository","value":"busybox"},{"name":"tag","value":"1.36.1"}]
selected worker: 86638e5e03a8
running echo Hello world! My password is VerySecret!
Hello world! My password is ((redacted))
succeeded

Expected results

The build log sent by the syslog drainer shouldn't contain the line running echo Hello world! My password is VerySecret!. Or otherwise, the secret should be redacted in this line.

Actual results

See steps to reproduce. We've seen this behaviour on our AWS EC2 Linux test environment as well as in this example using Docker on a Mac.

Triaging info

• Concourse version: 7.11.2
• Browser (if applicable):
• Did this used to work?

[marco-m]

To be 100% sure: with this configuration, are the secrets redacted as expected in the normal build log (in the web UI or from fly) ?

[robtaylorsky]

Yes. They are redacted in the web UI:

But if I use fly watch I see the same behaviour:

[robtaylorsky]

I raised this bug as I'm investigating using the syslog drainer functionality to enable log aggregation for a multi-tenant Concourse cluster with many teams.

In the case of fly watch - displaying unredacted secrets is perhaps ok. The user here will have permissions on the team the pipeline is set for - and most likely would have access to these secrets anyway in their secrets manager.

In the case of the syslog drainer - where logs are being sent to a different storage location and permissions may differ - it would be good if the build logs didn't contain unredacted secrets.

[marco-m]

In the case of the syslog drainer - where logs are being sent to a different storage location and permissions may differ - it would be good if the build logs didn't contain unredacted secrets.

I agree :-)

[marco-m-pix4d]

Seems to be related to #8477.