traefik: Woodpecker login via Gitea & OAuth2 times out

[justinmayer]

Component

server

Describe the bug

I have set up the Woodpecker server and agent using the Docker Compose configuration shown below. When I first went to the Woodpecker UI at https://ci.example.com and tapped on the Login button, I saw the OAuth2 confirmation page on Gitea and tapped the button to allow Woodpecker access. Indeed, I see that Woodpecker is listed under the Authorized OAuth2 Applications heading in the Gitea settings, so it seems that step completed successfully.

However, the login process in the browser seemed to take a long time, eventually showing an error: "Error while authenticating against OAuth provider"

Repeated login attempts always result in the same delay and ultimate error. See below for the corresponding error in the Docker Compose log. It is possible that this is related to a networking issue of some kind, but try as I might I cannot figure out how to resolve the problem.

System Info

Woodpecker version: 0.15.0

Docker Compose configuration (tap here to expand)

```yaml version: "3" services: traefik: image: traefik:v2.6 container_name: traefik restart: unless-stopped labels: - "traefik.enable=true" - "traefik.http.routers.api.rule=Host(`traefik.example.com`)" - "traefik.http.routers.api.entrypoints=https" - "traefik.http.routers.api.service=api@internal" - "traefik.http.routers.api.tls=true" - "traefik.http.routers.api.tls.certresolver=letsencrypt" command: - "--api" - "--api.dashboard=true" - "--api.insecure=true" - "--providers.docker=true" - "--providers.docker.exposedByDefault=false" - "--entrypoints.http=true" - "--entrypoints.http.address=:80" - "--entrypoints.http.http.redirections.entrypoint.to=https" - "--entrypoints.http.http.redirections.entrypoint.scheme=https" - "--entrypoints.https=true" - "--entrypoints.https.address=:443" - "--certificatesResolvers.letsencrypt.acme.email=security@example.com" - "--certificatesResolvers.letsencrypt.acme.storage=acme.json" - "--certificatesResolvers.letsencrypt.acme.httpChallenge.entryPoint=http" - "--log=true" - "--log.level=INFO" ports: - "80:80" - "443:443" - "8080:8080" networks: - internal_network - external_network volumes: - ./traefik/acme.json:/acme.json - /var/run/docker.sock:/var/run/docker.sock gitea: image: gitea/gitea:1.16 container_name: gitea restart: unless-stopped depends_on: traefik: condition: service_started db: condition: service_started redis: condition: service_healthy environment: - APP_NAME="Gitea" - RUN_MODE=prod - DOMAIN=code.example.com - SSH_DOMAIN=code.example.com - HTTP_PORT=3000 - ROOT_URL=https://code.example.com - START_SSH_SERVER=true - SSH_PORT=222 - SSH_LISTEN_PORT=22 - DEFAULT_BRANCH=main - GITEA__database__DB_TYPE=postgres - GITEA__database__HOST=db:5432 - GITEA__database__NAME=gitea - GITEA__database__USER=postgres - GITEA__database__PASSWD= - GITEA__cache__ENABLED=true - GITEA__cache__ADAPTER=redis - GITEA__cache__HOST=redis://redis:6379/0?pool_size=100&idle_timeout=180s - GITEA__cache__ITEM_TTL=24h - GITEA__webhook__ALLOWED_HOST_LIST=* labels: - "traefik.enable=true" - "traefik.http.routers.gitea.rule=Host(`code.example.com`)" - "traefik.http.routers.gitea.entrypoints=https" - "traefik.http.routers.gitea.tls.certresolver=letsencrypt" - "traefik.http.routers.gitea.service=gitea-service" - "traefik.http.services.gitea-service.loadbalancer.server.port=3000" ports: - "222:22" networks: - internal_network volumes: - gitea-data:/data - /etc/timezone:/etc/timezone:ro - /etc/localtime:/etc/localtime:ro woodpecker-server: image: woodpeckerci/woodpecker-server:latest container_name: woodpecker-server restart: unless-stopped depends_on: gitea: condition: service_started environment: - WOODPECKER_OPEN=true - WOODPECKER_ORGS=code - WOODPECKER_ADMIN=jane,alice - WOODPECKER_REPO_OWNERS=my-org - WOODPECKER_HOST=https://ci.example.com - WOODPECKER_GITEA=true - WOODPECKER_GITEA_URL=https://code.example.com - WOODPECKER_GITEA_CLIENT= - WOODPECKER_GITEA_SECRET= - WOODPECKER_AGENT_SECRET= labels: - "traefik.enable=true" - "traefik.http.routers.woodpecker.rule=Host(`ci.example.com`)" - "traefik.http.routers.woodpecker.entrypoints=https" - "traefik.http.routers.woodpecker.tls.certresolver=letsencrypt" - "traefik.http.routers.woodpecker.service=woodpecker-service" - "traefik.http.services.woodpecker-service.loadbalancer.server.port=8000" networks: - internal_network volumes: - woodpecker-server-data:/var/lib/woodpecker/ woodpecker-agent: image: woodpeckerci/woodpecker-agent:latest command: agent container_name: woodpecker-agent restart: unless-stopped depends_on: woodpecker-server: condition: service_started environment: - WOODPECKER_SERVER=woodpecker-server:9000 - WOODPECKER_AGENT_SECRET= networks: - internal_network volumes: - /var/run/docker.sock:/var/run/docker.sock db: […] redis: […] volumes: db: gitea-data: woodpecker-server-data: networks: external_network: internal_network: internal: true ```

Additional context

The Docker Compose logs show the attempted connection to Gitea and its resulting error output:

woodpecker-server  | {"level":"error","message":"cannot authenticate user. Post \"https://code.example.com/login/oauth/access_token\": dial tcp XXX.XXX.XXX.XXX:443: i/o timeout"}

### Validations

- [X] Read the [Contributing Guidelines](https://github.com/woodpecker-ci/woodpecker/blob/master/CONTRIBUTING.md).
- [X] Read the [docs](https://woodpecker-ci.org/docs/intro).
- [X] Check that there isn't [already an issue](https://github.com/woodpecker-ci/woodpecker/issues) that reports the same bug to avoid creating a duplicate.
- [X] Check that this is a concrete bug. For Q&A join our [Discord Chat Server](https://discord.gg/fcMQqSMXJy).

[6543]

I think this is an issue of how your reverse proxy works

woodpecker like to access via external url but can not since it will be directed to internal without ssl

that's my guess

[Flying--Dutchman]

~Facing the same issue without using a proxy in my compose file. Worked fine with an existing installation, but not anymore after I tried to revert to another version of woodpecker because of another issue.~

Woodpecker: v0.15.0-rc1 Gitea: 1.16.4

Edit: Issue doesn't appear anymore on my system

[IDerr]

Doesn't happens on my side, everything seems ok 👍

[justinmayer]

@IDerr / @Flying--Dutchman: Would you mind putting your working Docker Compose configuration in a pastebin and posting a link here so I can better understand what is required to make this work properly?

[Flying--Dutchman]

I'm currently on vacation but if needed I could post a pastebin link next Saturday.

[justinmayer]

@6543: Thank you for taking a look and offering a guess as to what is going on here. You said:

I think this is an issue of how your reverse proxy works

I can't tell whether this is specific to Traefik or instead caused by how Docker handles networking. Either way, I imagine that Traefik is a popular enough proxy that it should (eventually?) be fully supported by Woodpecker in a Docker Compose context.

woodpecker like to access via external url but can not since it will be directed to internal without ssl

I think this aspect of Woodpecker's current behavior — using WOODPECKER_GITEA_URL for all communication with Gitea — is a significant factor in causing this problem. If there were a separate WOODPECKER_GITEA_HOST setting for making the POST to /login/oauth/access_token, then the internal Docker service name (i.e., gitea) could be specified to ensure problem-free communication between the two containers.

What do you think?

[6543]

that's a idea - to have a variable for overwriting a public domain with a private address - since it would also have to alter all api requests

[6543]

@justinmayer -> #891

[Flying--Dutchman]

@justinmayer My compose files looks something like the following.

version: '3'

networks:
  git_tst_network:
    external: false

services:
# #################################################################################
# ############################ SETUP GITEA DB #####################################
# #################################################################################
  git_tst_gitea_db:
    image: mariadb:${MARIADB_TAG}
    restart: unless-stopped
    networks:
      - git_tst_network
    environment:
      - MARIADB_RANDOM_ROOT_PASSWORD=true
      - MARIADB_DATABASE=gitea_git
      - MARIADB_USER=gitea_git_usr
      - MARIADB_PASSWORD=${GITEA_DB_PASSWORD}
    volumes:
      - git_tst_gitea_db_data:/var/lib/mysql

# #################################################################################
# ############################ SETUP GITEA ########################################
# #################################################################################
  git_tst_gitea:
    image: gitea/gitea:${GITEA_TAG}
    restart: unless-stopped
    networks:
      - git_tst_network
    environment:
      - USER_UID=${GITEA_UID}
      - USER_GID=${GITEA_GID}
      - GITEA__database__DB_TYPE=mysql
      - GITEA__database__HOST=git_tst_gitea_db:3306
      - GITEA__database__NAME=gitea_git
      - GITEA__database__USER=gitea_git_usr
      - GITEA__database__PASSWD=${GITEA_DB_PASSWORD}
    volumes:
      - git_tst_gitea_data:/data
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
    ports:
      - ${GITEA_PORT}:3000
      - ${GITEA_PORT_SSH}:22
    depends_on:
      - git_tst_database

# #################################################################################
# ############################ SETUP WOODPECKER DB ################################
# #################################################################################
  git_tst_woodpecker_db:
    image: mariadb:${MARIADB_TAG}
    restart: unless-stopped
    networks:
      - git_tst_network
    environment:
      - MARIADB_RANDOM_ROOT_PASSWORD=true
      - MARIADB_DATABASE=woodpeckergit
      - MARIADB_USER=woodpecker_git_usr
      - MARIADB_PASSWORD=${WOODPECKER_DB_PASSWORD}
    volumes:
      - git_tst_woodpecker_db_data:/var/lib/mysql

# #################################################################################
# ############################ SETUP WOODPECKER CI ################################
# #################################################################################
  git_tst_woodpecker_ci:
    image: woodpeckerci/woodpecker-server:${WOODPECKER_TAG}
    restart: unless-stopped
    user: ${WOODPECKER_UID}:${WOODPECKER_GID}
    ports:
      - ${WOODPECKER_PORT}:8000
      - 9000
    networks:
      - git_tst_network
    environment:
      - WOODPECKER_DATABASE_DRIVER=mysql
      - WOODPECKER_DATABASE_DATASOURCE=woodpecker_git_usr:${WOODPECKER_DB_PASSWORD}@tcp(git_tst_woodpecker_db:3306)/woodpeckergit?parseTime=true
      - WOODPECKER_OPEN=true
      - WOODPECKER_HOST=${WOODPECKER_URL}
      - WOODPECKER_GITEA=true
      - WOODPECKER_GITEA_URL=${GITEA_URL}
      - WOODPECKER_GITEA_CLIENT=...
      - WOODPECKER_GITEA_SECRET=...
      - WOODPECKER_AGENT_SECRET=${WOODPECKER_SECRET}
      - GIN_MODE=release
    depends_on:
      - git_tst_woodpecker_db

# #################################################################################
# ############################ SETUP WOODPECKER AGENT #############################
# #################################################################################
  git_tst_woodpecker_agent:
    image: woodpeckerci/woodpecker-agent:${WOODPECKER_AGENT_TAG}
    restart: unless-stopped
    user: ${UID}:${GID}
    command: agent
    networks:
      - git_tst_network
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    environment:
      - WOODPECKER_SERVER=git_tst_woodpecker_ci:9000
      - WOODPECKER_AGENT_SECRET=${WOODPECKER_SECRET}
    depends_on:
      - git_woodpecker_tst_server

volumes:
    ...

[Blackskyliner]

@6543

that's a idea - to have a variable for overwriting a public domain with a private address - since it would also have to alter all api requests

Which should not only encompass the OAUTH but also everything around cloning from repository and many more and could make configuration more confusing.

Because the real problem with server+agent+gitea on the same host is that docker does not let us talk over the external ip (in regards to using a domain/dns resolving to our external ip) with each other (at least in default/secure ip stack and firewall configuration).

Especially the shown configuration, of issue creator, uses a reverse-proxy for all services, as I would also do because lets-encrypt and maybe more than just those services on the host and so on...

I also have the same problem in my setup.

I have a proxy-network with static subnet address (192.168.206.0/24) for my front-facing service (192.168.206.2) (nginx-proxy instead of traefik) which has git.domain.tld configured to point to the gitea instance.

Now I can add my server and agent to my proxy-network and via extra_hosts can also map the external name (git.domain.tld) to the internal ip (192.168.206.2) but only for those containers (server+agent).

With this setup you can circumvent the oauth problems and get going, everything seems to work, because now we have normal https routing through our nginx-proxy which correctly resolves to the gitea...

until... the spawned docker containers from the CI process come around and can't clone the sources again, because it uses (and must use it, to not confuse the user in the logs) the public facing URI (git.domain.tld/User/Repo.git) of the repository and in my setup it failes because the extra_hosts and proxy-network for the ci spawned task container is missing. The proxy-network I could add (or alltogether pack everything in a ci-network with static subnet and stuff) via docker backend configuration BUT I can't get the extra_hosts to the spawned container introducing my dns name (git.domain.tld) as internal host to them.

I have the feeling, that my solution is more involved around "how to docker network" but feels cleaner alltogether and you won't have to wield different names (internal/extenal) in general configuration. But question is, if it is feasable to read the extra_hosts and also put them in the agent spawned containers OR if it should be a static docker backend configuration.

What do you think? I hope I get around to create a minimal example compose about what I mean with all that. (if interested)

[haveachin]

I got a workaround working with Traefik. In your Woodpecker docker-compose.yml add this extra_host entry that makes Woodpecker directly resolve gitea.example.com to the internal Traefik IPv4 address:

version: "3.9"

services:
  woodpecker-server:
    ...
    networks:
      - traefik
      - woodpecker
    extra_hosts:
      # 172.0.16.100 is my Traefiks IPv4 address
      - "gitea.example.com:172.0.16.100"

networks:
  woodpecker:
    name: woodpecker
  traefik:
    external: true

In your Traefik docker-compose.yml you should probably hard code its IPv4 address and subnet so it doesn't change:

version: "3.9"

services:
  traefik:
    ...
    networks:
      traefik:
        # 172.0.16.100 is my Traefiks IPv4 address
        ipv4_address: 172.0.16.100

networks:
  traefik:
    driver: bridge
    ipam:
      config:
        # 172.0.16.0/24 is my Traefiks subnet
        - subnet: 172.0.16.0/24

You can look up your Traefiks network with docker network inspect traefik.

[MB175]

I got a workaround working with Traefik. In your Woodpecker docker-compose.yml add this extra_host entry that makes Woodpecker directly resolve gitea.example.com to the internal Traefik IPv4 address:

version: "3.9"

services:
  woodpecker-server:
    ...
    networks:
      - traefik
      - woodpecker
    extra_hosts:
      # 172.0.16.100 is my Traefiks IPv4 address
      - "gitea.example.com:172.0.16.100"

networks:
  woodpecker:
    name: woodpecker
  traefik:
    external: true

In your Traefik docker-compose.yml you should probably hard code its IPv4 address and subnet so it doesn't change:

version: "3.9"

services:
  traefik:
    ...
    networks:
      traefik:
        # 172.0.16.100 is my Traefiks IPv4 address
        ipv4_address: 172.0.16.100

networks:
  traefik:
    driver: bridge
    ipam:
      config:
        # 172.0.16.0/24 is my Traefiks subnet
        - subnet: 172.0.16.0/24

You can look up your Traefiks network with docker network inspect traefik.

Thanks @haveachin , this should be added to the docs

[haveachin]

sanjibukai#9767 on the Discord showed me an even better solution: In your Woodpeckers docker-compose.yml:

version: "3.9"

services:
  woodpecker-server:
    image: woodpeckerci/woodpecker-server:next # Use the next version
    ...
    networks:
      - woodpecker
      - traefik

  woodpecker-agent:
    image: woodpeckerci/woodpecker-agent:next # Use the next version
    ...
    environment:
      ...
      - WOODPECKER_BACKEND_DOCKER_NETWORK=traefik # Joins all workers automatically in the traefik network
    networks:
      - woodpecker
      - traefik

networks:
  woodpecker:
    name: woodpecker
  traefik:
    external: true

And in your Traefiks docker-compose.yml:

version: "3"

services:
  traefik:
    ...
    networks:
      traefik:
        aliases: # This will create host entries for the IP of the Traefik container in the traefik network
          - gitea.example.com
          - woodpecker.example.com

networks:
  traefik:
    name: traefik

[6543]

https://github.com/woodpecker-ci/woodpecker/blob/master/docs/docs/30-administration/70-proxy.md#traefik

pulls are welcome :heart:

[anbraten]

PRs to the docs are still welcome otherwise closing for now