Problems with connecting Woodpecker server with Gitea via Oauth2 (Docker on the same host)

ky4n commented 1 year ago

Component

server

Describe the bug

Short version: Issues with oauth setup between Gitea and Woodpecker - /authorize?code endpoint times out on response from Gitea.

Long version: I've set up Gitea and Woodpecker in Docker containers and I cannot get the authorization to work. Both services run in Docker on the same instance (VM). Both services are accessible via HTTPS through Nginx proxy, with valid TLS certificates. The domain is "internal", resolving to VPN interface of the instance.

Gitea:

fresh deployment with mostly default settings
1 admin user and 1 (empty) repo
Oauth "woodpecker" application created for user
ALLOWED_HOST_LIST added during troubleshooting, but probably unnecessary

Relevant (?) configuration part:

DOMAIN           = gitea.my.fancy.domain
SSH_DOMAIN       = gitea.my.fancy.domain
ROOT_URL         = https://gitea.my.fancy.domain/
OFFLINE_MODE     = true

[webhook]
ALLOWED_HOST_LIST=external, loopback

Woodpecker:

fresh server deployment
env settings:

WOODPECKER_OPEN="true"
WOODPECKER_HOST="https://woodpecker.my.fancy.domain"
WOODPECKER_AGENT_SECRET="A_SECRET"
# debugging
WOODPECKER_LOG_LEVEL="trace"
WOODPECKER_DEBUG_PRETTY="true"
WOODPECKER_DEBUG_NOCOLOR="true"
# Gitea config
WOODPECKER_GITEA="true"
WOODPECKER_GITEA_URL="https://gitea.my.fancy.domain"
WOODPECKER_GITEA_CLIENT="GITEA_CLIENT_UUID_HERE"
WOODPECKER_GITEA_SECRET="gto_VALID_GITEA_SECRET_HERE"

I'm logged into Gitea in the same browser window.

When I open Woodpecker main page and click on the log in button, the following happens:

first time only, I get redirected to Gitea page (it opens), where I authorize the app
there's a number of 303 redirects
it culminates in /authorize?code... request which times out after 60 seconds

Web server (Nginx) logs show properly passing the request to Woodpecker and (with trace logging) there's a log line acknowledging it reached the Woodpecker service itself, but the service is not giving any reply. After ~ 60 seconds the the request times out giving HTTP/504.

I've attached some parsed logs too.

System Info

- Gitea Docker image: `gitea/gitea:1.18.1`
- Woodpecker Docker image: `woodpeckerci/woodpecker-server:latest`

{"source":"https://github.com/woodpecker-ci/woodpecker","version":"0.15.6"}

Additional context

Browser networking logs:

GET https://woodpecker.my.fancy.domain/login?url=  -> 303
GET https://woodpecker.my.fancy.domain/authorize?url= -> 303
GET https://gitea.my.fancy.domain/login/oauth/authorize?client_id=GITEA_CLIENT_UUID_HERE&redirect_uri=https%3A%2F%2Fwoodpecker.my.fancy.domain%2Fauthorize&response_type=code&state=woodpecker -> 303
GET https://woodpecker.my.fancy.domain/authorize?code=gta_SOME_RANDOM_CHARACTERS&state=woodpecker -> 504

Parsed web server logs, mainly for timestamp correlation (they match browser times):

13:10:07 Request:"GET /login?url= HTTP/1.0" Status:303 Referer:"https://woodpecker.my.fancy.domain/do-login?url=/repos"
13:10:07 Request:"GET /authorize?url= HTTP/1.0" Status:303 Referer:"https://woodpecker.my.fancy.domain/do-login?url=/repos"
13:10:07 Request:"GET /login/oauth/authorize?client_id=GITEA_CLIENT_UUID_HERE&redirect_uri=https%3A%2F%2Fwoodpecker.my.fancy.domain%2Fauthorize&response_type=code&state=woodpecker HTTP/1.0" Status:303 Referer:"https://woodpecker.my.fancy.domain/"
13:11:07 Request:"GET /authorize?code=gta_SOME_RANDOM_CHARACTERS&state=woodpecker HTTP/1.0" Status:504 Referer:"https://woodpecker.my.fancy.domain/"

Gitea log:

13:10:07 [63cbe42f] router: completed GET /login/oauth/authorize?client_id=GITEA_CLIENT_UUID_HERE&redirect_uri=https%3A%2F%2Fwoodpecker.my.fancy.domain%2Fauthorize&response_type=code&state=woodpecker for MY_IP_ADDRESS:0, 303 See Other in 11.5ms @ auth/oauth.go:361(auth.AuthorizeOAuth)

Woodpecker trace log:

1:10PM TRC woodpecker/src/github.com/woodpecker-ci/woodpecker/server/router/router.go:38 > [GET] /login?url=
1:10PM INF woodpecker/src/github.com/woodpecker-ci/woodpecker/server/router/middleware/logger.go:45 >  ip=MY_IP_ADDRESS latency=0.08254 method=GET path=/login status=303 user-agent="MY USER AGENT"
1:10PM TRC woodpecker/src/github.com/woodpecker-ci/woodpecker/server/router/router.go:38 > [GET] /authorize?url=
1:10PM INF woodpecker/src/github.com/woodpecker-ci/woodpecker/server/router/middleware/logger.go:45 >  ip=MY_IP_ADDRESS latency=0.166481 method=GET path=/authorize status=303 user-agent="MY USER AGENT"
1:10PM TRC woodpecker/src/github.com/woodpecker-ci/woodpecker/server/router/router.go:38 > [GET] /authorize?code=gta_SOME_RANDOM_CHARACTERS&state=woodpecker
1:14PM ERR woodpecker/src/github.com/woodpecker-ci/woodpecker/server/api/login.go:59 > cannot authenticate user. Post "https://gitea.my.fancy.domain/login/oauth/access_token": dial tcp MY_IP_ADDRESS:443: connect: connection timed out
1:14PM INF woodpecker/src/github.com/woodpecker-ci/woodpecker/server/router/middleware/logger.go:45 >  ip=MY_IP_ADDRESS latency=260590.451521 method=GET path=/authorize status=303 user-agent="MY USER AGENT"

While there are no seconds in Woodpecker timestamp this line is logged shortly after all the other requests: TRC ...:38 > [GET] /authorize?code=gta_SOME_RANDOM_CHARACTERS&state=woodpecker

Afterwards the request hangs for about a minute in the browser and times out.

Validations

[X] Read the Contributing Guidelines.
[X] Read the docs.
[X] Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
[X] Check that this is a concrete bug. For Q&A join our Discord Chat Server or the Matrix room.

smainz commented 1 year ago

Can you check that the woodpecker container canconnect to https://gitea.my.fancy.domain/?

Log into the woodpecker container
nslookup gitea.my.fancy.domain
ping gitea.my.fancy.domain

The log indicates that this is not the case and woodpecker cannot verify the token provided in the URL. May be related to your VPN setup and you will have to provide a DNS entry to the woodpecker container for gitea.

Admicos commented 1 year ago

Here's the hacky way I just managed to solve this issue:

Use the public URL of both your Gitea & Woodpecker instances

    woodpecker-server:
        environment:
            WOODPECKER_HOST: https://ci.example.com
            WOODPECKER_GITEA_URL: https://git.example.com

Alias those domains to your reverse proxy container (I did say this was hacky)

Make sure to use the correct network name. Mine is web

    caddy:
        networks:
            web:
                aliases:
                  - ci.example.com
                  - git.example.com

???
Profit

ky4n commented 1 year ago

OK, after taking a break and coming back to it with a fresh mind, I've solved it on my own. TL;DR: firewall

I will add some more information in case someone finds it useful in the future.

First, clarification: MY_IP_ADDRESS was the IP address of the host running Docker containers, not the client (my browser).

Second, I did not assume it was a firewall because the error was happening during Oauth authorization phase and I assumed the only communication then is browser <-> Gitea and browser <-> Woodpecker. That part worked fine all the way.

Small note, Woodpecker server Docker image builds FROM scratch so @smainz comment won't actually work directly. I did use a different container connected the same way as Woodpecker server to test the connectivity (and spot the root cause).

Regarding @Admicos comment](https://github.com/woodpecker-ci/woodpecker/issues/1560#issuecomment-1399327364) and general Woodpecker - Gitea connectivity it took me a while to build a valid mental model of the communication. There was another issue I stumbled upon earlier which basically said Woodpecker uses only 1 URL for Gitea, so no way to have two: one for feeding to the users (browser) directly and another for communication between Woodpecker and Gitea. Side note: this might be a good idea, if Docker is one of the "blessed" ways of setting up Woodpecker.

I've actually addressed the potential DNS issue by having domains for both Woodpecker & Gitea resolve to the same IP address for containers and users (browsers). This allowed me to use the same HTTPS URL in Woodpecker configuration and in the browser. Basically, both gitea.my.fancy.domain & woodpecker.my.fancy.domain pointing to MY_IP_ADDRESS (port 443 covered by Nginx with a publicly valid TLS certificate).

At the time I was troubleshooting this, the firewall however was only allowing connections to `MY_IP_ADDRESS on the VPN network interface, not on the (bridge) network interface to which both containers were plugged.

To sum up:

Running Gitea & Woodpecker on the same host in Docker containers can be problematic. Official docs don't cover all possible setups (K8s, Docker compose, other orchestrators), but I don't good improvement suggestions here, sorry. I needed to make sure I use the same domain & port for internal (Gitea-Woodpecker) and external (browser to both) communication, as well as making sure the containers can talk with each other directly (firewall in my case) and do it over the same URL as the browser does.

I 'll close this issue, thanks for your time spent on this.

woodpecker-ci / woodpecker