Downgrade to HTTP when there is no trailing /

[Wadeck]

Service(s)

jenkins.io

Summary

When we request a page on jenkins.io without a trailing slash, there is a redirection to HTTP, before another one to HTTPS.

Expected behavior Stay with HTTPS

Reproduction steps

When executing curl -v https://www.jenkins.io 2>&1 | grep -i location:, no output. But when there is a path:

$ curl -v https://www.jenkins.io/doc 2>&1 | grep -i location:
$ < location: http://www.jenkins.io/doc/

When browsing, the page is then redirected back to HTTPS.

[dduportal]

At first quick glance, it sounds like a setting in fastly as the server triggering the redirect is Varnish (Jenkins infra uses Nginx for jenkins.io):

$ curl -v https://www.jenkins.io -o /dev/null 2>&1 | grep -i via
< via: 1.1 varnish
$ curl -v https://www.jenkins.io/doc -o /dev/null 2>&1 | grep -i via
< via: 1.1 varnish

I don't know if it is a setting we can change but this should help to properly diagnose

[dduportal]

Should be fixed by https://github.com/jenkins-infra/fastly/pull/86

[dduportal]

Ok, now that SSL is enforced on Fastly's side, the problem should be gone.

BUT it is not: the redirect is defined on the original website and was cached by fastly:

$ curl -v https://www.origin.jenkins.io/doc -H 'Host: www.jenkins.io' 2>&1 | grep -i location:
< location: http://www.jenkins.io/doc/

(www.origin.jenkins.io hostname bypasses fastly and directly hits our publick8s cluster)

[dduportal]

The culprit is https://github.com/jenkins-infra/kubernetes-management/blob/01984688ec75b9e90c79071b0714c14a1afe3290/config/jenkinsio.yaml#L6

Fix proposal here: https://github.com/jenkins-infra/kubernetes-management/pull/4220

[dduportal]

Fix applied, but it seems like it does not work 🤦

Going through the rabbit hole and checking if the redirect isn't triggered internally inside the backend webserver (https://github.com/jenkins-infra/helm-charts/blob/main/charts/jenkinsio/templates/nginx-configmap.yaml)

[dduportal]

Update: it looks like the downgrade to HTTP is triggered inside the backend service:

# From inside one of the pods of jenkinsio in 'publick8s'
$ curl -H 'Host: www.jenkins.io' http://localhost:80/doc -o /dev/null -v 2>&1 | grep -i location:
< Location: http://www.jenkins.io/doc/

[dduportal]

For info, the ingress change caused https://github.com/jenkins-infra/datadog/pull/209 (minor issue in monitoring, no website issue)

[dduportal]

Found the issue: https://nginx.org/en/docs/http/ngx_http_core_module.html#absolute_redirect

(edited) In fact, when the (backend, not the ingress) Nginx server receieves the /doc request, it tries to serve the file doc in its document root.

Of course, there are no file named doc but a directory doc/. The (current) default behavior of Nginx with its current configuration is to issue a redirect to /doc/: it's a good default behavior because it avoids having to setup tedious rewrites to handle with or without the trailing slash.

But the problem is that the issued redirection defines an absolute URL as target: this is the why there is an http:// scheme as the (backend) only works with HTTP (TLS is terminated by the ingress, we don't have end-to-end TLS with mTLS so the backend does not have any knowledge that it should use the HTTPS scheme).

Setting the directive absolute_redirect inside the location / solves the issue: the redirection is still issued but with a relative path (e.g. /doc/ instead of http://www.jenkins.io/doc/) which solves the problem:

### Current setup, without the patched configuration
$ kubectl -n jenkinsio exec jenkinsio-786c7ddb69-n97jv -ti -- curl -v -o /dev/null -H 'Host: www.jenkins.io' http://localhost/doc 2>&1 | grep -i location
< Location: http://www.jenkins.io/doc/

### With the patched configuration
$ kubectl -n jenkinsio exec jenkinsio-786c7ddb69-q2tqp -ti -- curl -v -o /dev/null -H 'Host: www.jenkins.io' http://localhost/doc 2>&1 | grep -i location
< Location: /doc/

### With the pacthed configuration applied globally to the service in Azure:
$ curl -v -H 'Host: www.jenkins.io' https://www.origin.jenkins.io/doc 2>&1 | grep -i location:
< location: /doc/

### With the patched configuration applied globally to the service in Azure and in Fastly (with a cache purge on the URI `/doc`):
$ curl -v https://www.jenkins.io/doc 2>&1 | grep -i location:                                 
< location: /doc/

=> incoming PR to persist this patch

[dduportal]

Patch deployed: it works as expected!

$ curl -v https://www.jenkins.io/doc 2>&1 | grep -i location:
$ < /doc/