🐛 [BUG] - service-proxy messes up with the api server path in the URL

[richardtief]

Priority

(Medium) I'm annoyed but I'll live

Description

The service-proxy logs:

2024-08-02T14:06:05Z    INFO    Forwarded request 
{"host": "plutono--ccloud--obs-eu-nl-1.ccloud.greenhouse-qa.eu-nl-1.cloud.sap", 
"url": "/api/v1/namespaces/kube-monitoring/services/http:plutono:80/proxy/public/build/angular~app.3851969a931d8bee6112.js", 
"method": "GET", 
"cluster": "obs-eu-nl-1", 
"namespace": "ccloud", 
"name": "plutono", 
"status": 401, 
"upstream": "https://api.my-cluster/api/v1/namespaces/kube-monitoring/services/http:plutono:80/proxy/api/v1/namespaces/kube-monitoring/services/http:plutono:80/proxy/public/build/angular~app.3851969a931d8bee6112.js"
}

plutono logs: (look at the messed up path)

t=2024-08-02T14:06:04+0000 lvl=info msg="Request Completed" logger=context userId=0 orgId=0 uname= method=GET path=/ status=302 remote_addr=10.180.1.234 time_ms=0 size=29 referer=
t=2024-08-02T14:06:05+0000 lvl=info msg="Request Completed" logger=context userId=0 orgId=0 uname= method=GET path=/api/v1/namespaces/kube-monitoring/services/http:plutono:80/proxy/public/build/plutono.dark.3851969a931d8bee6112.css status=401 remote_addr=10.180.1.234 time_ms=0 size=26 referer=https://plutono--ccloud--obs-eu-nl-1.ccloud.greenhouse-qa.eu-nl-1.cloud.sap/login
t=2024-08-02T14:06:05+0000 lvl=info msg="Request Completed" logger=context userId=0 orgId=0 uname= method=GET path=/api/v1/namespaces/kube-monitoring/services/http:plutono:80/proxy/public/build/runtime.3851969a931d8bee6112.js status=401 remote_addr=10.180.1.234 time_ms=0 size=26 referer=https://plutono--ccloud--obs-eu-nl-1.ccloud.greenhouse-qa.eu-nl-1.cloud.sap/login
t=2024-08-02T14:06:05+0000 lvl=info msg="Request Completed" logger=context userId=0 orgId=0 uname= method=GET path=/api/v1/namespaces/kube-monitoring/services/http:plutono:80/proxy/public/build/app.3851969a931d8bee6112.js status=401 remote_addr=10.180.1.234 time_ms=0 size=26 referer=https://plutono--ccloud--obs-eu-nl-1.ccloud.greenhouse-qa.eu-nl-1.cloud.sap/login
t=2024-08-02T14:06:05+0000 lvl=info msg="Request Completed" logger=context userId=0 orgId=0 uname= method=GET path=/api/v1/namespaces/kube-monitoring/services/http:plutono:80/proxy/public/build/moment~app.3851969a931d8bee6112.js status=401 remote_addr=10.180.1.234 time_ms=0 size=26 referer=https://plutono--ccloud--obs-eu-nl-1.ccloud.greenhouse-qa.eu-nl-1.cloud.sap/login
t=2024-08-02T14:06:05+0000 lvl=info msg="Request Completed" logger=context userId=0 orgId=0 uname= method=GET path=/api/v1/namespaces/kube-monitoring/services/http:plutono:80/proxy/public/build/vendors~app.3851969a931d8bee6112.js status=401 remote_addr=10.180.1.234 time_ms=0 size=26 referer=https://plutono--ccloud--obs-eu-nl-1.ccloud.greenhouse-qa.eu-nl-1.cloud.sap/login
t=2024-08-02T14:06:05+0000 lvl=info msg="Request Completed" logger=context userId=0 orgId=0 uname= method=GET path=/api/v1/namespaces/kube-monitoring/services/http:plutono:80/proxy/public/build/unicons~app.3851969a931d8bee6112.js status=401 remote_addr=10.180.1.234 time_ms=0 size=26 referer=https://plutono--ccloud--obs-eu-nl-1.ccloud.greenhouse-qa.eu-nl-1.cloud.sap/login
t=2024-08-02T14:06:05+0000 lvl=info msg="Request Completed" logger=context userId=0 orgId=0 uname= method=GET path=/api/v1/namespaces/kube-monitoring/services/http:plutono:80/proxy/public/build/angular~app.3851969a931d8bee6112.js status=401 remote_addr=10.180.1.234 time_ms=0 size=26 referer=https://plutono--ccloud--obs-eu-nl-1.ccloud.greenhouse-qa.eu-nl-1.cloud.sap/login

Reproduction steps

Goto https://plutono--ccloud--obs-eu-nl-1.ccloud.greenhouse-qa.eu-nl-1.cloud.sap

Manifests

No response

Screenshots

Plutono can't be loaded.

[IvoGoman]

@uwe-mayer can you help here?

[uwe-mayer]

The following

In Plutonos index.html a HTML <base> tag is used. This serves as base for relative URLs in the html document. Checking a plutono behind the service-proxy this looks like this:

<base href="/api/v1/namespaces/kube-monitoring/services/http:plutono-obs-eu-de-1:80/proxy/"/>

Then again checking the headers of the service-proxy we see the following through the exposed-service debug plugin:

"headers": {
    "host": "api.obs-eu-de-1.greenhouse.shoot.canary.k8s-hana.ondemand.com",
    ...
    "upgrade-insecure-requests": "1",
    "x-forwarded-for": "100.64.1.13, 10.64.64.116",
    "x-forwarded-host": "exposed-service-obs-eu-de-1--ccloud--obs-eu-de-1.ccloud.greenhouse-qa.eu-nl-1.cloud.sap",
    "x-forwarded-port": "443",
    "x-forwarded-proto": "http",
    "x-forwarded-scheme": "https",
    "x-forwarded-uri": "/api/v1/namespaces/ccloud/services/http:exposed-service-obs-eu-de-1:8080/proxy/",
    "x-real-ip": "100.64.4.1",
    "x-request-id": "8fecab3f2e3081b6459398808ac48315",
    "x-scheme": "https"
  },

The x-fowarded-uri header seems to have no standard and should be

an exact copy of the incoming request

for what it is worth.

Looking at Plutonos code, the <base> tag is filled by the AppSubUrl var.

@richardtief Do you see any configuration possibility on Plutonos side for this? I will investigate further to check if the base tag is actually filled by that header, since for now that is just an assumption.

[uwe-mayer]

k8s api-servers sets x-forwarded-uri on proxy: https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apimachinery/pkg/util/proxy/transport.go#L90

[uwe-mayer]

I just tested running a local plutono and requesting the frontend with the header

x-forwarded-uri: /some/uri

set. This did not change the <base> tag of the index.html

[richardtief]

Thanks Uwe, I appreciate your efforts. Besides the path, I also stumbled upon the HTTP 401 code from the service proxy logs. It looks like Plutono is the first application that relies on the Authorization header proxied by the apiserver.

https://github.com/kubernetes/kubernetes/issues/38775