Reverse proxy from docker traefik to AWX on same host

[toutas]

Greetings

I have been fiddling around with trying to get AWX running behind a reverse proxy. I have a host with traefik listening to 80/443 already, running on docker, so the default k3s ingress is not available. I would like to modify AWX and the k3s ingress to listen to 8080/8443, so I can reverse proxy from my traefik instance to localhost:8080 if it matches my awx.my.domain route.

I have defined the traefik service as required to load balance requests to :8080, but I am not sure how to modify AWX to listen on 8080.

I have attempted to use the following modification to ingress on k3s:

sudo tee /var/lib/rancher/k3s/server/manifests/traefik-config.yaml <<EOF
---
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: traefik
  namespace: kube-system
spec:
  valuesContent: |-
    ports:
      web:
        exposedPort: 8080
      websecure:
        exposedPort: 8443
    additionalArguments:  
      - "--entryPoints.web.proxyProtocol.insecure" 
      - "--entryPoints.web.forwardedHeaders.insecure"
    logs:
      access:
        enabled: true
EOF

and this allows me to do the following

~ kubectl get svc traefik -n kube-system
NAME      TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                         AGE
traefik   LoadBalancer   10.43.90.180   10.10.100.249   8080:31383/TCP,8443:31821/TCP   94m

~ kubectl -n awx get ingress
NAME          CLASS     HOSTS                 ADDRESS         PORTS     AGE
awx-ingress   traefik   awx.my.domain 10.10.100.249   80, 443   97m

➜  ~ curl --resolve awx.my.domain:80:10.10.100.249 http://awx.my.domain/api/
Moved Permanently    
➜  ~ curl --resolve awx.my.domain:8080:10.10.100.249 http://awx.my.domain:8080/api/
{"description":"AWX REST API","current_version":"/api/v2/","available_versions":{"v2":"/api/v2/"},"oauth2":"/api/o/","custom_logo":"","custom_login_info":"","login_redirect_override":""}

to me this shows AWX can respond to requests on port 8080.

I have defined my docker network with static docker_network_gateway_ip as 172.19.0.1, and I can see that requests are received in the k3s traefik ingress log when using curl:

~ curl -kv http://172.19.0.1:8080/api
*   Trying 172.19.0.1:8080...
* Connected to 172.19.0.1 (172.19.0.1) port 8080
> GET /api HTTP/1.1
> Host: 172.19.0.1:8080
> User-Agent: curl/8.5.0
> Accept: */*
>
< HTTP/1.1 404 Not Found
< Content-Type: text/plain; charset=utf-8
< X-Content-Type-Options: nosniff
< Date: Fri, 05 Jul 2024 11:52:15 GMT
< Content-Length: 19
<
404 page not found
* Connection #0 to host 172.19.0.1 left intact

~ kubectl -n kube-system logs -f traefik-ff5467899-fnkdx
10.42.0.29 - - [05/Jul/2024:11:52:15 +0000] "GET /api HTTP/1.1" 404 19 "-" "-" 493 "-" "-" 0ms

but when I attempt to curl it just says 404 page not found. This means redirects from docker traefik to the ingress just display 404 page not found when I try to access awx.my.domain. Any ideas as to what I can modify?

The issue is that requests to my host port 8080 do not get routed to the awx instance for some reason.

[kurokobo]

Hi,

curl -kv http://172.19.0.1:8080/api

Have you tried this with --resolve option?

[toutas]

@kurokobo I did try that.

But I just uninstalled k3s and did a reinstall from scratch (with the same playbook I always use, no modifications) and it works fine now. My docker traefik instance redirects perfectly fine to the k3s ingress traefik instance on 8080.

Sorry for this issue, seems to be due to some weird happening on my host. The host has never had k3s before, so not sure why uninstalling and then running my setup again made it work.

closing issue since it's fixed!