Problems with X-Forwarded-For Header 15.0.0

[dvzuykov]

ISSUE TYPE

• Bug Report

COMPONENT NAME

• API

SUMMARY

X-Forwarded-For stops work after upgrade to 15.0.0. When I submit callback request with header like below: curl -H "'X-Forwarded-For': '10.91.191.227'" --verbose https://awxhost:443/api/v2/job_templates/9/callback/ --data "host_config_key=388b1d5d-6121-42bd-8259-f74814582e58" AWX just initiating job for host which send request instead of using of IP from header. And in case of load balancer it basically use load balancer IP and return "no matching hosts"

ENVIRONMENT

• AWX version: 15.0.0
• AWX install method: kubernetes
• Ansible version: 2.9.13
• Operating System: CentOS 8.2.2004
• Web Browser: curl / powershell Invoke-Request

STEPS TO REPRODUCE

Run callback request from machine under load balancer or from different machine: curl -H "'X-Forwarded-For': '10.91.191.227'" --verbose https://awx-load-balancer.com.com:443/api/v2/job_templates/9/callback/ --data "host_config_key=388b1d5d-6121-42bd-8259-f74814582e58"

EXPECTED RESULTS

AWX should initiate job for machine defined in header (x-forwarded-for).

ACTUAL RESULTS

AWX return "no matched hosts" in case of load balancer or initiate job for machine which send request (in case of direct connection).

ADDITIONAL INFORMATION

Case with Load balancer: Host in inventory associated with Job template and AWX can resolve DNS for that. curl -H "'X-Forwarded-For': '10.91.191.227'" https://awx-load-balancer.com:443/api/v2/job_templates/20/callback/ --data "host_config_key=3f6226cd-e3a3-4544-a209-38189abff6f4" {"msg":"No matching host could be found!"}

Case with direct connection: nnvapp009:~# curl -H "'X-Forwarded-For': '10.91.191.227'" https://awx-direct.com:443/api/v2/job_templates/9/callback/ --data "host_config_key=388b1d5d-6121-42bd-8259-f74814582e58"

Result

Try to look on configuration, play with headers config and in particular added USE_X_FORWARDED_HOST = True, but nothing helped.

[soulbrat]

Try this:

set Settings --> Configure Tower --> System in the REMOTE HOST HEADERS: "HTTP_X_FORWARDED_FOR, HTTP_TRUE_CLIENT_IP, REMOTE_ADDR, REMOTE_HOST"

$ curl -H "X-Forwarded-For: node1.example.com" --data "host_config_key=6e941383-22c0-4aa3-a26f-c69bd2544a21" https://awx.example.com:443/api/v2/job_templates/24/callback/

$ curl -H "True-Client-IP: 10.0.3.3" --data "host_config_key=6e941383-22c0-4aa3-a26f-c69bd2544a21" https://awx.example.com:443/api/v2/job_templates/24/callback/

[dvzuykov]

I found root cause of issue. It's in kubernetes ingress. It's just moving "x-forwarded-for" to "x-original-forwarded-for" and rewriting "x-forwarded-for" by ip of system which send request (load balancer for example). And AWX just get wrong ip address.

I fixed it by adding new header to AWX config. Also possible to reconfigure k8s ingress controller.

[rchaud]

I found root cause of issue. It's in kubernetes ingress. It's just moving "x-forwarded-for" to "x-original-forwarded-for" and rewriting "x-forwarded-for" by ip of system which send request (load balancer for example). And AWX just get wrong ip address.

I fixed it by adding new header to AWX config. Also possible to reconfigure k8s ingress controller.

@dvzuykov Do you have an example of the changes you made? I am running into this issue as well.

[dvzuykov]

@chinochao I modified config map "awx-config", key "awx_settings". It's should have such line: REMOTE_HOST_HEADERS = ['HTTP_X_FORWARDED_FOR', 'HTTP_X_ORIGINAL_FORWARDED_FOR']

And after redeploying you will able to see these headers in settings of AWX:

[rchaud]

@soulbrat @dvzuykov Do you guys have any input in how it can be for this setup?

Client -> LBaaS -> K8s Ingress -> AWX/Nginx container.

AWX/Nginx container is the awx container running nginx proxying to the application uswgi.

[rotcivo]

@chinochao this is quite old but posting here in case someone else has a similar set up and problem. Our environment is:

Client -> AWS ALB -> K8s Nginx Ingress -> AWX

After setting all for forwarded headers properly, we we're still getting an http reply URL from our idp, instead of https. Seems like the k8s ingress-nginx turns off the use of forwarded headers by default: https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#use-forwarded-headers

Once we set use-forwarded-headers to true in our k8s ingress-nginx config, SAML authentication worked as expected.