CORS errors after upgrading to AWX 22.3.0

[multco-troy]

Please confirm the following

• [X] I agree to follow this project's code of conduct.
• [X] I have checked the current issues for duplicates.
• [X] I understand that the AWX Operator is open source software provided for free and that I might not receive a timely response.

Bug Summary

We have an external load balancer in from of our kubernetes based install that terminates SSL, and passes it to the host. URL configured for ingress on external load balancer: https://vip.example.com backend server load balancer sends traffic to (the kubernetes node): http://awx.example.com:30080 After upgrading from version 21.x, we are now seeing the following error: POST api/v2/job_templates/31/launch/ 403 CSRF Failed: Origin checking failed - https://vip.example.com does not match any trusted origins. As a work-around we are using http://awx.example.com:30080 URL to login.

AWX Operator version

2.2.1

AWX version

22.3.0

Kubernetes platform

kubernetes

Kubernetes/Platform version

Major:"1", Minor:"25"

Modifications

no

Steps to reproduce

awx.yaml file I deployed with:

apiVersion: awx.ansible.com/v1beta1 kind: AWX metadata: name: awx-kubernetes spec: service_type: nodeport

Expected results

Expect to allow CORS headers

Actual results

giving error. Even tried updating the awx.yaml with:

apiVersion: awx.ansible.com/v1beta1 kind: AWX metadata: name: awx-kubernetes spec: service_type: nodeport hostname: vip.example.com extra_settings:

• setting: MIDDLEWARE value: "['corsheaders.middleware.CorsMiddleware'] + MIDDLEWARE"
• setting: CORS_ORIGIN_ALLOW_ALL value: "True"

Additional information

No response

Operator Logs

No response

[solarisfire]

Yeah, this is giving me a headache too...

2023-05-19 11:56:42,143 WARNING [939c83ec75a945f1b0d72e9e333b8fc2] django.security.csrf Forbidden (Origin checking failed - https://192.168.50.62:4001 does not match any trusted origins.): /api/login/

So django now won't trust origins that are external load balancers (I'm using HA Proxy).

But there is no documentation on how to add a trusted origin, and no settings to change, I can't find any way to easily change settings.py on the fly as you can access it by getting a shell on the container but don't have write access to the file. You also can't update the configmap without awx operator just undoing any changes that are made. it's driving me nuts!

[shanemcd]

I suspect this may be fallout from https://github.com/ansible/awx/pull/13961.

[fosterseth]

@solarisfire @multco-troy you may need to add CSRF_TRUSTED_ORIGINS to your django settings

for k8s awx deployment (via operator) pass the extra setting this way

  extra_settings:
  - setting: CSRF_TRUSTED_ORIGINS
    value:
      - https://localhost:3001

for docker-compose environment, add CSRF_TRUSTED_ORIGINS = ['https://localhost:3001'] to your awx/settings/local_overrides.py file

[shanemcd]

After some digging I came across this blog post with a bit more context: https://noumenal.es/notes/til/django/csrf-trusted-origins/. I'm currently trying to understand if we need to change any default config on our end. I'm assuming the same value for CSRF_TRUSTED_ORIGINS won't work for everyone. If anyone has a better understanding of this issue, please do chime in.

[solarisfire]

Yeah adding that k8s config

---
apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
  name: awx
spec:
  service_type: nodeport
  projects_persistence: true
  projects_storage_access_mode: ReadWriteOnce
  web_extra_volume_mounts: |
    - name: static-data
      mountPath: /var/lib/projects
  extra_volumes: |
    - name: static-data
      persistentVolumeClaim:
        claimName: public-static-data-pvc
  extra_settings:
  - setting: CSRF_TRUSTED_ORIGINS
    value:
      - https://192.168.50.62:4001

and doing a:

kubectl apply -f awx-instance-deployment.yml -n awx

And now I can log in just fine at https://192.168.50.62:4001 through the haproxy with SSL termination on haproxy...

As for whether or not this is best practice though... That's a little above my paygrade...

[multco-troy]

@solarisfire @multco-troy you may need to add CSRF_TRUSTED_ORIGINS to your django settings

for k8s awx deployment (via operator) pass the extra setting this way

  extra_settings:
  - setting: CSRF_TRUSTED_ORIGINS
    value:
      - https://localhost:3001

for docker-compose environment, add CSRF_TRUSTED_ORIGINS = ['https://localhost:3001'] to your awx/settings/local_overrides.py file

That did it, thanks!

[fosterseth]

@solarisfire yeah I think django 4 clamped down a bit on CSRF compared to django 3

it seems to be django's recommended way to explicitly list proxy servers

https://docs.djangoproject.com/en/4.2/ref/csrf/#how-it-works see step 4

[janorn]

I can't get this work. I run awx in kubernetes but I use teleport as ingress. Everything worked before I upgraded to the recent (22.5.0) version. I added the external web address of the teleport still no go. teleport has a debug funktion and these are the headers being sent to AWX.

X-Forwarded-Host: awx.teleport.domain.com
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Forwarded-Server: teleport-agent-0
X-Forwarded-Ssl: on
X-Real-Ip: X.X.X.X

I have added http://awx.teleport.domain.com and https://awx.teleport.domain.com to CSRF Trusted Origins List but no go.

2023-07-13 15:57:41,883 WARNING  [322edfeefad94897b1ffbe73ae67a04b] django.security.csrf Forbidden (Origin checking failed - https://awx.teleport.domain.com does not match any trusted origins.): /api/login/
Y.Y.Y.Y - - [13/Jul/2023:15:57:41 +0000] "POST /api/login/ HTTP/1.1" 403 1019 "https://awx.teleport.domain.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/114.0" "-"   

[janorn]

Turns out that my issue was an incorrect rewrite of the Origin header. It was missing an URL scheme. Adding that and I could drop all CSRF Trusted Origin config.

[xBr0th3rx]

@solarisfire @multco-troy you may need to add CSRF_TRUSTED_ORIGINS to your django settings

for k8s awx deployment (via operator) pass the extra setting this way

  extra_settings:
  - setting: CSRF_TRUSTED_ORIGINS
    value:
      - https://localhost:3001

for docker-compose environment, add CSRF_TRUSTED_ORIGINS = ['https://localhost:3001'] to your awx/settings/local_overrides.py file

Hey i Use k8s awx deployment (via operator) where exactly is this file where I have to adjust it?

Otherwise I only went into the awx-operator folder and then just did "make deploy".

[fosterseth]

@xBr0th3rx you would add those extra settings to your AWX spec file. (the same file in the kubectl apply -f awx-demo.yml step, after make deploy)

UPDATE

as of AWX 22.5.0, this setting can be configured in the UI settings > miscellaneous. So you don't need to pass it in via extra_settings anymore

:construction: note, if you pass it in via the spec file, then you cannot set it in the UI!

[xBr0th3rx]

@xBr0th3rx you would add those extra settings to your AWX spec file. (the same file in the kubectl apply -f awx-demo.yml step, after make deploy)

UPDATE

as of AWX 22.5.0, this setting can be configured in the UI settings > miscellaneous. So you don't need to pass it in via extra_settings anymore

construction note, if you pass it in via the spec file, then you cannot set it in the UI!

Thanks it works

[phanimullapudi]

Add this to your AWX CR

extra_settings:
  - setting: CSRF_TRUSTED_ORIGINS
    value:
      - https://ingress-domainname

[lbrigman124]

I'm not using https but I am getting the same error: django.security.csrf Forbidden does not match any trusted origins.

[phanimullapudi]

I'm not using https but I am getting the same error: django.security.csrf Forbidden does not match any trusted origins.

In that cause use your http ingress instead...

[kgillmidmark]

I have tried all of the above, including adding extra_settings to the awx-deploy.yml & within the GUI. My deployment is fronted by a trafik ingress point to enable https on the awx URL. I'm assuming I need to do something with the traefik configuration, but the documentation for AWX & traefik is lacking. Can anyone help me get this working?

Here is my awx-deploy.yml

`--- apiVersion: awx.ansible.com/v1beta1 kind: AWX metadata: name: awx spec: service_type: nodeport projects_persistence: true projects_storage_access_mode: ReadWriteOnce web_extra_volume_mounts: |

• name: static-data mountPath: /var/lib/projects extra_volumes: |

• name: static-data persistentVolumeClaim: claimName: static-data-pvc extra_settings:

  • setting: CSRF_TRUSTED_ORIGINS value:
    • https://localhost:3001 ldap_cacert_secret: ldap-certs ldap_password_secret: ldap-bind bundle_cacert_secret: ldap-certs`

  Here is the traefik config file:

  `apiVersion: networking.k8s.io/v1 kind: Ingress metadata: namespace: awx name: awx-ingress annotations:

  HTTPS as entry point

  traefik.ingress.kubernetes.io/router.entrypoints: websecure

  Enable TLS

  traefik.ingress.kubernetes.io/router.tls: "true"

  CORS ????

  traefik.ingress.kubernetes.io/enable-cors: "true" traefik.ingress.kubernetes.io/custom-response-headers: "Access-Control-Allow-Origin:*||Access-Control-Allow-Methods:GET,POST,OPTIONS||Access-Control-Allow-Headers:DNT,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range||Access-Control-Expose-Headers:Content-Length,Content-Range" spec: tls:

  • hosts:
• awx.domain.net secretName: tls-awx-ingress rules:
• host: awx.domain.net http: paths:
  • path: / pathType: Prefix backend: service: name: awx-service port: number: 80`

[josue-soares]

@xBr0th3rx you would add those extra settings to your AWX spec file. (the same file in the kubectl apply -f awx-demo.yml step, after make deploy)

UPDATE

as of AWX 22.5.0, this setting can be configured in the UI settings > miscellaneous. So you don't need to pass it in via extra_settings anymore construction note, if you pass it in via the spec file, then you cannot set it in the UI!

Thanks it works

Works for me too, but I needed delete awx-web pod after.

[josue-soares]

I put "https://*.my-domain"