search

ansible / awx

AWX provides a web-based user interface, REST API, and task engine built on top of Ansible. It is one of the upstream projects for Red Hat Ansible Automation Platform.
Other
13.97k stars 3.41k forks source link

SAML Login Fails With Invalid Response #1016

Closed newlandk closed 6 years ago

newlandk commented 6 years ago
ISSUE TYPE
COMPONENT NAME
SUMMARY

When logging into AWX with SAML configured the final redirect appears to be to an incorrect port that is of the container and not the web interface.

ENVIRONMENT
STEPS TO REPRODUCE

Configure AWX with a SAML Identity Provider. The SAML portion of the authentication appears to work up to the point where a redirect occurs to an incorrect port and protocol (that appears to be the port and protocol on the docker container) and not that which the web/API is hosted from.

It appears this may be a regression from the transition from social-app-django to social-auth-core and social-auth-app-django.

There has been some activity here that seems related: https://github.com/ansible/awx/issues/119 but it looks like there have been changes in the authentication back-end since it was closed and the issue covers multiple challenges.

EXPECTED RESULTS

Redirected to the URL that includes the hostname and port of the web/API logged in.

ACTUAL RESULTS

An error after successfully logging into the IDp.

Authentication failed: SAML login failed: ['invalid_response'] (The response was received at http://awx.test.com:8052/sso/complete/saml/ instead of https://awx.test.com/sso/complete/saml/).

newlandk commented 6 years ago

I'm not too familiar with social-core or Python, but I've done my best to understand it. It looks like social-app-django may be using the container to determine port and protocol since it is the source of the SAML request

newlandk commented 6 years ago

It turns out the issue faced was proxy related and I had to modify the settings.py configuration to allow the use of the X-Forwarded-Port. After this django provides the correct port to social-auth for the complete URL.

Good reference if you run into the same issue: Tower Proxy Documentation

This issue was especially confusing since the there are multiple places in the UI that state the full URL of the server including protocol and social-auth determines these values from django instead.

BenJuan26 commented 6 years ago

Can you explain a little about how you mounted settings.py as a volume? I tried changing awx/installer/roles/image_build/files/settings.py but even after rebuilding all containers, my version of the settings was not located at /etc/tower/settings.py inside the container.

I'm getting the same error you've provided here.

newlandk commented 6 years ago

@BenJuan26 we just mount a modified settings.py outside the container and mount it in the container as a volume for the time being.

When you run your docker-compose add arguments similar to this for awx_web:

services:
  awx_web:
    volumes:
      - /src/settings.py:/etc/awx/settings.py

More information here on volumes/mounts: https://docs.docker.com/storage/volumes/#start-a-container-with-a-volume

sudomateo commented 6 years ago

So the way I fixed this to work correctly with SAML was by building the container images locally and pushing them to the remote AWX server. This alleviates the need to mount a volume.

First I commented out the following lines in installer/inventory:

#dockerhub_base=ansible
#dockerhub_version=latest

This allows you to build the container image locally instead of pulling from ansible's dockerhub images.

Next I added the following lines to installer/roles/image_build/files/settings.py

USE_X_FORWARDED_PORT = True
USE_X_FORWARDED_HOST = True

This is what gets injected into the container images for the web and task containers.

Then I configured the nginx reverse proxy on the AWX remote host. My config looks like this:

server {
    listen 80;
    server_name awx.example.com;

    location / {
        return 301 https://awx.example.com$request_uri;
    }
}
server {  
    listen 443 ssl;
    server_name awx.example.com;

    ssl_certificate cert.pem;
    ssl_certificate_key key.pem;
    ssl_protocols TLSv1.2;
    ssl_prefer_server_ciphers on;
    ssl_ciphers "ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-ECDSA-AES256-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA:ECDHE-RSA-AES128-SHA"; 
    ssl_ecdh_curve secp384r1;
    ssl_session_cache shared:SSL:10m;
    ssl_session_tickets off;
    ssl_stapling on;
    ssl_stapling_verify on;

    add_header Strict-Transport-Security "max-age=63072000; includeSubdomains; preload";
    add_header X-Content-Type-Options nosniff;

    location / {
        proxy_pass http://127.0.0.1:8080;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-Port 443;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
    }
}

The important line here is proxy_set_header X-Forwarded-Port 443;. This forwards the correct port along for AWX to use with the USE_X_FORWARDED_HOST = True setting above.

Lastly, I added the HTTP_X_FORWARDED_FOR header in Settings > System > Remote Host Headers

I used https://gist.github.com/defionscode/fc21488e44d73cdd919f81ee1b43e204 as a reference for how my SAML config should look.

jeunii commented 5 years ago

Hello @sudomateo I encountered a similar issue. Actually I have the exact same issue. My AWX setup is deployed in a kubernetes cluster. And rather than nginx as my Ingress, I have HA Proxy. Would you by any chance know what would be the equivalent haproxy config ?

newlandk commented 5 years ago

I believe you would have configuration options like this under your frontend to overcome the challenges we faced noted above. http-request set-header X-Forwarded-Proto https if https http-request set-header X-Forwarded-Proto http if !https http-request set-header X-Forwarded-Port 443 option forwardfor

Good luck!

sudomateo commented 5 years ago

@jeunii I'm sorry I do not have an exact HAProxy configuration handy but what @newlandk commented is a good start. As long as the headers are properly set it should be fine.

svrraja commented 5 years ago

How to fix the issue if its running in kubernetes, How to configure the NGINX reverse proxy

naren-ambati commented 5 years ago

@sudomateo I'm facing the same issue, But the changes I did for enabling tls is completely different of what you used in nginx.conf. 

#user awx;

worker_processes  1;

pid        /tmp/nginx.pid;

events {
    worker_connections  1024;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log /dev/stdout main;

    map $http_upgrade $connection_upgrade {
        default upgrade;
        ''      close;
    }

    sendfile        on;
    #tcp_nopush     on;
    #gzip  on;

    upstream uwsgi {
        server 127.0.0.1:8050;
        }

    upstream daphne {
        server 127.0.0.1:8051;
    }

    server {
                listen 8052 default_server;

        # If you have a domain name, this is where to add it
        server_name awx.project.qa;
        keepalive_timeout 65;
        ssl_certificate           /etc/nginx/certs/awx.project.qa.chain.pem;
        ssl_certificate_key       /etc/nginx/certs/awx.project.qa.key;
        ssl on;
        ssl_session_cache  builtin:1000  shared:SSL:10m;
        ssl_protocols TLSv1.2;
        ssl_ciphers HIGH:!aNULL:!eNULL:!EXPORT:!CAMELLIA:!DES:!MD5:!PSK:!RC4;
        ssl_prefer_server_ciphers on;
        access_log            /var/log/nginx/awx.access.log;
        ssl_session_tickets off;
        ssl_stapling on;
        ssl_stapling_verify on;

        # HSTS (ngx_http_headers_module is required) (15768000 seconds = 6 months)
        add_header Strict-Transport-Security max-age=15768000;
        add_header Content-Security-Policy "default-src 'self'; connect-src 'self' ws: wss:; style-src 'self' 'unsafe-inline'; script-src 'self' 'unsafe-inline' cdn.pendo.io; img-src 'self' data:; report-uri /csp-violation/";
        add_header X-Content-Security-Policy "default-src 'self'; connect-src 'self' ws: wss:; style-src 'self' 'unsafe-inline'; script-src 'self' 'unsafe-inline' cdn.pendo.io; img-src 'self' data:; report-uri /csp-violation/";

        # Protect against click-jacking https://www.owasp.org/index.php/Testing_for_Clickjacking_(OTG-CLIENT-009)
        add_header X-Frame-Options "DENY";

        location /nginx_status {
          stub_status on;
          access_log off;
          allow 127.0.0.1;
          deny all;
        }

        location /static/ {
            alias /var/lib/awx/public/static/;
        }

        location /favicon.ico { alias /var/lib/awx/public/static/favicon.ico; }

        location /websocket {
            # Pass request to the upstream alias
            proxy_pass http://daphne;
            # Require http version 1.1 to allow for upgrade requests
            proxy_http_version 1.1;
            # We want proxy_buffering off for proxying to websockets.
            proxy_buffering off;
            # http://en.wikipedia.org/wiki/X-Forwarded-For
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            # enable this if you use HTTPS:
            proxy_set_header X-Forwarded-Proto https;
            # pass the Host: header from the client for the sake of redirects
            proxy_set_header Host $http_host;
            # We've set the Host header, so we don't need Nginx to muddle
            # about with redirects
            proxy_redirect off;
            # Depending on the request value, set the Upgrade and
            # connection headers
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection $connection_upgrade;
        }

        location / {
            # Add trailing / if missing
            rewrite ^(.*)$http_host(.*[^/])$ $1$http_host$2/ permanent;
            uwsgi_read_timeout 120s;
            uwsgi_pass uwsgi;
            include /etc/nginx/uwsgi_params;

      }
    }
}

I'm trying to make use of your nginx configuration, but my awx_web is not working and I don't see any errors. Do you have any turn around so that I can use your configuration in the above. Any help is appriciated, Thanks.

loitho commented 4 years ago

I'm putting my reply here too. This applies if you're running the AWX_WEB with docker-compose with the default port configuration with SSL

If the Nginx that forward its requests to the Django server (the uwsgi_pass uwsgi;) isn't running on a standard port (like here, it's running in the 8053) you can either :

And off you go. The error :

The response was received at https://XXXXX:8053/sso/complete/saml/ instead of https://XXXXX/sso/complete/saml/

Should be gone. I'm attaching my nginx.conf so that you can check it out (it's the AWS 9.0.1 default file, just with the line addition)

#user awx;

worker_processes  1;

pid        /tmp/nginx.pid;

events {
    worker_connections  1024;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;
    server_tokens off;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log /dev/stdout main;

    map $http_upgrade $connection_upgrade {
        default upgrade;
        ''      close;
    }

    sendfile        on;
    #tcp_nopush     on;
    #gzip  on;

    upstream uwsgi {
        server 127.0.0.1:8050;
        }

    upstream daphne {
        server 127.0.0.1:8051;
    }

        server {
        listen 8052 default_server;
        server_name _;

        # Redirect all HTTP links to the matching HTTPS page
        return 301 https://$host$request_uri;
    }

    server {
#        listen 443 ssl;
        listen 8053 ssl;

        ssl_certificate /etc/nginx/awxweb.pem;
        ssl_certificate_key /etc/nginx/awxweb.pem;

        # If you have a domain name, this is where to add it
        server_name _;
        keepalive_timeout 65;

        # HSTS (ngx_http_headers_module is required) (15768000 seconds = 6 months)
        add_header Strict-Transport-Security max-age=15768000;
        add_header Content-Security-Policy "default-src 'self'; connect-src 'self' ws: wss:; style-src 'self' 'unsafe-inline'; script-src 'self' 'unsafe-inline' *.pendo.io; img-src 'self' *.pendo.io data:; report-uri /csp-violation/";
        add_header X-Content-Security-Policy "default-src 'self'; connect-src 'self' ws: wss:; style-src 'self' 'unsafe-inline'; script-src 'self' 'unsafe-inline' *.pendo.io; img-src 'self' *.pendo.io data:; report-uri /csp-violation/";

        # Protect against click-jacking https://www.owasp.org/index.php/Testing_for_Clickjacking_(OTG-CLIENT-009)
        add_header X-Frame-Options "DENY";

        location /nginx_status {
          stub_status on;
          access_log off;
          allow 127.0.0.1;
          deny all;
        }

        location /static/ {
            alias /var/lib/awx/public/static/;
        }

        location /favicon.ico { alias /var/lib/awx/public/static/favicon.ico; }

        location /websocket {
            # Pass request to the upstream alias
            proxy_pass http://daphne;
            # Require http version 1.1 to allow for upgrade requests
            proxy_http_version 1.1;
            # We want proxy_buffering off for proxying to websockets.
            proxy_buffering off;
            # http://en.wikipedia.org/wiki/X-Forwarded-For
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            # enable this if you use HTTPS:
            proxy_set_header X-Forwarded-Proto https;
            # pass the Host: header from the client for the sake of redirects
            proxy_set_header Host $http_host;
            # We've set the Host header, so we don't need Nginx to muddle
            # about with redirects
            proxy_redirect off;
            # Depending on the request value, set the Upgrade and
            # connection headers
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection $connection_upgrade;
        }

        location / {
            # Add trailing / if missing
            rewrite ^(.*)$http_host(.*[^/])$ $1$http_host$2/ permanent;
            uwsgi_read_timeout 120s;
            uwsgi_pass uwsgi;
            include /etc/nginx/uwsgi_params;

            proxy_set_header X-Forwarded-Port 443;
           # that's the important bit
            uwsgi_param HTTP_X_FORWARDED_PORT 443;
        }
    }
}

I haven't seen much ADFS SAML configuration available and since I had quite some trouble setting it up, I'm also adding my ADFS configuration for "SAML ENABLED IDENTITY PROVIDERS" 

{
 "saml_ms_adfs": {
  "attr_last_name": "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/surname",
  "entity_id": "http://signin.server.com/adfs/services/trust",
  "x509cert": "<redacted>",
  "url": "https://signin.server.com/adfs/ls/",
  "attr_username": "http://schemas.microsoft.com/ws/2008/06/identity/claims/windowsaccountname",
  "attr_email": "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/emailaddress",
  "attr_first_name": "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/givenname",
  "attr_user_permanent_id": "http://schemas.microsoft.com/ws/2008/06/identity/claims/windowsaccountname"
 }
}

Big thanks to this thread : https://github.com/pallets/werkzeug/issues/1465#issuecomment-469722847 for the enlightenment

ryanpetrello commented 4 years ago

For those following along in this thread, @loitho has contributed a patch which should alleviate this issue (which I'm merging shortly):

https://github.com/ansible/awx/pull/5577

peacengell commented 1 year ago

Hello All,

I found a fix, tested and working. Credit goes to [ https://medium.com/@_jonas/traefik-kubernetes-ingress-and-x-forwarded-headers-82194d319b0e ] This kind man, thank you.

If you using k3s or Kubernetes and Traefik, you can edit this file

Traefix is setup using Helm Chart. The file path is /var/lib/rancher/k3s/server/manifests/traefik.yaml

Edit it and add the three line with #< with it.

valuesContent: |-
    podAnnotations:
      prometheus.io/port: "8082"
      prometheus.io/scrape: "true"
    providers:
      kubernetesIngress:
        publishedService:
          enabled: true
    additionalArguments: #<
      - "--entryPoints.web.proxyProtocol.insecure" #<
      - "--entryPoints.web.forwardedHeaders.insecure" #<

Helm will redeploy your traefix pod, once restarted give it a 10 min.

Then check.