ansible / awx

AWX provides a web-based user interface, REST API, and task engine built on top of Ansible. It is one of the upstream projects for Red Hat Ansible Automation Platform.
Other
13.96k stars 3.41k forks source link

Intermittent GET -1 errors in UI - ERR_CONTENT_LENGTH_MISMATCH #7588

Open zigaSRC opened 4 years ago

zigaSRC commented 4 years ago
ISSUE TYPE
SUMMARY

Intermittent errors retrieving data from the api through the user interface. Sometimes it's even impossible to access the settings page because of the ERR_CONTENT_LENGTH_MISMATCH.

ENVIRONMENT
STEPS TO REPRODUCE

Use AWX normally -> Errors occur intermittantly everywhere (from the login, home, settings page to specific job runs).

EXPECTED RESULTS

I expect the pages to load successfully as it's rather annoying getting error messages every couple of clicks :)

ACTUAL RESULTS

image image

ADDITIONAL INFORMATION

The are no errors inside the container itself, just the content length in the response is set incorrectly which leads to the browsers not loading/displaying the data.

AdrianDutu commented 4 years ago

I also get these all the time.

jakemcdermott commented 4 years ago

AWX version: 13.0.0 and 12.0.0

Hi. Have you made any changes to your nginx config?

zigaSRC commented 4 years ago

No, no manual changes have been made to the configuration other than trough the inventory for the playbook (things like changing the default hostname and specifying the SSL cert). The only exception is the CSS color code to color it red for production use, but the errors were present before that change was made. I only specified both versions since we had the problem on 12 too (before upgrading to 13).

For brevity's sake here's the config nginx is using:

bash-4.4# nginx -T
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
# configuration file /etc/nginx/nginx.conf:
#user awx;

worker_processes  1;

pid        /tmp/nginx.pid;

events {
    worker_connections  1024;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;
    server_tokens off;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log /dev/stdout main;

    map $http_upgrade $connection_upgrade {
        default upgrade;
        ''      close;
    }

    sendfile        on;
    #tcp_nopush     on;
    #gzip  on;

    upstream uwsgi {
        server 127.0.0.1:8050;
        }

    upstream daphne {
        server 127.0.0.1:8051;
    }

        server {
        listen 8052 default_server;
        server_name _;

        # Redirect all HTTP links to the matching HTTPS page
        return 301 https://$host$request_uri;
    }

    server {
                listen 8053 ssl;

        ssl_certificate /etc/nginx/awxweb.pem;
        ssl_certificate_key /etc/nginx/awxweb.pem;

        # If you have a domain name, this is where to add it
        server_name _;
        keepalive_timeout 65;

        # HSTS (ngx_http_headers_module is required) (15768000 seconds = 6 month                                                                                                                                                                                                                                             s)
        add_header Strict-Transport-Security max-age=15768000;
        add_header Content-Security-Policy "default-src 'self'; connect-src 'sel                                                                                                                                                                                                                                             f' ws: wss:; style-src 'self' 'unsafe-inline'; script-src 'self' 'unsafe-inline'                                                                                                                                                                                                                                              *.pendo.io; img-src 'self' *.pendo.io data:; report-uri /csp-violation/";
        add_header X-Content-Security-Policy "default-src 'self'; connect-src 's                                                                                                                                                                                                                                             elf' ws: wss:; style-src 'self' 'unsafe-inline'; script-src 'self' 'unsafe-inlin                                                                                                                                                                                                                                             e' *.pendo.io; img-src 'self' *.pendo.io data:; report-uri /csp-violation/";

        # Protect against click-jacking https://www.owasp.org/index.php/Testing_                                                                                                                                                                                                                                             for_Clickjacking_(OTG-CLIENT-009)
        add_header X-Frame-Options "DENY";

        location /nginx_status {
          stub_status on;
          access_log off;
          allow 127.0.0.1;
          deny all;
        }

        location /static/ {
            alias /var/lib/awx/public/static/;
        }

        location /favicon.ico { alias /var/lib/awx/public/static/favicon.ico; }

        location /websocket {
            # Pass request to the upstream alias
            proxy_pass http://daphne;
            # Require http version 1.1 to allow for upgrade requests
            proxy_http_version 1.1;
            # We want proxy_buffering off for proxying to websockets.
            proxy_buffering off;
            # http://en.wikipedia.org/wiki/X-Forwarded-For
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            # enable this if you use HTTPS:
            proxy_set_header X-Forwarded-Proto https;
            # pass the Host: header from the client for the sake of redirects
            proxy_set_header Host $http_host;
            # We've set the Host header, so we don't need Nginx to muddle
            # about with redirects
            proxy_redirect off;
            # Depending on the request value, set the Upgrade and
            # connection headers
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection $connection_upgrade;
        }

        location / {
            # Add trailing / if missing
            rewrite ^(.*)$http_host(.*[^/])$ $1$http_host$2/ permanent;
            uwsgi_read_timeout 120s;
            uwsgi_pass uwsgi;
            include /etc/nginx/uwsgi_params;            proxy_set_header X-Forwa                                                                                                                                                                                                                                             rded-Port 443;
            uwsgi_param HTTP_X_FORWARDED_PORT 443;
        }
    }
}

# configuration file /etc/nginx/mime.types:

types {
    text/html                                        html htm shtml;
    text/css                                         css;
    text/xml                                         xml;
    image/gif                                        gif;
    image/jpeg                                       jpeg jpg;
    application/javascript                           js;
    application/atom+xml                             atom;
    application/rss+xml                              rss;

    text/mathml                                      mml;
    text/plain                                       txt;
    text/vnd.sun.j2me.app-descriptor                 jad;
    text/vnd.wap.wml                                 wml;
    text/x-component                                 htc;

    image/png                                        png;
    image/svg+xml                                    svg svgz;
    image/tiff                                       tif tiff;
    image/vnd.wap.wbmp                               wbmp;
    image/webp                                       webp;
    image/x-icon                                     ico;
    image/x-jng                                      jng;
    image/x-ms-bmp                                   bmp;

    application/font-woff                            woff;
    application/java-archive                         jar war ear;
    application/json                                 json;
    application/mac-binhex40                         hqx;
    application/msword                               doc;
    application/pdf                                  pdf;
    application/postscript                           ps eps ai;
    application/rtf                                  rtf;
    application/vnd.apple.mpegurl                    m3u8;
    application/vnd.google-earth.kml+xml             kml;
    application/vnd.google-earth.kmz                 kmz;
    application/vnd.ms-excel                         xls;
    application/vnd.ms-fontobject                    eot;
    application/vnd.ms-powerpoint                    ppt;
    application/vnd.oasis.opendocument.graphics      odg;
    application/vnd.oasis.opendocument.presentation  odp;
    application/vnd.oasis.opendocument.spreadsheet   ods;
    application/vnd.oasis.opendocument.text          odt;
    application/vnd.openxmlformats-officedocument.presentationml.presentation
                                                     pptx;
    application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
                                                     xlsx;
    application/vnd.openxmlformats-officedocument.wordprocessingml.document
                                                     docx;
    application/vnd.wap.wmlc                         wmlc;
    application/x-7z-compressed                      7z;
    application/x-cocoa                              cco;
    application/x-java-archive-diff                  jardiff;
    application/x-java-jnlp-file                     jnlp;
    application/x-makeself                           run;
    application/x-perl                               pl pm;
    application/x-pilot                              prc pdb;
    application/x-rar-compressed                     rar;
    application/x-redhat-package-manager             rpm;
    application/x-sea                                sea;
    application/x-shockwave-flash                    swf;
    application/x-stuffit                            sit;
    application/x-tcl                                tcl tk;
    application/x-x509-ca-cert                       der pem crt;
    application/x-xpinstall                          xpi;
    application/xhtml+xml                            xhtml;
    application/xspf+xml                             xspf;
    application/zip                                  zip;

    application/octet-stream                         bin exe dll;
    application/octet-stream                         deb;
    application/octet-stream                         dmg;
    application/octet-stream                         iso img;
    application/octet-stream                         msi msp msm;

    audio/midi                                       mid midi kar;
    audio/mpeg                                       mp3;
    audio/ogg                                        ogg;
    audio/x-m4a                                      m4a;
    audio/x-realaudio                                ra;

    video/3gpp                                       3gpp 3gp;
    video/mp2t                                       ts;
    video/mp4                                        mp4;
    video/mpeg                                       mpeg mpg;
    video/quicktime                                  mov;
    video/webm                                       webm;
    video/x-flv                                      flv;
    video/x-m4v                                      m4v;
    video/x-mng                                      mng;
    video/x-ms-asf                                   asx asf;
    video/x-ms-wmv                                   wmv;
    video/x-msvideo                                  avi;
}

# configuration file /etc/nginx/uwsgi_params:

uwsgi_param  QUERY_STRING       $query_string;
uwsgi_param  REQUEST_METHOD     $request_method;
uwsgi_param  CONTENT_TYPE       $content_type;
uwsgi_param  CONTENT_LENGTH     $content_length;

uwsgi_param  REQUEST_URI        $request_uri;
uwsgi_param  PATH_INFO          $document_uri;
uwsgi_param  DOCUMENT_ROOT      $document_root;
uwsgi_param  SERVER_PROTOCOL    $server_protocol;
uwsgi_param  REQUEST_SCHEME     $scheme;
uwsgi_param  HTTPS              $https if_not_empty;

uwsgi_param  REMOTE_ADDR        $remote_addr;
uwsgi_param  REMOTE_PORT        $remote_port;
uwsgi_param  SERVER_PORT        $server_port;
uwsgi_param  SERVER_NAME        $server_name;
AdrianDutu commented 4 years ago

I changed the port of the service, nothing else. It looks like in my case it happens only if I have more than one pod (i'm running on k8s). If I scale it down to only one pod, I don't see the popups anymore.

unlikelyzero commented 3 years ago

@AdrianDutu please try to repro on awx 16.0.0