Job output not refreshing automatically in AWX 18.0.0

[dimovstanimir]

ISSUE TYPE

• Bug Report

SUMMARY

Job output not refreshing automatically in AWX 18.0.0. I need to manually hit F5 or refresh from the browser so I could see what is happening during the job execution.

ENVIRONMENT

• AWX version: 18.0.0
• AWX install method: minikube
• Ansible version: 2.9.13
• Operating System: Ubuntu 20.04 LTS
• Web Browser: ALL

[lumarel]

@rooso Thank you too for the solution, I was never able to solve it myself, but this worked now! Finally! 😀🎉 (time to find a solution that is PR worthy I guess)

[aimcod]

In my case, I was using for the ingress. https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.3.1/deploy/static/provider/baremetal/deploy.yaml

I switched to the following and progress tracking and UI updates now work. https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/baremetal/deploy.yaml

I caught this as I am using awx in two different environments. 3 if you count my home lab. Home lab and production were using the main branch, while the development env was using v1.3.1

My deployment script here: https://raw.githubusercontent.com/aimcod/k8awx/main/awxSetup.sh I hope this helps.

[Sasa993]

We are experiencing the same issue on version 23.0.0. In our scenario, there is a conflict with hybrid node instances. It's reproducible in dev environment:

1. Run the containers with two hybrid nodes: CONTROL_PLANE_NODE_COUNT=2 make docker-compose
2. Disable awx_1 instance node in Instance Groups
3. Launch a job (Demo Job Template) on awx_1 instance (_https://:8043_)
4. The jobs starts on with awx_2 as a controller node
5. While still connected to awx_1, the job output doesn't refresh (stuck on "Waiting for job output..")

I noticed in DevTools > Network that websocket/ is stuck on "Switching protocols" status - not sure if that's relevant:

[LukasPod23]

We are experiencing the same issue on AWX version 23.4.0 deployed on K3S CentOS Stream 8 with nodeport. Job output isn't updating via WS. On version AWX 21.3.0 update was working. Was there any progress to resolve this?

Thanks,

[kma617]

I have the same issue on fresh installation of 23.6.0 in kubernetes I tried different ways to fix it: use nodeport, nginx ingress with websocket support, contour ingress. Nothing helped

[shaleenbathla]

I had the same issue as : https://github.com/ansible/awx/issues/9747#issuecomment-1755458168

In my config, proxy_pass for websocket location was commented out and the daphne part

location /websocket {
    # Pass request to the upstream alias
    proxy_pass http://daphne/;
    ...
}

[ygemici]

I think here we can check best working config: https://github.com/ansible/awx/blob/15.0.1/tools/docker-compose/nginx.vh.default.conf

@xBr0th3rx , @Donny-Ta all working config was published a long time ago.

here is the latest version https://github.com/ansible/awx/blob/17.0.1/tools/docker-compose/nginx.vh.default.conf

my working config:

map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}

upstream minikube {
    server 192.168.49.2:30080;
}

server {
    listen 80;
    listen [::]:80;
    listen 443 ssl http2;
    listen [::]:443 ssl http2;
    server_name awx.example.com;
    keepalive_timeout 65;

    location ~ /.well-known {
        auth_basic off;
        allow all;
    }

    if ($scheme != "https") {
        rewrite ^ https://$host$uri permanent;
    }

    ssl_session_timeout 1d;
    ssl_session_cache shared:MozSSL:10m;
    ssl_session_tickets off;
    ssl_protocols TLSv1.3;
    ssl_prefer_server_ciphers off;
    add_header Strict-Transport-Security "max-age=63072000" always;

    # OCSP stapling
 #   ssl_stapling on;
 #   ssl_stapling_verify on;

    ssl_certificate /etc/letsencrypt/live/awx.example.com/fullchain.pem; 
    ssl_certificate_key /etc/letsencrypt/live/awx.example.com/privkey.pem; 

    # HSTS (ngx_http_headers_module is required) (15768000 seconds = 6 months)
    add_header Strict-Transport-Security max-age=15768000;

    location / {
        proxy_set_header        Host $host;
        proxy_set_header        X-Real-IP $remote_addr;
        proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header        X-Forwarded-Proto $scheme;
        proxy_pass              http://minikube;
        proxy_read_timeout      90;
    }

    location ~ ^/websocket {
        # Pass request to the upstream alias
        proxy_pass http://minikube;
        # Require http version 1.1 to allow for upgrade requests
        proxy_http_version 1.1;
        # We want proxy_buffering off for proxying to websockets.
        proxy_buffering off;
        # http://en.wikipedia.org/wiki/X-Forwarded-For
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        # enable this if you use HTTPS:
        proxy_set_header X-Forwarded-Proto https;
        # pass the Host: header from the client for the sake of redirects
        proxy_set_header Host $http_host;
        # We've set the Host header, so we don't need Nginx to muddle about with redirects
        proxy_redirect off;
        # Depending on the request value, set the Upgrade and connection headers
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
    }
}

Hi amg-web, Unfortunately this doesn't work.

[AlekseiBochkarevBS]

For me fix problem this nginx proxy config (don't forget add map and location ~ ^/websocket). I use AWX docker install

map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}
server {
    listen 80;
    server_name awx.mysite.biz;
    return 404;

    access_log /var/log/nginx/awx.mysite.biz_access.log;
    error_log /var/log/nginx/awx.mysite.biz_error.log;

    location /.well-known/acme-challenge/ {
        root /var/www/html;
    }

    location / {
        return 301 https://$host$request_uri;
    }
}

server {
    listen 443 ssl http2;
    server_name awx.mysite.biz;

    access_log /var/log/nginx/awx.mysite.biz_access.log;
    error_log /var/log/nginx/awx.mysite.biz_error.log;

    ssl_certificate /etc/letsencrypt/live/awx.mysite.biz/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/awx.mysite.biz/privkey.pem;
    include /etc/letsencrypt/options-ssl-nginx.conf;

    location / {
        proxy_connect_timeout 300;
        proxy_pass https://127.0.0.1:8043;
        proxy_read_timeout 300;
        proxy_send_timeout 300;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_ssl_verify off;
        send_timeout 300;
    }

    location ~ ^/websocket {
        proxy_pass https://127.0.0.1:8043;
        proxy_ssl_verify off;
        proxy_http_version 1.1;
        proxy_buffering off;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header Host $host;
        proxy_redirect off;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
    }
}