Job output not refreshing automatically in AWX 18.0.0

[dimovstanimir]

ISSUE TYPE

• Bug Report

SUMMARY

Job output not refreshing automatically in AWX 18.0.0. I need to manually hit F5 or refresh from the browser so I could see what is happening during the job execution.

ENVIRONMENT

• AWX version: 18.0.0
• AWX install method: minikube
• Ansible version: 2.9.13
• Operating System: Ubuntu 20.04 LTS
• Web Browser: ALL

[jakemcdermott]

Hello. Thanks for creating this issue.

I haven't been successful at reproducing this on devel:

Do you see any websocket events in the network tab? And are sockets generally working in other places, like on job and project lists where the status icons are supposed to update?

[dimovstanimir]

If you have some long playbook to be executed such as mines where I have playbooks taking more than 20 minutes I am only seeing blank output windows until the playbook has finished. So I am waiting around 20 minutes to see the whole log output in a second. Isn't it possible to see all the steps while they are executed and not wait for when the whole playbook has been finished so I would be able to see the whole log dumped at the end?

[amg-web]

same problem. (existed on v16 and v17) v.15 worked great. need to refresh page to get it updated again and again

for v18 reinstalled from scratch with minikube

I see websocket connection error

[dimovstanimir]

same problem. (existed on v16 and v17) v.15 worked great. need to refresh page to get it updated again and again

for v18 reinstalled from scratch with minikube

I see websocket connection error

I totaly agree with @amg-web . But how hard could it be for this to be fixed. It has been 4 major releases till now in which this has not been fixed and this functionality is truly needed.

[ryanpetrello]

But how hard could it be for this to be fixed.

Hey @dimovstanimir,

PRs are encouraged and welcome - let us know what you find!

[jakemcdermott]

:wave: Hi folks

Any comment on my question above? Specifically, I'd like to know if any of our socket-driven features appear to be working for you on any of the other pages (updating status icons on job lists, etc.). It'll help isolate the problem for when we work on addressing the bug.

Do you see any websocket events in the network tab? And are sockets generally working in other places, like on job and project lists where the status icons are supposed to update?

[dimovstanimir]

But how hard could it be for this to be fixed.

Hey @dimovstanimir,

PRs are encouraged and welcome - let us know what you find!

This thing was workin

But how hard could it be for this to be fixed.

Hey @dimovstanimir,

PRs are encouraged and welcome - let us know what you find!

@ryanpetrello this thing was working 4 versions ago and someone broke it. Why should I be the one fixing it? Just find who broke it and who approved the PR request. Everything is in git.

[ryanpetrello]

@dimovstanimir

Why should I be the one fixing it? Just find who broke it and who approved the PR request. Everything is in git.

I mean, why should anyone be obligated to fix it? It's open source software, and it's provided for free. While our team of maintainers tries their best to address reports of issues, this software - like much open source software - is provided as-is.

It sounds like that there might be a legitimate regression here, but until someone finds the time to look into it and address it (which I'm sure they will at some point), you (or anyone else) is free to pitch in (or exercise some patience and wait).

@jakemcdermott asked some questions above to clarify:

https://github.com/ansible/awx/issues/9747#issuecomment-811167381 https://github.com/ansible/awx/issues/9747#issuecomment-816789508

Do you know if you're seeing this on other pages that make use of websockets, or just this one?

[dimovstanimir]

@ryanpetrello I don't want to blame people and etc. but If it was you, would you approve 4 new releases before having such a major functionality fixed after knewing that prior that it was crippled? I know I wouldn't.

[jakemcdermott]

@dimovstanimir Do sockets and status icon updates work for you on other pages, such as the jobs and project list?

[jakemcdermott]

@dimovstanimir

I'm talking about these status icons, on a different page:

On your installation, do their statuses update without a refresh?

[amg-web]

@jakemcdermott I'm talking about these status icons, on a different page: On your installation, do their statuses update without a refresh?

no. no status updates. I saw 404 in Firefox for a short time, after it changed to 'finished'

[jakemcdermott]

Good info, thank you. @amg-web

@dimovstanimir To help us isolate the problem, can you please comment on this as well? https://github.com/ansible/awx/issues/9747#issuecomment-816896536

[dimovstanimir]

@jakemcdermott I've just checked and the status icons on the Jobs page are refreshing normally without the need of me hitting the refresh button. Unfortunately when I start a job template and when I want to see some output I am not getting any. I am waiting for about 5 to 20 minutes depending on the playbook and when it finishes it dumps all the stages/steps at once and shows also how much time did it take to run with all relevant details from the dump but the output happens by itself ONLY after the whole playbook has run till the very end.

[amg-web]

if start job and go Jobs I see animations, but it's endless. It does not update state. I have to make page refresh

[DenisMedeirosBBD]

Also having the same issue (used to work before but now it's broken after updating AWX).

Some notes:

1. On Google Chrome console, it shows a few messages like WebSocket is already in CLOSING or CLOSED state..

2. When I launch a job template, the Output screen opens but nothing happens (even the timer does not start to run). Then, after a few minutes (when the run is finished), the screen is updated automatically with the whole output and the timer with the total elapsed time.

[dimovstanimir]

Also having the same issue (used to work before but now it's broken after updating AWX).

Some notes:

1. On Google Chrome console, it shows a few messages like WebSocket is already in CLOSING or CLOSED state..
2. When I launch a job template, the Output screen opens but nothing happens (even the timer does not start to run). Then, after a few minutes (when the run is finished), the screen is updated automatically with the whole output and the timer with the total elapsed time.

This is the most acurate explanation and this is the very same behavior which is happening in my installation.

[eschek87]

AWX 19.0.0 affected, too.

[amg-web]

I manage to fix it by correcting nginx proxy config. Config I used described here: https://www.linkedin.com/pulse/how-use-nginx-reverse-proxy-https-wss-self-signed-ramos-da-silva?articleId=6678584723419226112

[eschek87]

I manage to fix it by correcting nginx proxy config. Config I used described here: https://www.linkedin.com/pulse/how-use-nginx-reverse-proxy-https-wss-self-signed-ramos-da-silva?articleId=6678584723419226112

And what of the described settings helped?

[dimovstanimir]

I manage to fix it by correcting nginx proxy config. Config I used described here: https://www.linkedin.com/pulse/how-use-nginx-reverse-proxy-https-wss-self-signed-ramos-da-silva?articleId=6678584723419226112

@amg-web can you make a short video and show us how it looks when being fixed and working?

Thanks in advance

[amg-web]

And what of the described settings helped?

about websocket proxy config. This is not the only place. You can search a lot of manuals about websocket proxy configs

can you make a short video and show us how it looks when being fixed and working?

All animations is working.

[amg-web]

But there is another error in browser debug as you can see. useWebsocket.js:20 WebSocket is already in CLOSING or CLOSED state. t @ useWebsocket.js:20 (anonymous) @ useWebsocket.js:37 setTimeout (async) Kn.c.current.onclose @ useWebsocket.js:36

[amg-web]

I think here we can check best working config: https://github.com/ansible/awx/blob/15.0.1/tools/docker-compose/nginx.vh.default.conf

[amg-web]

@jakemcdermott how do you use it? Which nginx or apache config is in use? my current config does not update time counter and have some errors in browser debug window

[garnser]

@amg-web I've given the suggested configuration a spin, although most animations are working the auto-update during a template runtime is still hanging until the play is finished. Is this resolved for you as well?

I'm getting the same issues in the console as you're describing useWebsocket.js:20 WebSocket is already in CLOSING or CLOSED state. t @ useWebsocket.js:20 (anonymous) @ useWebsocket.js:37 setTimeout (async) Kn.c.current.onclose @ useWebsocket.js:36

[passe4wedo]

Are there any news about this issue?

[lumarel]

I finally was able to upgrade to a recent version, and have to report, this problem is unfortunately still there in:

Ansible AWX version: 19.3.0
Ingress controller: Contour in default config

[fabfabbar]

I finally was able to upgrade to a recent version, and have to report, this problem is unfortunately still there in:

Ansible AWX version: 19.3.0
Ingress controller: Contour in default config

+1

[jdi1983]

If you have some long playbook to be executed such as mines where I have playbooks taking more than 20 minutes I am only seeing blank output windows until the playbook has finished. So I am waiting around 20 minutes to see the whole log output in a second. Isn't it possible to see all the steps while they are executed and not wait for when the whole playbook has been finished so I would be able to see the whole log dumped at the end?

We have the Same Issue and are running AWX 13.0.0 Role Ansible Juniper 2.9.10 Some PB need to wait about 10 to 30 minutes sometimes that the action is finished (systeme snapshot or reload) It's all the more difficult to use as we have approval windows in the WorkFlow ....

[jdi1983]

Hello. Thanks for creating this issue.

I haven't been successful at reproducing this on devel:

Do you see any websocket events in the network tab? And are sockets generally working in other places, like on job and project lists where the status icons are supposed to update?

Can you explain what do you call network tab ? I suppose you are talking about the browser's one ? here is a screenshot at the bottom of FireFox

[amg-web]

select All (left on HTML), and refresh the page

[lumarel]

This is how all websockets end here: It doesn't exchange any message via websocket.

[jdi1983]

Do there is a recommanded Web Browser for AWX ? I'm using FireFox 78.14.0esr

[AdamMack2007]

This is affecting Automation Controller 4.1.0 (AAP 2.1) as well in both Firefox 78.15.0esr and Chrome 96.0.4664.93

Inventory syncs, projects and templates/workflows all hang and will show no activity until I refresh or after a certain duration 2+ minutes.

This was done from a Tower 3.8.3 single node cluster upgraded to AAP 2.1

[ffirg]

related? #9085

[diptajeet]

Is there any updates on this? I am facing similar issues - No refresh of Job Outputs till its done forcefully using the browser. After reading several threads on similar issues, it seems that this issue is pertinent on most of the versions, not just on v18

[pslencinas]

Hi Guys. I'm having the same issue. Do you have any update about it?

AWX: 17.1.0 Docker installation Ubuntu 20

[coreydeli]

I have this exact same issue in version 20 on microk8s on Ubuntu 20.04.4 LTS.

[Donny-Ta]

Hi,

I am also experiencing this issue.

AWX: 21.0.1 (dev) Docker installation Centos 8.0

It seems like the websocket hangs maybe?

Below is the network traffic of a run where I am stuck with the job constantly saying "running" but nothing ever shows up and "running" never turns into "succeeded"

And this is of a run where the job jumps from "running" to "succeeded" but you can see that nothing showed up in the panel once again

If I navigate to the Details tab and then back to Output, the output box does refresh with new data in it.

[xBr0th3rx]

Hello, everyone,

does anyone have an update on this topic?

We also have the same problem, the result is only displayed after the job has been completed or I have to keep refreshing the page.

I don't see an error in the network tab

But in the console tab I get "Websocket Connection ... Failed ... JobOutput".

[amg-web]

I think here we can check best working config: https://github.com/ansible/awx/blob/15.0.1/tools/docker-compose/nginx.vh.default.conf

@xBr0th3rx , @Donny-Ta all working config was published a long time ago.

here is the latest version https://github.com/ansible/awx/blob/17.0.1/tools/docker-compose/nginx.vh.default.conf

my working config:

map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}

upstream minikube {
    server 192.168.49.2:30080;
}

server {
    listen 80;
    listen [::]:80;
    listen 443 ssl http2;
    listen [::]:443 ssl http2;
    server_name awx.example.com;
    keepalive_timeout 65;

    location ~ /.well-known {
        auth_basic off;
        allow all;
    }

    if ($scheme != "https") {
        rewrite ^ https://$host$uri permanent;
    }

    ssl_session_timeout 1d;
    ssl_session_cache shared:MozSSL:10m;
    ssl_session_tickets off;
    ssl_protocols TLSv1.3;
    ssl_prefer_server_ciphers off;
    add_header Strict-Transport-Security "max-age=63072000" always;

    # OCSP stapling
 #   ssl_stapling on;
 #   ssl_stapling_verify on;

    ssl_certificate /etc/letsencrypt/live/awx.example.com/fullchain.pem; 
    ssl_certificate_key /etc/letsencrypt/live/awx.example.com/privkey.pem; 

    # HSTS (ngx_http_headers_module is required) (15768000 seconds = 6 months)
    add_header Strict-Transport-Security max-age=15768000;

    location / {
        proxy_set_header        Host $host;
        proxy_set_header        X-Real-IP $remote_addr;
        proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header        X-Forwarded-Proto $scheme;
        proxy_pass              http://minikube;
        proxy_read_timeout      90;
    }

    location ~ ^/websocket {
        # Pass request to the upstream alias
        proxy_pass http://minikube;
        # Require http version 1.1 to allow for upgrade requests
        proxy_http_version 1.1;
        # We want proxy_buffering off for proxying to websockets.
        proxy_buffering off;
        # http://en.wikipedia.org/wiki/X-Forwarded-For
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        # enable this if you use HTTPS:
        proxy_set_header X-Forwarded-Proto https;
        # pass the Host: header from the client for the sake of redirects
        proxy_set_header Host $http_host;
        # We've set the Host header, so we don't need Nginx to muddle about with redirects
        proxy_redirect off;
        # Depending on the request value, set the Upgrade and connection headers
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
    }
}

[xBr0th3rx]

I think here we can check best working config: https://github.com/ansible/awx/blob/15.0.1/tools/docker-compose/nginx.vh.default.conf

@xBr0th3rx , @Donny-Ta all working config was published a long time ago.

here is the latest version https://github.com/ansible/awx/blob/17.0.1/tools/docker-compose/nginx.vh.default.conf

my working config:

map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}

upstream minikube {
    server 192.168.49.2:30080;
}

server {
    listen 80;
    listen [::]:80;
    listen 443 ssl http2;
    listen [::]:443 ssl http2;
    server_name awx.example.com;
    keepalive_timeout 65;

    location ~ /.well-known {
        auth_basic off;
        allow all;
    }

    if ($scheme != "https") {
        rewrite ^ https://$host$uri permanent;
    }

    ssl_session_timeout 1d;
    ssl_session_cache shared:MozSSL:10m;
    ssl_session_tickets off;
    ssl_protocols TLSv1.3;
    ssl_prefer_server_ciphers off;
    add_header Strict-Transport-Security "max-age=63072000" always;

    # OCSP stapling
 #   ssl_stapling on;
 #   ssl_stapling_verify on;

    ssl_certificate /etc/letsencrypt/live/awx.example.com/fullchain.pem; 
    ssl_certificate_key /etc/letsencrypt/live/awx.example.com/privkey.pem; 

    # HSTS (ngx_http_headers_module is required) (15768000 seconds = 6 months)
    add_header Strict-Transport-Security max-age=15768000;

    location / {
        proxy_set_header        Host $host;
        proxy_set_header        X-Real-IP $remote_addr;
        proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header        X-Forwarded-Proto $scheme;
        proxy_pass              http://minikube;
        proxy_read_timeout      90;
    }

    location ~ ^/websocket {
        # Pass request to the upstream alias
        proxy_pass http://minikube;
        # Require http version 1.1 to allow for upgrade requests
        proxy_http_version 1.1;
        # We want proxy_buffering off for proxying to websockets.
        proxy_buffering off;
        # http://en.wikipedia.org/wiki/X-Forwarded-For
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        # enable this if you use HTTPS:
        proxy_set_header X-Forwarded-Proto https;
        # pass the Host: header from the client for the sake of redirects
        proxy_set_header Host $http_host;
        # We've set the Host header, so we don't need Nginx to muddle about with redirects
        proxy_redirect off;
        # Depending on the request value, set the Upgrade and connection headers
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
    }
}

Hey thanks for the Answer

In which Docker i must set this?

[amg-web]

docker has it. it's for nginx if you use it as a proxy

[lumarel]

I spent some time with that topic again now, and in case of Contour it looks like the Websocket support has to be explicitly enabled. https://projectcontour.io/docs/v1.21.0/config/websockets/

Furthermore it looks to me like it can be enabled via annotations, but I didn't have luck up to now by just adding it to the / rule. https://github.com/projectcontour/contour/issues/3461#issuecomment-800609663

Maybe its the same issue also for other Ingress Controllers.

[Donny-Ta]

@amg-web, I am using the docker deployment with 2 control nodes and that spins up HAProxy. The internal nginx config is the same as the one you posted though, but there's a disconnect with the HAProxy config, I suppose?

[amg-web]

I'm just started with k8s so use simple config

spec:
  service_type: nodeport
  nodeport_port: 30080

it works behind nginx

[arodriguezd]

Hello everyone, I also had this problem, but it only happened when I was behind a Haproxy, if I connected directly to the machine it didn't give me that problem. The configuration I made was based on this repo at https://github.com/kurokobo/awx-on-k3s thanks to kurokobo x his repo is very complete.

• AWX 21.3.0
• AWX-Operator 0.25.0
• Ubuntu 20.X
• K3S

When I wanted to pass it through a Haproxy it showed me the websocket error, so I had to make a configuration, based on this issue https://github.com/dani-garcia/vaultwarden/issues/750 , which although it has nothing to do with awx, it can help you, in my case it helped me and now it no longer shows me the websocket error when it goes through the Haproxy

I hope it helps you Good luck

[tarmael]

Same problem on Red Hat Automation Platform 2.2.1 No idea what version of AWX that's based on

[rooso]

I spent some time with that topic again now, and in case of Contour it looks like the Websocket support has to be explicitly enabled. https://projectcontour.io/docs/v1.21.0/config/websockets/

Furthermore it looks to me like it can be enabled via annotations, but I didn't have luck up to now by just adding it to the / rule. projectcontour/contour#3461 (comment)

Maybe its the same issue also for other Ingress Controllers.

Thanks @lumarel for that hint. Was able to resolve it with the annotation projectcontour.io/websocket-routes: "/websocket" and an additional ingress rule for that path (/websocket). Here an example:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    projectcontour.io/websocket-routes: "/websocket"
    kubernetes.io/ingress.class: contour
  name: awx-ingress
spec:
  rules:
    - host: awx.cluster.io
      http:
        paths:
          - backend:
              service:
                name: awx-service
                port:
                  number: 80
            path: /
            pathType: Prefix
          - backend:
              service:
                name: awx-service
                port:
                  number: 80
            path: /websocket
            pathType: Prefix
  tls:
    - hosts:
        - awx.cluster.io
      secretName: my-secret-name