AWX live job output stops updating and/or gets disconnected

Please confirm the following

[X] I agree to follow this project's code of conduct.
[X] I have checked the current issues for duplicates.
[X] I understand that AWX is open source software provided for free and that I might not receive a timely response.
[X] I am NOT reporting a (potential) security vulnerability. (These should be emailed to security@ansible.com instead.)

Bug Summary

I’ve found that sometimes, in long-ish job runs, say 10-20 minutes or longer, with about 200 hosts, the log output in the “Output” tab of the job that’s currently running will stop updating. When I look at the pod logs for the automation-job pod, the job is still running and logging, but the AWX UI is not updating with the new logs. This makes it seem like the job is stuck.

Often, when the job completes, the log data will load into the textarea, but while it’s running it’s stuck.

Initially saw this in AWX 22.3.0, running in Google Kubernetes Engine (GKE). Upgraded to 24.6.1 this week and still seeing the issue.

Note: this doesn't happen every time, but by my guesstimation, I'd say it's 20-30% of the time with these longer-running jobs.

AWX version

24.6.1

Select the relevant components

[X] UI
[ ] UI (tech preview)
[ ] API
[ ] Docs
[ ] Collection
[ ] CLI
[ ] Other

Installation method

kubernetes

Modifications

Ansible version

No response

Operating system

No response

Web browser

Chrome

Steps to reproduce

Create inventory with many hosts and/or job template that runs for > 10 minutes (not sure which of these conditions are necessary).
Watch job output and click "Follow"
Wait for output

Expected results

Output is updated in real time as job proceeds

Actual results

UI stops updating, but I can look in the pod logs for automation-job and see that it's still running. It seems to eventually "reconnect" and start updating again, but there will be 10+ minute gaps where I don't see any updates and need to use an external app to check kube pod logs. Refreshing the UI (with browser refresh button) does not help; it will still show the output stuck at the exact same spot.

Additional information

No response

ansible / awx