awx ad-hoc command (ping) job stay in running status indefinitely

bskou57 commented 9 months ago

Please confirm the following

[X] I agree to follow this project's code of conduct.
[X] I have checked the current issues for duplicates.
[X] I understand that AWX is open source software provided for free and that I might not receive a timely response.
[X] I am NOT reporting a (potential) security vulnerability. (These should be emailed to security@ansible.com instead.)

Bug Summary

Hi

I run ping ad-hoc command via awx web interface with high fork

At the beginning it is running faster and finally stay in running state with forking processes 2 by 2 only

why is forking value/capacity decrease over the processing time ?

Thanks for your support

AWX version

23.5.0

Select the relevant components

[ ] UI
[ ] UI (tech preview)
[x] API
[ ] Docs
[ ] Collection
[ ] CLI
[ ] Other

Installation method

kubernetes

Modifications

no

Ansible version

2.14.2

Operating system

Redhat 8.8

Web browser

Chrome

Steps to reproduce

run awx adhoc command (ping module for example) via inventory on big server list (over 4000) and never finished Note that I have the same behavior with command module

Expected results

job completed in failed or successful status but stay in running status indefinitely.

Actual results

adhoc (ping module) command job is running indefinitely

Additional information

Nothing to highlight in pods logs or podman logs

fosterseth commented 9 months ago

thanks for raising this issue. how are you monitoring # of active forks at a time? Are you inferring that based on job stdout, or are you exec-ing into pod and looking at running ansible-playbook PIDs?

what fork value are you setting in the ac-hoc command?

where is this job running (i.e. remote execution node, container group, cluster automation pod, etc).

Are there CPU limits set on the cluster where this is running? How many actual CPU cores is on that k8s worker node?

also this sounds more like an ansible issue, not AWX, and you might gain traction by creating a new thread on forum.ansible.com under Get Help section

bskou57 commented 9 months ago

Hello

I used ps command to count number of ssh connection which should be equal zo number of forks

Yes I am looking for pids

we do not have cpu limit on execution node (where the podman run) and the control node where I have awx installed with pods on k3s single cluster node

How can I check resources limite on k3s ?

ok will raise request to ansible team

Thanks for your support

ansible / awx