Uneven usage of Instances in the instance group

mimianddaniel commented 2 weeks ago

Please confirm the following

[X] I agree to follow this project's code of conduct.
[X] I have checked the current issues for duplicates.
[X] I understand that AWX is open source software provided for free and that I might not receive a timely response.
[X] I am NOT reporting a (potential) security vulnerability. (These should be emailed to security@ansible.com instead.)

Bug Summary

Running: 23.7.0 AWX with 2.11.0 Operator. There are 4 same resource settings for awx-task instances and they are all in ready status with 40 forks avail per instance. I am seeing uneven jobs being scheduled among them. It seems like the very first one listed in the Instances is used mostly, in terms of ratio, 96% of the time, the first instance is used for running the jobs. AFAIK, there is no way to choose the specific instance from the Project.

I expect that each instance will run somewhat of a distributed job among the 4 instances and not on a single instance all the time.

The odd thing is that out of 3 AWX clusters we run, only one cluster instance shows this behavior.

AWX version

23.7.0

Select the relevant components

[X] UI
[ ] UI (tech preview)
[ ] API
[ ] Docs
[ ] Collection
[ ] CLI
[ ] Other

Installation method

kubernetes

Modifications

no

Ansible version

No response

Operating system

No response

Web browser

No response

Steps to reproduce

Under administration look at Instances and select the first instance. look at the number of jobs run and compare with other instances.

Expected results

Expect to see more less even or somewhat normalized numbers on each instances running the jobs. instance

Actual results

See 96% of the jobs running on the first instance and the rest does %4

Additional information

No response

jessicamack commented 2 weeks ago

The instances are working as expected. AWX does not attempt to evenly distribute jobs between instances.

mimianddaniel commented 2 weeks ago

The instances are working as expected. AWX does not attempt to evenly distribute jobs between instances.

Thanks for the reply, this is not impacting the service so, I don't really see a problem, but it's odd to see only one instance of the cluster showing this behavior .

ansible / awx