adhoc jobs block other jobs from being processed in the queue

jchilcher-godaddy commented 1 year ago

Please confirm the following

[X] I agree to follow this project's code of conduct.
[X] I have checked the current issues for duplicates.
[X] I understand that AWX is open source software provided for free and that I might not receive a timely response.
[X] I am NOT reporting a (potential) security vulnerability. (These should be emailed to security@ansible.com instead.)

Bug Summary

When you submit an adhoc job all other jobs (other adhoc, and job templates) are stuck in a pending state until the running adhoc job completes.

AWX version

23.4.0

Select the relevant components

[ ] UI
[ ] UI (tech preview)
[ ] API
[ ] Docs
[ ] Collection
[ ] CLI
[X] Other

Installation method

kubernetes

Modifications

no

Ansible version

No response

Operating system

No response

Web browser

Firefox

Steps to reproduce

submit any adhoc job (I used a shell command sleep 30; whoami)
submit a job from "Templates"
observe the job queue

Expected results

all jobs in the unified job queue should process according to capacity. I expect to see more than one job in a running state.

Actual results

The adhoc job is in a running state. All other jobs are stuck in pending waiting for the adhoc command to finish.

Additional information

I am using a container group to run jobs. Capacity of the managed nodes have been visually verified to be < 25% cpu and mem usage. I tried creating a duplicate container group for job templates, which still was stuck in pending, but now with a message waiting for adhoccommand-371 to finish

AlanCoding commented 1 year ago

Are all jobs blocked, or only those running in the same inventory as the ad hoc command? Because if that's the case, then this is expected behavior.

https://github.com/ansible/awx/blob/d8a28b3c06a8f1019fb8ab29e836eb180d4f4d88/awx/main/scheduler/dependency_graph.py#L88-L90

jchilcher-godaddy commented 1 year ago

adhoc jobs appear to block only when running jobs from the same inventory, yes. I had no idea that was expected. I assumed that a job was a job and they could all be run simultaneously (excluding the inventory/project updates).

Seems like a fair workaround would be to duplicate the inventory and use one for tempates and the other for adhoc jobs, does that sound about right?

AlanCoding commented 1 year ago

Discussion has come up before, and we would prefer to allow these rules to be fully user-customizable, @chrismeyersfsu was a particular advocate for this. There may be some existing related issues. The dependency_graph.py contains most of this type of blocking logic, which I would prefer to call "exclusion rules" to distinguish from other types of dependencies like update-on-launch dependencies. As you can see, that file isn't terribly complicated, but I don't quite know how to put that in a language where a user could customize it with a dictionary-valued setting, for example.

fosterseth commented 1 year ago

I think we could change the way we mark adhocommands so that they aren't treated as inventory updates. Instead we can introduce a mark_adhoccommand method. We should still block on inventory updates though.

https://github.com/ansible/awx/blob/d8a28b3c06a8f1019fb8ab29e836eb180d4f4d88/awx/main/scheduler/dependency_graph.py#L141

imDarshanGK commented 4 days ago

Hey, is this issue still open? I can work on this if needed.. Also saw #14645

ansible / awx