Fan-out project updates across cluster

ansible / awx

AWX provides a web-based user interface, REST API, and task engine built on top of Ansible. It is one of the upstream projects for Red Hat Ansible Automation Platform.

Other

13.9k stars 3.4k forks source link

Fan-out project updates across cluster #289

Open AlanCoding opened 6 years ago

AlanCoding commented 6 years ago

ISSUE TYPE

Feature Idea

SUMMARY

As an extension of #287, this issue is to implement the ability to easily fan-out a project update to all nodes in the cluster at once.

With 287 solved, this means that subsequent jobs will not need access to source control. Without the fan-out ability, the solution to low-availability source control will only work for a standalone deployment of AWX. The fan-out will allow a user to prepare their entire cluster for continuous operation with total loss of connectivity to source control.

STEPS TO REPRODUCE

Deploy AWX with multiple instances in the cluster
Update a project

EXPECTED RESULTS

The ability to have this update happen on all nodes on the cluster - bring them all up to sync with the current source control, such that subsequent jobs using that project do not need to reach out to source control.

ACTUAL RESULTS

Can not do this.

wenottingham commented 6 years ago

Questions:

what does this mean for offline cluster nodes that then rejoin the cluster?
what does this mean if it fails on node X?
what does this mean for roles/requirements.yml that refer to git HEAD of a role repo... that may change between syncs on different cluster nodes?

AlanCoding commented 6 years ago

All good questions.

what does this mean if it fails on node X?

Technically this is an optimization. If connectivity does exist, then the project sync on a job ran on node X should sync to the correct revision before running a job.

That's just my 2 cents, of course. Same would apply to the 1st bullet, and I can't come up with an answer to the 3rd bullet off the top of my head.

AlanCoding commented 5 years ago

After working on https://github.com/ansible/awx/pull/2287, I think I have a fairly good grasp on an implementation that would work.

on whatever node it lands on, perform a check type project update
in final_run_hook or post_run_hook
- create "sync" type jobs for all nodes in system
- submit a broadcast_all task with kwargs of node names & job ids
In that task, import RunProjectUpdate, with some slightly new re-entrant logic, pick up existing update id, and run the task
in the original node, poll the database for status of the child sync jobs, also watching for cancel_flag

AlanCoding commented 5 years ago

I withdraw my last comment. Since the introduction of sliced jobs, it makes much more sense to continue the pattern of workflow-like jobs on an as-needed basic. My new preferred design:

as now, perform check type project update
Introduce new changed status to show when project_update.scm_revision != project.scm_revision, thereby signaling a change to canonical version
If and only if project update is changed, spawn a followup workflow job with project syncs for entire cluster (perhaps excluding the origin node)
- this requires additional work to manage update parameters
this workflow job may succeed or fail. We don't care. User could relaunch it, but again... we don't care. Out of data nodes will run their sync on-demand. This problem is already solved.

kevmimcc commented 3 years ago

what if I have a single node setup. Today github was down and I was unable to run a job that has not had updates in months

jangel97 commented 1 year ago

From a customer point of view and in my humble opinion, it may make sense that once I click on sync project, it gets synced in all controllers, and so, when I spawn a job I do not need to wait for it to be synced in whatever controller instance that gets assigned to control my job.