Add the concept of reconcile Procedures

[JohnStrunk]

Describe what this PR does In #51 reconcile Actions were introduced. This PR brings in the concept of a Procedure as a collection of Actions. A single Procedure should fully define the steps necessary to reconcile the operator state.

Multiple Procedures may exist within the operator to handle upgrades. Each Procedure has a minVersion from which it can upgrade.

Is there anything that requires special attention? There is currently just pseudo-code to plug this into the actual sdk reconcile function.

Related issues: Follow-on to #51

[rohantmp]

Do we want to redo actions when they're listed as in both Procedure.actions and Procedure.actions[i].prereqs[j] or are we okay parsing the Procedure to generate a list of actions to be executed in the correct order?

I can see this is a safety/perf tradeoff, and that it's good to at the very minimum ensure that the prereqs are expressed, but does it really help to actually run them each time?

Edit: NVM, I guess we could just not include actions that are in Procedure.actions[i].prereqs[j] in Procedure.actions :P What about repeated prereqs?

[JohnStrunk]

Do we want to redo actions...?

I think I clarified over chat, but just to log the answer here (and to make sure I did actually understand/answer the question)...

Each Action has a cache: https://github.com/gluster/anthill/blob/fb6d2c616e40c9020dca9f4e202e9a30260effc4/pkg/reconcileaction/reconcileaction.go#L35 ...that records the result of execution, and unless that cache is cleared (action.Clear()), the Action.action func will not be re-run. The Procedure's Execute() starts by clearing that cache for all actions and their prereqs before evaluating the component Actions. This has the effect of permitting all Actions (including prereq actions) to execute at most once per invocation of Procedure.Execute() (aka, a controller reconcile cycle).

The benefits of the above are:

• Prereqs are evaluated consistently. If I have 2 Actions that have a common prereq, I am assured that the prereq will return the same result in both cases
• Expensive checks are cached. We may have to probe gd2 or other systems for checking, and repeatedly doing this could cause unnecessary load and slow down the reconcile loop.

[rohantmp]

I see now :) LGTM!

[humblec]

@JohnStrunk few comments from my side.

[rohantmp]

@JohnStrunk I want to clarify some things: 1) When the next version of operator-sdk comes out We'll have to generate the scaffolding for the new version and manually apply our changes? with diff,etc?

2) We will create a new procedure version when GCS changes enough that the same actions have to be implemented differently?

3) If both 1) and 2) are true, then how do we handle the versioned actions? subpackages like v0? with an exported procedure and exported actions?

[humblec]

LGTM .. Merging ..