feat: plugin loading from contract

roleyfoley commented 3 years ago

Description

executor implementation of https://github.com/hamlet-io/engine/pull/1469 which implements plugin sourcing and configuration from a CMDB via a contract.

The engine generates a loader contract which outlines the plugins which need to be loaded. Using this information the executor establishes a plugin cache and retrieves the plugins from their sources.

The cache allows for the plugins to be shared across different districts when working locally

The process also generates a state file which can be used to determine the version of the different hamlet plugins that were used for a given pass. If the setup script isn't used to load the plugins then the existing processes will continue working

Also adds some utility routines to handle urls for git auth

Motivation and Context

The motivation behind this change is to handle version management for plugins as part of a CMDB. This ensures consistency across users working in the CMDB. It also provides visibility on where plugins are sourced and how they behave

How Has This Been Tested?

Tested locally

Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[ ] Refactor (non-breaking change which improves the structure or operation of the implementation)
[X] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Followup Actions

[x] Requires https://github.com/hamlet-io/engine/pull/1469

Checklist:

[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.
[ ] I have added tests to cover my changes.
[ ] All new and existing tests passed.
[X] None of the above.

roleyfoley commented 3 years ago

There is a lot of git porcelain/plumbing commands used here, this is mostly to reduce the need to go out and retrieve things from git and reduce the traffic required to make things work

roleyfoley commented 3 years ago

Example workflow

cd < your account or segment dir> 
${GENERATION_DIR}/setup.sh 
# Will clone and setup your providers 

${GENERATION_DIR}/createTemplate.sh 
# Will generate templates like it normally does based on the provider state

RossMurr4y commented 3 years ago

I'm not clear on how to test this one - I've checked out locally the engine branch from hamlet-io/engine#1469 alongside this PR, but the git commands added in this PR are trying to checkout content that only exists on the fork.

master└─ $ $GENERATION_DIR/setup.sh -i composite
Enter MFA code for arn:aws:iam::017206355635:mfa/RossMurr4y: 
(Info) Generating generationcontract file ...

(Info) Generating documents from generation contract /tmp/cot_Ydxtvj/create_template_S02B7o/create_template_8wDyOJ/results/loader-generation-contract.json
(Info) Generating providercontract file ...

(Info) Differences detected ...
(Info) ... updating loader-generation-contract.json ...
(Info) ... updating loader-providercontract.json ...
fatal: no upstream configured for branch 'feat-cmdb-providers'
fatal: No such remote ''
(Info) Loading providers from contract...
(Info) [*] id: - type:
/home/hamlet/cmdb/hamlet/executor/cli/setup.sh: line 204: /home/hamlet/.hamlet/cache/providers/provider-state.json: No such file or directory

Is there anything else required in order to test @roleyfoley ?

roleyfoley commented 3 years ago

I'm not clear on how to test this one - I've checked out locally the engine branch from hamlet-io/engine#1469 alongside this PR, but the git commands added in this PR are trying to checkout content that only exists on the fork.

master└─ $ $GENERATION_DIR/setup.sh -i composite
Enter MFA code for arn:aws:iam::017206355635:mfa/RossMurr4y: 
(Info) Generating generationcontract file ...

(Info) Generating documents from generation contract /tmp/cot_Ydxtvj/create_template_S02B7o/create_template_8wDyOJ/results/loader-generation-contract.json
(Info) Generating providercontract file ...

(Info) Differences detected ...
(Info) ... updating loader-generation-contract.json ...
(Info) ... updating loader-providercontract.json ...
fatal: no upstream configured for branch 'feat-cmdb-providers'
fatal: No such remote ''
(Info) Loading providers from contract...
(Info) [*] id: - type:
/home/hamlet/cmdb/hamlet/executor/cli/setup.sh: line 204: /home/hamlet/.hamlet/cache/providers/provider-state.json: No such file or directory

Is there anything else required in order to test @roleyfoley ?

What's in your CMDB as the provider you are loading?

roleyfoley commented 3 years ago

Converting to draft based on https://github.com/hamlet-io/engine/issues/1480

hamlet-io / executor-bash