Make Reference lookup processing dyanmically loaded

[roleyfoley]

Expected Behaviour

When multiple providers are loaded you can perform reference lookups across the loaded providers

Current Behaviour

Currently we override the getExistingReference and getReference functions each time we load a provider. As a result of this we need to be careful that we load the providers in an appropriate order or that we don't load overlapping providers ( i.e. the aws and azure providers )

Possible Solution

Replace the existing functions with a common function which then performs a provider specific lookup

Context

This is to handle a couple of user cases

• Loading all providers at run time
• Handling cross provider references

[ken9ross]

Proposed solution

Define a single instance of getReference and getExistingReference in the engine providers/shared/services/resource.ftl (with the additional parameter of providerName, which will do the common processing, and then call invokeFunction from base.ftl on "get"+providerName+"Reference", and a function called get{providerName}Reference will then need to be defined in every (provider within) providerName/services/resource.ftl

[roleyfoley]

@ken9ross the name of the function to call should align with our current naming for dyanmic macros

<provider>_referenceformat_<deployment_framework>_<entrance>

So this would say for a given provider ( aws ) we have a function called referenceformat and that can be different for each deployment framework ( cf - Cloudformation ) and different entrances ( deployment ) being the initial one

This allows for us to have multiple internal reference formatting services depending on what you are doing

[ken9ross]

@roleyfoley I don't understand how the getReference function (this specific case) can be different for the different frameworks nor entrances

On the assumption that's correct, should the function name be aws_getReference (and azure_getReference), or should there be two trailing underscores (as the separators without values for deployment framework and entrance). Is there a case convention for provider (always uppercase, always lowercase, title case, something else)? I assume it's just whatever is put in the AZURE_PROVIDER or AWS_PROVIDER assignment in provider.ftl, correct?

[RossMurr4y]

An AWS function to get a reference for Cloud Formation Deployments

[#function aws_referenceformat_cf_deployment] ...

An Azure function to get a reference for an ARM Deployment

[#function azure_referenceformat_arm_deployment] ...

A shared provider function to get a reference for a schema

[#function shared_referenceformat_default_schema] 

Done this way, functions that perform dynamic lookups based on the existance of a function/macro with the specified name can be used - it tells Hamlet "I'm going to give you an array of strings, find me the first function that matches that naming convention and invoke it"

[roleyfoley]

@ml019 is including this as part of the input processing work.

Provider contributes a seeder as part of input processing which standardizes outputs to conform with what we expect. This should allow for cross provider referencing

[roleyfoley]

@ml019 Could you look at this as a next step of state handling

[ml019]

The introduction of input pipelines goes some of the way;

• the config pipeline has a normalisation stage to which providers can contribute transformers to convert their specific form of stack outputs (contributed via the cmdb config seeder) into the standard pointSet format. Implementations are in place for azure and aws
• the state pipeline has a shared seeder for the cmdb stage that contributes any matches from the pointSets provided by the config pipeline.
• the getStatePoint() function uses the state pipeline to obtain the current state of a point.

There are currently still separate copies of getReference in each of the deployment providers. The next rationalisation is to move these into the shared provider.