Implement externalise config for CDK constructs

[victorskl]

Context:

• About the task - major baseline code building block setup for constructs and config sharing

Design Guide:

• No Parameter Store -- i.e. Use of SSM consider anti-pattern as CDK construct sharing mainly due to "dynamically" modifiable as runtime after deployment made (ditto Flo)
• Can be "static config files" -- json yaml
• Consider dotenv if that help
• Must study and adopt the guideline from the following two articles:
  • https://github.com/aws-samples/aws-deployment-pipeline-reference-architecture
  • https://github.com/leegilmorecode/Serverless-AWS-CDK-Best-Practices-Patterns

Actions:

• Create a new feature branch out from refactor-setup i.e. pending PR #13
• Consider refactor-setup as future main
• Refer any TODO tasks that denote comments - // FIXME externalise config
• Refactor them accordingly with the static config arrangement setup

[andrewpatto]

Can you define what config items you mean here? vpc ids? subnets? or app config?

[victorskl]

Sure. At the mo, this scopes to -- how to refer/reconstruct/import what has created in long-running infrastructure (stateful) stack into short-live (self-mutating in CodePipeline) application (stateless) stack.

Currently is tightly coupled.

https://github.com/umccr/orcabus/blob/3cae7ada7a3d2f0c7907df2dadbd7894c809b59d/lib/workload/orcabus-stateless-stack.ts#L19-L28

[andrewpatto]

Whilst I totally agree with trying to avoid SSM parameters - it seems like the consensus of the internet is that the only practical way to share from disconnected 'stateful' stacks to 'stateless' stacks in SSM. (I tried multiple other ways that all failed - in particular they fail in not being able to be handled in the synthesis phase - which is where the CDK lookup() stuff comes in)

[victorskl]

Sure. Understood. Trust me, currently Portal use lot SSM for this between disconnected (stateful) terraform to (stateless) (self-mutating) serverless/local-dev, yes.

Historically in early Portal days inception -- it was environment variables .env -- that get out-of-hand as its has grown.

Here is the retrospect; changes from .env to SSM.

• https://github.com/search?q=repo%3Aumccr%2Fdata-portal-apis+.env&type=commits
• https://github.com/search?q=repo%3Aumccr%2Fdata-portal-client+.env&type=commits
• https://github.com/search?q=repo%3Aumccr%2Finfrastructure+portal+parameter+store&type=commits

Back then, thought was SSM is natural fit. Because elsewhere tech stack; in more generally, we do this using some key-value store for decoupling config; like Consul, etcd, redis... Or, some sort of service discovery...

I saw you have used Service Discovery in AWS... or Cloud Map or the like?? I will read/dig up and, probably need help/pick your brain -- probably in devops catchup...

[victorskl]

This is now Trello-ed -- https://trello.com/c/4hmvdUDX/1422-orcabus-v1-implement-externalise-config-for-cdk-constructs

And, will be continue tracked there.

Part of Trello migration.

[victorskl]

Use as follows in order of preference.

1. Use constants (TypeScript source file)
2. Use SSM parameter store
3. Use Secret Manager

Closing.