search

data-dot-all / dataall

A modern data marketplace that makes collaboration among diverse users (like business, analysts and engineers) easier, increasing efficiency and agility in data projects on AWS.
https://data-dot-all.github.io/dataall/
Apache License 2.0
221 stars 78 forks source link

Feature Flags for different environments like Dev, UAT, Prod #1136

Open shivalkarrahul opened 3 months ago

shivalkarrahul commented 3 months ago

Hello,

We have a requirement as follows.

  1. We have 3 different Environments, viz Dev, Staging and Prod.

  2. We should be able to have Dev, Staging & Prod Environment on different versions of Data.all (Dev on 2.3.0, Staging on 2.2.0, Prod on 2.1.0)

  3. There should be a way to Enable or Disable features based on Environments. e.g. Dev can have modules.dashboards.active enabled, and Staging/Prod can have it disabled. (We believe different config.json would come into picture here) e.g Dev environment would be having few functionalities which are under development and not present in staging or Prod, Staging Environment would be having few functionalities getting tested but will not be moved to Prod until UAT is done

  4. Do we need to have config-dev.json, config-staging.json, config-prod.json files to enable to disable features?

  5. How do we enable or disable UI for the features?

Any guidance would be appreciated.

Thanx in advance.

dlpzx commented 3 months ago

Hi @shivalkarrahul thanks for the issue! Definitely an interesting one.

For the issue on having multiple code versions in different deployment environments we can either:

  1. go for a trunk-based approach and define a ManualApproval stage between the development stages. In data.all there is a parameter called "with_approval": "boolean_ADD_CODEPIPELINE_APPROVAL_STEP|DEFAULT=false", that creates a manual approval stage between the deployment environments.
  2. or go for a gitflow approach and deploy multiple CICD pipelines, each one would be like an standalone data.all deployment reading from a branch in the repository. You would control promotion between stages by branch merging rules.

The second issue is related to the config.json files. For gitflow approach, every deployment would have its own config.json, so it is already solved. For the trunk-based approach which is the standard in data.all we would need to introduce multiple config.json files as you pointed out in point4.

Last thing, I am not sure I understand your point number 5, could you give us some more details?

Bests

petrkalos commented 3 weeks ago

@shivalkarrahul currently our config system is a bit convoluted.

We have 2 config files cdkjson and configjson the first is responsible for configuring only the cdk code. The second though is configuring the backend (lambda/ecs tasks, migration scripts etc), the frontend but also the cdk. As suggested one solution could be introducing multiple files but that will add even more mental overhead. So we want to take some time and design a better solution.

In the meantime in #1318 I introduced a way to override config flags using an SSM parameter (i.e /dataall/dev/configjson). The caveat for using it is that you have to keep in mind that if the flag you are changing also requires changes in the cdk or frontend you should manually release a change in the pipeline.

anmolsgandhi commented 3 weeks ago

Thanks @petrkalos. Based on the above, we will be moving this to v2.7 scope.