search

moj-analytical-services / data-and-analytics-engineering-tech-radar

https://moj-analytical-services.github.io/data-and-analytics-engineering-tech-radar/
0 stars 0 forks source link

Replace "HOLD" with "RETAIN" ring assignment #299

Closed SoumayaMauthoorMOJ closed 1 month ago

SoumayaMauthoorMOJ commented 1 month ago

Context

@calumabarnett has requested changes with the ring assignments:

I think I would frame hold solely around technologies we currently use (and are happy to continue using for the time being) but don’t want to adopt in new projects If we think something is not worth investigating, I don’t think it should be on the radar Retire is where we should be taking steps to replace

This ticket is about confirming what ring assignments we should use.

A good example to consider is black: we should be replacing with ruff but perhaps not worth the hassle if everything is working ok.

Another example is dagster: there's too much overlap with Airflow so we it's not worth investing time prototyping without first agreeing with AP

Note that we want to limit the number of blips to 25 per category so altogether 100 to make it meaningful and maintainable, so the assignments should help us to limit the number of blips on the radar.

Options

Option 0: Zalando

Assignment Reaction Short Explanation Example Radar
ADOPT 🚀 Expand / carry-on using Inner
TRIAL 👀 Start using Third
ASSESS 👍 Prototype Second
HOLD ☹️ Not (yet) recommended, migrate if possible Outer

Option 1: Existing

Combine TRIAL into ADOPT, and split off RETIRE from HOLD

Assignment Reaction Short Explanation Example Radar
ADOPT 🚀 Start / expand / carry-on using create-a-pipeline, AWS, Python, tech-radar Inner
ASSESS 👍 Prototype dlt Second
HOLD 👀 Not (yet) recommended dagster , sql mesh Third
RETIRE ☹️ Migrate to alternative if possible glue 2.0, glue, register-my-data, pulumi Outer
CLOSE 👎 Not relevant typescript

Option 2: Calum's suggestion

Exclude technologies which are not (yet) recommended and split up RETIRE into REPLACE and RETAIN

Assignment Reaction Short Explanation Example Radar
ADOPT 🚀 Start / expand / carry-on using create-a-pipeline, AWS, Python, tech-radar Inner
ASSESS 👍 Prototype dlt Second
RETAIN 😄 Use but don't expand register-my-data, glue Third
REPLACE ☹️ Migrate to alternative glue 2.0, pulumi Outer
CLOSE 👎 Not relevant, not (yet) recommended typescript, dagster, sqlmesh

Option 3

Same as option 2 but split off PAUSE from CLOSE

Assignment Reaction Short Explanation Example Radar
ADOPT 🚀 Start / expand / carry-on-using create-a-pipeline, AWS, Python, tech-radar Inner
ASSESS 👍 Prototype dlt Second
RETAIN 😄 Use but don't expand register-my-data, glue Third
REPLACE ☹️ Migrate to alternative glue 2.0, pulumi Outer
CLOSE 👎 Not relevant typescript
PAUSE 👀 Not (yet) recommended dagster, sqlmesh
SoumayaMauthoorMOJ commented 1 month ago

Following discussion with @tmpks @pricemg agreed to proceed with Option 3

SoumayaMauthoorMOJ commented 1 month ago

See #302