search

ministryofjustice / analytical-platform

Analytical Platform • This repository is defined and managed in Terraform
https://docs.analytical-platform.service.justice.gov.uk
MIT License
8 stars 4 forks source link

Investigate and Define solution for Schema Versioning #898

Closed PriyaBasker23 closed 12 months ago

PriyaBasker23 commented 1 year ago

Schema versioning

  1. Investigate options of how we could implement it
  2. Record the solution in confluence
  3. Present to TA's and Alex VF , Greg

DOD: Create ticket fir implementation

Ed-Bajo commented 1 year ago

These are other questions that needs to be considered under the versioning work:

  1. What are data engineering requirements for managing data refreshes?
  2. What are data engineering requirements for managing schema changes?
  3. Is there one approach to this, or multiple approaches, across different DE / MoJ teams?
  4. Briefly describe the approach(es)?
  5. Anything that has proven to be difficult in the current approach(es)?
  6. What benefits do data engineering’s customers get from the current approach(es)?
  7. For what (if anything) do data engineering’s customers need or use the versioning information for data refreshes?
  8. For what (if anything) do data engineering’s customers need or use the versioning information for schema changes?
  9. How do data refreshes and schema changes relate to (or interact with) project versions (version control on GitHub)?
  10. How will existing things change with a data product architecture coming into play?