Reusable pattern on Greenfield Pipeline for Data Classification and Integration with BC Data Catalogue

[ArogeG]

Describe the task Create a reusable pattern for FSA apps that allows teams to integrate with BC data catalogue through BC Catalogue API**.

• The pipeline should carve out a pattern for dev team to integrate their data source with BC data catalogue
• The pipeline should import contents including data classification, custodian, metadata to an DDI/XML format that can be consumed by BC Catalogue. : Other fields required can be found here: https://bcgov.github.io/data-publication/pages/dps_bcdc_w_record_mgmt_ui.html

Acceptance Criteria

• [ ] Data classification file should be in DDI/XML format
• [ ]With creation or update of a database model, DDI/XML file should be updated to include information on changes.
• [ ] This pattern should not integrate data with BC catalogue automatically, but should provide a re-usable means for other FSA programs to connect with BC Data Catalogue Would this be from ER studio or from dev environment?

Other useful resources -https://bcgov.github.io/data-publication/pages/tips_tricks_making_useful_metadata.html https://bcgov.github.io/data-publication/pages/dps_bcdc_api_w_resource_mgmt.html

(First stab of ticket, further refinement is required

[gormless87]

Some options for storing DC: Store DC as parse-able column comments Store DC in metadata tables Store DC is a separate repo (E/R Studio)

Document API's rather than database

@rparfig - please advise

[basilv]

DC = data classification. Some considerations:

1. If we aren't maintaining the data model in E/R studio but updating E/R studio from application script, that will make it hard to use E/R studio as the source of truth for additional metadata.
2. I'd like to make it easy for a dev team to modify the db schema (e.g. add table or add column) and also supply necessary metadata.
3. Would be nice to have the metadata (data classification) defined in the same source as the schema definition (DDL).