search

Data472-Individual-Project-Pipeline / Docs

All docs about work flow, requirements, and technique
MIT License
0 stars 0 forks source link

Define Data Contract #1

Closed aemooooon closed 5 months ago

aemooooon commented 6 months ago

What kind of Data Contract should we follow? SDMX? How to implement it?

Jake1441 commented 6 months ago

We can follow a SDMX outline which will save us having to define certain values a recommendation was to not strictly follow the format but just use it as a guideline.

Implementation of the SDMX is simply using it as a guideline i.e how we format our datasets or names for columns, from a technical perspective we could use Docker postgres sql containers using postgrest api to pull data from each different ingress source

Jake1441 commented 5 months ago

We created a pipeline process for ingestion of data.

We expect cleaning and sanitizing to be completed by each individual

To follow the data protocol and sdmx guideline as best possible

To look at guides on implementing different services such as flask