Document Ingestion Pipeline

urbanbigdatacentre / ideamaps-models

Models of deprivation sub-domains for the IDEAMAPS data ecosystem project. This repo contains the source code used to run models and the model outputs. It also contains logic to upload model outputs to the IDEAMAPS platform.

MIT License

2 stars 0 forks source link

Document Ingestion Pipeline #6

Closed andymithamclarke closed 5 months ago

andymithamclarke commented 6 months ago

We need to create a document that describes the process of ingesting data from the model repo to the platform.

This document needs to detail the following elements:

Data formats for outputs
Hasura endpoints and requirements
Links to the schema
How to use the Action
Quality Control processes
Error and response codes - with messages

...

This issue can be closed once the document has been created and uploaded to the /models repo within it's own isolated space.

andymithamclarke commented 6 months ago

This should also include the data management side of things.

Serkan suggests that we may not want to keep old models - a clean up process. We should document this as part of the document.

andymithamclarke commented 6 months ago

Reccomendation from Serkan - should we not be storing our data on Github - we should be using a data repository to store these files instead.

diegopajarito commented 5 months ago

Data storage in GitHub is to be defined.

The initial version of the documentation is defined. Closing this issue to start a new one for specific details on the technical design.