Create better data update flow - Githubissues

edman3d / ageofinfo-gcp

Age of Info GCP

1 stars 0 forks source link

Create better data update flow #51

Closed edman3d closed 11 months ago

edman3d commented 11 months ago

Goals:

Skip csv to JSON step
Skip renaming of each file and dragging into project
Use a csv parser to form the data types etc.

Proposed flow (Keep using Google Sheets as place of truth for data):

[x] Create new folder in project /rawData/ which has the raw csv files from Google Sheets (such as /rawData/AOE2_DE Data v3 - Technologies.csv)
[x] Create a new script in package.json that parses each of the csv files and outputs them as JSON into /data/ folder (overwriting whats in there)

Then flow looks like this:

Update data on sheets
Download each of 4 CSVs
Drag CSVs into rawData folder
Use the npm run generate-json-data script once (no need to parse every time we save code)

edman3d commented 11 months ago

done & released on 10/28/23. should make data validation tests way easier