Generate new sources.csv from PostgreSQL database

MobilityData / mobility-feed-api

Apache License 2.0

8 stars 3 forks source link

Generate new sources.csv from PostgreSQL database #779

Open emmambd opened 1 week ago

emmambd commented 1 week ago

Describe the problem

As discussed, we want to shift away from the catalogs repo being our source of truth so we can include more of the dynamic info that comes from our data pipeline.

Proposed solution

Override location with extracted locations
Generate new spreadsheet from postgres database
Re-generate the spreadsheet every Tuesday and Friday

Alternatives you've considered

No response

Additional context

No response

davidgamez commented 1 day ago

Tasks:

Keep the "legacy" csv file and add a new csv based on the DB information
Investigate if we can use the same shortened URL for the new sources.csv
Re-generate the spreadsheet every Tuesday and Friday
Test populate script with DB that has feeds not present in the "legacy" csv