search

NYCPlanning / db-factfinder

data ETL for population fact finder (decennial + acs)
https://nycplanning.github.io/db-factfinder/factfinder/
MIT License
2 stars 3 forks source link

pipeline related operations #33

Closed SPTKL closed 3 years ago

SPTKL commented 3 years ago

Workflow: (everything happens in a python script)

  1. run python script -> create output/acs.csv -> upload output/acs.csv to s3
  2. load output/acs.csv to database using psql / psycopg.cursor.copy_expert

To-do

  1. write dataframe/file to s3 using s3fs https://s3fs.readthedocs.io/en/latest/api.html -> see S3FileSystem.put
SPTKL commented 3 years ago

implemented