Blog Post: Using Python for Extract, Transform, Load

[clhenrick]

Documenting the ETL script for NYC Crash Mapper would make a good blog post as it touches on a lot of helpful topics.

Topics to cover:

• why Python works well for data wrangling
• over all structure of the script
• use case / context (NYC Crash Mapper project)
• what is ETL? why is it useful?
• retrieving data from Socrata's SODA API
• constructing SQL INSERT queries to pass to CARTO's SQL API
• using a Heroku Scheduler to run the script
• loading environment variables from a .env file
• running the script locally using foreman
• probably some other stuff

Other:

• draw a diagram showing data flow & process