Home Page proof of concept and ETL POC

[smcalilly]

Overview

Context

First let me give a quick overview of this application.

The backend is a Django app. It serves JSON to the React frontend.

I have several goals for the current iteration-in-progress:

• Create a home page with a map of Mississippi that shows the counties and has some interactions with the map. A user will be able to see some demographic information about each county. Once I nail down these patterns,
• Create a voting precinct map explorer. A user will be able to examine a county's voting precinct boundaries, polling locations, and that precinct's demographics. Ultimately I want to be able to compare the location of the polling places for each election in the past 20 years. I'm working towards something like these maps but I want to use React so that there can be some exploration and interactions with the map.
• Build up a database for Mississippi data, consumable via JSON APIs.

PR

There are some things that are wrong with this application's code that isn't include in this PR, especially since it's been where I've learned Django, Docker, and how to do stuff with maps (and don't even get me started on that messy README). Feel free to pick apart anything if you'd like, but it might give you more work than you want. With that said, this PR needs the most help with the data and ETL work.

ETL

I'm trying to download demographic data for Mississippi counties. I'm getting surveys from multiple years so that I can compare the years and make some visualizations with the data. This is all happening in this processing script. (I'm probably doing this all wrong.)

I'm also fetching some geojson. (Note that I'm storing the geojson for a general map of Mississippi in the React app, so that it's served with that code in order to load the home page map the most efficiently -- at least this is how it was working like two months ago). The rest of the geojson will be stored in a database and served via a JSON API.

I have yet to advance to the level of Make where I can do these things (but it makes more sense after reading the doc again today):

• automatically clean up the files (hi gitignore with shapefile extensions)
• orchestrate multiple makefiles (e.g. do the county demographics in one Makefile, the geojson in another Makefile, and then orchestrate those children processes in the project's root Make). You will see a few .mk files that I'm not using right now, but figuring that out has been on my todo list.
• the "load" part -- at what point is it loaded into Django as a fixture? Is this part of the Make process? Or does this process only load it into the backend as "raw" data, which is then up to a dev to load into Django with a management script?

Testing

• You can pull the docker image down. You might have some headaches with the localhost since I'm using React and doing some proxy stuff with Docker (that I don't completely understand and need to fix). You gotta visit localhost:3000 instead of localhost:8000.
• Or you can visit the review app: http://secret-dusk-91150.herokuapp.com. It's a sleepy free heroku app, so it might need to wake up on the first request. (ugh there is a bug in my map rendering so the map won't show up right and i gotta go get my covid shot)

[smcalilly]

@hancush I can't express enough how much I appreciate this code review. I'm confused by what I was trying to do with this pipeline (why am I executing the python script twice?), so it must've been difficult for you to understand, too... I've made some improvements to the Python and now figuring out the Make stuff. I'll re-request a review once it's ready.

[smcalilly]

@hancush This is ready for review. You should be able to do make all and it will create output data, but you will first need to cd etl so you're in the correct directory (changing that is on my to-do). The data is currently going to the backend and frontend directories and this confuses me, so I need to rethink that strategy before I make any big changes (especially the ~/backend/raw dir because it has some data that I'm already using).

Some questions I have:

• What is the preferred way to manage system requirements? A computer needs gdal and the Python census library. The app is dockerized, so I'm guessing I should work Make into a container, but would be curious to hear your thoughts.
• How can I make it so that the unzipped files from county_borders_2020.zip are cleaned up afterwards? Am I supposed to add all the files as intermediate targets? And is there a way to do it so I don't have to hardcode the filenames? I'm currently cleaning up the files with make clean, but ideally this would happen within the Makefile process.
• Does a Django load data management command typically reference the output data created by Make? My questions here might be best to ask via a screen share.

[smcalilly]

@hancush It's getting better. This branch is in a place where you can pull down the code and run docker-compose run --rm backend make all.

Currently, the pipeline is writing to one place, an output directory. It doesn't yet load the data into the application's DB, but that's my next step. This might entail some design changes, so I need to think more about how I want to do this.

I also ran into some docker gotchas. This gave me a chance to think more about how to design docker images/have some ideas on improvement. (For example, the etl stuff lives in the backend directory, because this directory has its own docker image due to this app's docker design. For now, it's easiest to package the etl code into that image.)

In your notes, you said:

Don’t forget to use the phony target for recipes that don’t generate a file

Can you explain where to do this, again? I don't remember the specifics about this note.

And, please let me know if there are any other ways I can improve this pipeline, or weird things about.

[smcalilly]

@hancush I've reached the goal where you can run docker-compose run --rm backend make all. This single command will load the data into the database, using a custom management command (I copied one of the patterns in an ETL example you showed me). Let me know if you have any problems and see areas of improvement.

There are a few issues that came out of this work, so I need to focus on improving those. I especially need to learn more about geojson and postgis, because I'm doing some naive stuff with that. For now, I think this goal has met the objective of "Learn ETL". (At least, I'm familiar with it, but don't really know, you know?) I've identified some ways to improve the data I need, which will require some non-ETL decisions/changes (as well as some ETL changes). So, I think this PR is ready to come in -- this code is a solid basis for some next steps, and merging the code allow me to regroup/refocus.

[smcalilly]

Thanks @hancush! I learned a lot from your review, especially about Make and Django. Got a lot of things in the backlog to improve now.