NYCPlanning / db-developments

🏠 🏘️ 🏗️ Developments Database
https://nycplanning.github.io/db-developments
8 stars 2 forks source link

Integrate geocode into build #599

Open SashaWeinstein opened 2 years ago

SashaWeinstein commented 2 years ago

Medium PR, could use a second reviewer if Max is interested 🌍

Addresses issues #597 and #598

Existing implementation

Currently the geocoding is done by the data sync action that sends the geocoded data to data library. The downside of this implementation is that if the data sync hasn't been done recently, the data can be out of date. It's also hard to test changes.

New implementation

Geocode is run on our local machines from a devcontainer. I think this is better as having a linear pipeline that can be built in from one command is best practice

Changes to HNY

Inner join makes the hny_devdb table more readable

SashaWeinstein commented 2 years ago

Also I save HNY_lookup to a table instead of keeping it as a common table expression as it makes the code easier to debug

td928 commented 2 years ago

also flagging that the mc installation is needed for devcontainer

bash/config.sh: line 113: mc: command not found
SashaWeinstein commented 2 years ago

Ok word, so next steps are mini io installation and then run on github actions and review output on app?

td928 commented 2 years ago

Ok word, so next steps are mini io installation and then run on github actions and review output on app?

see my comments above the github actions seem not to be working for geocoding for some other reason as well. Didn't dig too deep to the cause

SashaWeinstein commented 2 years ago

This is a work in progress as I haven't gotten the docker compose run to work yet. I'm not stuck and it doesn't seem all that hard, I just hard to move to other things. The goal here is for devDB to run like facDB. The next step is to get the docker compose stack to work. I hope that this work gets picked up eventually. I think the enhancement is close to being done and will make future builds faster.