search

NYCPlanning / db-equitable-development-tool

Data Repo for the equitable development tool (EDDT)
MIT License
0 stars 0 forks source link

Build/Github Workflow Improvements #321

Closed fvankrieken closed 1 year ago

fvankrieken commented 1 year ago

Making these changes in separate branch for clarity while reviewing. Sequel to #320, will merge this back into my other feature branch before merging back to main

fvankrieken commented 1 year ago

@damonmcc fyi - wanted to get some more testing integrated into git, and also wanted to clean up the process of actually building and exporting, as it's a bunch of manual steps right now to get everything fully exported

fvankrieken commented 1 year ago

So would welcome any thoughts, things you'd like implemented as well since you just did this build process

fvankrieken commented 1 year ago

I think this is pretty set other than updating the readme. Other than getting tests into github workflows that get run automatically, changes workflows around building and exporting as well as the entrypoint scripts for this

  1. added eddt.sh script as way of accessing functionality. This can be run with
    • eddt.sh dataloading to load data
    • eddt.sh build to build. Optional arguments after this are census or category, which can optionally be followed by more specific args for the python files. So valid examples include
      • eddt.sh build census builds all years, econ and dem, all geographies
      • eddt.sh build census 2000 builds for year 2000, econ and dem, all geographies
      • eddt.sh build census all demographics citywide
      • eddt.sh build category all
      • eddt.sh build category housing_production puma
    • eddt.sh export, can optionally specify category (no differentiation between census and category stuff here - so demographics, economics, housing_production, housing_quality, quality_of_life are the options here, or no arg provided)
      1. changes the workflow yml files for build/export. Right now, there's an on: push for one of them because it has no on: workflow_dispatch on main so there's no way to trigger it from any branch as far as I can tell. Left as two actions for now. For census, broke into by year just for some granularity with being able to see if a specific one fails. For the other, there's a dropdown to select category but no further granularity (for geography, just does all of them). These functions never had ability to select year, and default to latest. This could be changed as well.

I'll add above info to readme, but would be happy for review and feedback in the meantime

damonmcc commented 1 year ago

@fvankrieken sounds good. when it's done, may be nice to merge this to main rather than only the PUMS branch

I have a commit I'll push just to clean up the on: sections of the test actions

fvankrieken commented 1 year ago

@damonmcc gonna merge this into my other feature branch if that sounds good