SatcherInstitute / health-equity-tracker

Health Equity Tracker is a free-to-use data visualization platform that is enabling new insights into the impact of COVID-19 and other social and political determinants of health on historically underrepresented groups in the United States.
https://healthequitytracker.org/
MIT License
17 stars 24 forks source link

Consider adding an Auth flow to load directly from restricted sites instead of manually via `/data` #1565

Open benhammondmusic opened 2 years ago

benhammondmusic commented 2 years ago

Is your feature request related to a problem? Please describe. Currently for Women in Legislative Office, we need to utilize our /data folder to manually download the data sets (line item by race from CAWP and Congress totals from PROPUBLICA) and run our python scrips against those.

Describe the solution you'd like Since politicians change somewhat frequently and a single change in # of women legislators can have a profound impact on some of our numbers, if would be nice if this flow could be performed automatically rather than requiring manual download / upload.

Describe alternatives you've considered The alternative is to continue as we are currently, and performing manual updates periodically

Additional context By "automatic", we would still need to click the DAG flow button; right now nothing is truly automatic. That is perhaps another issue we could consider

benhammondmusic commented 2 years ago
# this URL could be used for an API endpoint but needs authentication
# CAWP_LINE_ITEMS_URL = ("https://cawpdata.rutgers.edu/women-elected-officials/"
#    "race-ethnicity/export-roles/csv?current=1&yearend_filter=All"
#    "&level%5B0%5D=Federal%20Congress&level%5B1%5D=State%20Legislative"
#    "&level%5B2%5D=Territorial/DC%20Legislative&items_per_page=50"
#    "&page&_format=csv")

# Table for STATE_COL_LINE LEG. TOTALS
# WEBSITE FOR TOTALS https://cawp.rutgers.edu/facts/levels-office/state-legislature/women-state-legislatures-2022#table