Epic - Daskify HQTA

[tiffanychu90]

After receiving a research request, use this template to plan and track your work. Be sure to also add the appropriate project-level label to this issue (eg gtfs-rt, DLA).

Epic Information - Daskify HQTA

Summary

• HQTA workflow currently relies on running several notebooks each month.
• Move existing work from notebook into scripts, and also incorporate dask and dask_geopandas to shorten the run time from several hours to...something way less.
• Refactor the code to make it more modular within each script. Also break apart steps into distinct stages so it shows how we translate an ambiguous and complicated policy definition into data processing stages.
• Retain some notebooks to double check visualizations at various stages
• There's some overlaps with rt_delay work, so move over the utilities functions related there into shared_utils to be rt_utils, etc.
• Research how to move dask computations to the cloud, once it runs locally successfully, to make full use of the partitioning.
• Update: 8/23/22 discussion with Jameelah: we're not ready to spin up a dask cluster yet, since HQTA itself is not a compelling enough use case to justify Jarvus resource cost to spin it up. If the RT work, where more of the complex spatial processing cannot be done in BigQuery could rely on Python to do it, then we should rewrite the existing Python code to use dask. Some RT stuff is done once for all (cutting road network into 1 km segments), but other stuff is done daily. How much of the daily stuff would be actually be the spatial processing, and how much should remain in SQL? Revisit this discussion when we've rewritten some of the spatial processing of vehicle positions using dask and have a better understanding of RT table schemas.
• Update: 9/26/22 Jarvus sprint discussion: need to spin up dask cluster now for testing RT processing of vehicle positions, even if it's a smaller cluster for now

Research required:

• How to use dask, dask_geopandas
• Reconstruct the logic to use the more limited operations in dask_geopandas, and use geopandas if we really can't

Notes, misc:

Reviewers [Stakeholders]

1. @edasmalchi

Issues

• [x] 1. #313 - initial rewrite / grouping into scripts of work present in notebooks. mostly data download, data cleaning, processing.
• [x] 2. #320 - reconceptualize using dask_geopandas.sjoin first to narrow down rows to do dask_geopandas.clip. new step to cut down computing time.
• [x] 3. #332 - step through rest of workflow to create final hqta_points, hqta_areas geoparquets
• [x] 4. #353 - refine download data step to use gtfs_utils. accommodate the fact that different operators might be "valid" each month, and enforce the conditions in which all the merges should be successful (so later steps don't break down)
• [x] 5. #358 - refine, instead of picking longest shape_id by route_length, figure out the symmetric_difference and pick longest shape_id in each direction, combine into 1 row, and then cut that one line into segments.
• [x] 6. #400 - debug error (#398) of points in a forest. run through a full hqta workflow for July and Aug, add logs to better debug
• [x] 7. #402 - catch error where hqta_details is missing for hqta_type==hq_corridor_bus and rerun D2 to create hqta_areas for July and Aug. July file gdb sent for open data portal is first time stepping through entire workflow post-dask-rewrite.
• [x] 8. #411 - Aug HQTA update, add back clipping to CA boundary (but Yuma remains), dissolve hqta polygons by operator-route, add in route_id to both hqta_points and hqta_areas
• [x] 9. #413 - clean up notebooks, archive Eric's original notebooks and group together in the order they're supposed to run. update README to step through workflow
• [x] 10. #461 - notice data download for rail, ferry, brt points is running fresh queries. rely on cached parquets to cut down on similar queries.

Deliverables

• Scripts, maybe even a simple Makefile command to run each month to generate the HQTA data for open data portal
• Clear documentation on data processing stages, where to check results in notebooks, etc

[tiffanychu90]

@hunterowens: CLOSING THIS! @ian-r-rose: Only now as I'm going through to clean up my issues am I noticing you've been keeping up with this! Getting my feet wet with dask, and haven't yet fully explored...!

[ian-r-rose]

@ian-r-rose: Only now as I'm going through to clean up my issues am I noticing you've been keeping up with this! Getting my feet wet with dask, and haven't yet fully explored...!

It's been fun to watch, nice work!

[tiffanychu90]