Running the full pipeline on Tallgrass

DOI-USGS / national-flow-observations

This repository pulls national flow data from NWIS

Other

4 stars 8 forks source link

Running the full pipeline on Tallgrass #27

Closed lindsayplatt closed 2 years ago

lindsayplatt commented 2 years ago

This PR includes changes to the slurm scripts and README to be able to build this pipeline on Tallgrass (thanks to @padilla410 and @jesse-ross). We also built the pipeline and included all the build indicators. Thanks to the switch to Tallgrass + change from rds to qs formats for intermediate files (see #21), this build was faster than previous builds! The full build took 12.5 hours instead of the 42 hrs we have seen before (adding values from #4)

lindsayplatt commented 2 years ago

@wdwatkins requested your review for the two slurm scripts + README @limnoliver requested your review to verify the .ind and build/status/ files

limnoliver commented 2 years ago

This PR looks good to me and rebuilds all expected indicators and build files (8 of each for rebuilding sections 1 and 2). Would be nice to see the output from 3_summarize to verify the pull did what we expected it to do. Will that be a separate PR?