search

DOI-USGS / lake-temperature-model-prep

Pipeline #1
Other
6 stars 13 forks source link

NHD paths and column names #360

Open SimonTopp opened 1 year ago

SimonTopp commented 1 year ago

Does anyone know when this path was last updated? I'm poking around to re-run mostly identical code in lake-surface-temperature-prep. It looks like the correct S3 paths are now

https://prd-tnm.s3.amazonaws.com/StagedProducts/Hydrography/NHD/State/GDB/NHD_H_%s_State_GDB.zip

rather than

https://prd-tnm.s3.amazonaws.com/StagedProducts/Hydrography/NHD/State/HighResolution/GDB/NHD_H_%s_State_GDB.zip

But, with the top path the waterbodies layer has all lower case columns whereas existing code indicates columns are mixed case. I believe the top is still NHD HR, but the change in column case makes me wonder if they're different. Regardless, there's no */State/HighResolution/* in the S3 directory as of today (12/6/22).

Maybe @lindsayplatt, @padilla410, either of you have an idea of how long ago this was updated? I know NHD is notoriously bad at documenting their versioning. Also wanted to let folks know in case downstream changes with column names need to be made.

SimonTopp commented 1 year ago

Just updating this to confirm that https://prd-tnm.s3.amazonaws.com/StagedProducts/Hydrography/NHD/State/GDB/NHD_H_%s_State_GDB.zip is the correct path. See this PR for other minor changes that might be necessary.

lindsayplatt commented 1 year ago

Looks like the last time we updated 1_crosswalk_fetch/out/canonical_lakes_sf.rds (which is the file created from the linked function, create_nhd_HR_download_plan()) was Nov 2021 (based on the Google Drive "Last Modified" field). Good to know that if we need to rebuild, we will have to do some updating. Given where we are in this project, I am hesitant to do any updating right now before our data release is complete.