update quantiles for water storage GF

[mhines-usgs]

reached out to Steve after logging into Denali, I do see a pkwater_equiv.csv file in the location that David suggested we look, but it's updated since that time, so I'm not sure the files are still there ?

From David:

cd /caldera/projects/usgs/water/wbeep/NHM/gf_v11/gridmet/output/ #moving to directory with the data
#the wildcard 2019-12-31_*.gz matched all the files I wanted (* = anything)
rsync -azvP 2019-12-31_*.gz wwatkins@yeti.cr.usgs.gov:/cxfs/projects/usgs/water/iidd/data-sci/wbeep/ 

rsync -azvP 2019-12-31_*.gz wwatkins@yeti.cr.usgs.gov:/cxfs/projects/usgs/water/iidd/data-sci/wbeep/

On Denali today I see the following when limited to the file names with pk in them:

mhines@denali-login1:/caldera/projects/usgs/water/wbeep/NHM/gf_v11/gridmet/output> ls -alhrt *pk*.csv
-rw-rw-r-- 1 markstro wbeep 9.8G Nov  6 02:29 pkwater_equiv.csv
-rw-rw-r-- 1 markstro wbeep  10G Nov  6 02:29 pk_temp.csv
-rw-rw-r-- 1 markstro wbeep 9.7G Nov  6 02:29 pk_precip.csv
-rw-rw-r-- 1 markstro wbeep 9.8G Nov  6 02:29 pk_ice.csv
-rw-rw-r-- 1 markstro wbeep 9.9G Nov  6 02:29 pk_depth.csv

I messaged him on teams this morning after looking around again. No reply yet.

[lindsayplatt]

I think we want just the updated pkwater_equiv, right? That's the one that has been fixed since we ran this before and there were unrealistic values?

[mhines-usgs]

Does the date matter though from when it was put on Denali? I'm confused because I thought David said that before it was a file named 2019-12-31 or something, these don't have any particular dates in the name, and looks like all the output is from November 6, so I don't even know if this is the correct file?

[mhines-usgs]

here's a full directory listing, I am not sure what I'm looking for at this point, I understand I need pkwater but not sure how to verify if this is still the same file he mentioned was ready on Oct 1. Steve hasn't responded to my teams chat which is frustrating.

mhines@denali-login1:/caldera/projects/usgs/water/wbeep/NHM/gf_v11/gridmet/output> ls -alhrt
total 706G
drwxrwsr-x  4 markstro wbeep  20K Sep 14 11:11 historical
drwxrwsr-x  2 markstro wbeep 4.0K Oct  9 10:20 last_run
-rw-rw-r--  1 markstro wbeep 193M Oct 14 09:11 PRMS_RESTART_2019-12-31.nc.init_66_vars
-rw-rw-r--  1 markstro wbeep  89M Oct 16 08:45 pkwater_equiv_short.csv.gz
-rw-rw-r--  1 markstro wbeep  341 Oct 20 12:10 stream_temp_quantiles.slurm
-rw-rw-r--  1 markstro wbeep 6.1K Oct 20 14:02 stream_temp_quantiles.py
-rw-rw-r--  1 markstro wbeep 5.5M Oct 21 16:13 seg_tave_water_quantiles.txt
-rw-rw-r--  1 markstro wbeep 763K Oct 23 10:29 seg_outflow_really_short.csv.gz
-rw-rw-r--  1 markstro wbeep 254K Oct 23 10:30 seg_tave_water_really_short.csv.gz
-rw-r--r--  1 markstro wbeep 2.6K Oct 24 10:12 power_plant_plotter.py
-rw-r--r--  1 markstro wbeep 2.8K Oct 24 10:12 power_plant_plotter_2015.py
drwxrwsr-x  2 markstro wbeep 4.0K Oct 24 10:41 plots_2015
drwxrwsr-x  2 markstro wbeep 4.0K Oct 24 10:43 plots
-rw-rw-r--  1 markstro wbeep  11M Oct 24 10:51 plots.tar.gz
-rw-rw-r--  1 markstro wbeep 5.0M Oct 24 10:52 plots_2015.tar.gz
drwxrwsr-x  7 markstro wbeep  20K Oct 30 11:07 .
drwxrwsr-x  2 markstro wbeep 4.0K Oct 30 11:21 for_rich
drwxrwsr-x 16 markstro wbeep 4.0K Nov  5 09:54 ..
-rw-rw-r--  1 markstro wbeep 9.7G Nov  6 02:29 unused_potet.csv
-rw-rw-r--  1 markstro wbeep  12G Nov  6 02:29 soil_moist_tot.csv
-rw-rw-r--  1 markstro wbeep 9.7G Nov  6 02:29 hru_ppt.csv
-rw-rw-r--  1 markstro wbeep  13G Nov  6 02:29 tminf.csv
-rw-rw-r--  1 markstro wbeep  13G Nov  6 02:29 tmaxf.csv
-rw-rw-r--  1 markstro wbeep  13G Nov  6 02:29 tavgf.csv
-rw-rw-r--  1 markstro wbeep  15G Nov  6 02:29 swrad.csv
-rw-rw-r--  1 markstro wbeep 9.7G Nov  6 02:29 ssres_flow.csv
-rw-rw-r--  1 markstro wbeep 9.7G Nov  6 02:29 sroff.csv
-rw-rw-r--  1 markstro wbeep 9.7G Nov  6 02:29 slow_flow.csv
-rw-rw-r--  1 markstro wbeep 9.7G Nov  6 02:29 potet.csv
-rw-rw-r--  1 markstro wbeep  12G Nov  6 02:29 hru_streamflow_out.csv
-rw-rw-r--  1 markstro wbeep 9.7G Nov  6 02:29 hru_sroffp.csv
-rw-rw-r--  1 markstro wbeep 9.7G Nov  6 02:29 hru_sroffi.csv
-rw-rw-r--  1 markstro wbeep 9.7G Nov  6 02:29 hru_snow.csv
-rw-rw-r--  1 markstro wbeep 9.7G Nov  6 02:29 hru_rain.csv
-rw-rw-r--  1 markstro wbeep 9.7G Nov  6 02:29 hru_actet.csv
-rw-rw-r--  1 markstro wbeep 9.7G Nov  6 02:29 hortonian_flow.csv
-rw-rw-r--  1 markstro wbeep 9.7G Nov  6 02:29 gwres_flow.csv
-rw-rw-r--  1 markstro wbeep 9.7G Nov  6 02:29 dunnian_flow.csv
-rw-rw-r--  1 markstro wbeep 9.7G Nov  6 02:29 dprst_vol_open_frac.csv
-rw-rw-r--  1 markstro wbeep  12G Nov  6 02:29 dprst_vol_open.csv
-rw-rw-r--  1 markstro wbeep 9.7G Nov  6 02:29 dprst_stor_hru.csv
-rw-rw-r--  1 markstro wbeep 9.7G Nov  6 02:29 dprst_sroff_hru.csv
-rw-rw-r--  1 markstro wbeep 9.7G Nov  6 02:29 dprst_seep_hru.csv
-rw-rw-r--  1 markstro wbeep 9.7G Nov  6 02:29 dprst_insroff_hru.csv
-rw-rw-r--  1 markstro wbeep 9.7G Nov  6 02:29 dprst_evap_hru.csv
-rw-rw-r--  1 markstro wbeep  12G Nov  6 02:29 dprst_area_open.csv
-rw-rw-r--  1 markstro wbeep 9.7G Nov  6 02:29 soil_to_ssr.csv
-rw-rw-r--  1 markstro wbeep 9.7G Nov  6 02:29 soil_to_gw.csv
-rw-rw-r--  1 markstro wbeep  12G Nov  6 02:29 soil_rechr.csv
-rw-rw-r--  1 markstro wbeep 9.7G Nov  6 02:29 snowmelt.csv
-rw-rw-r--  1 markstro wbeep 9.9G Nov  6 02:29 snowcov_area.csv
-rw-rw-r--  1 markstro wbeep  11G Nov  6 02:29 snow_free.csv
-rw-rw-r--  1 markstro wbeep 9.7G Nov  6 02:29 snow_evap.csv
-rw-rw-r--  1 markstro wbeep 9.7G Nov  6 02:29 pref_flow.csv
-rw-rw-r--  1 markstro wbeep 9.8G Nov  6 02:29 pkwater_equiv.csv
-rw-rw-r--  1 markstro wbeep  10G Nov  6 02:29 pk_temp.csv
-rw-rw-r--  1 markstro wbeep 9.7G Nov  6 02:29 pk_precip.csv
-rw-rw-r--  1 markstro wbeep 9.8G Nov  6 02:29 pk_ice.csv
-rw-rw-r--  1 markstro wbeep 9.9G Nov  6 02:29 pk_depth.csv
-rw-rw-r--  1 markstro wbeep 9.7G Nov  6 02:29 perv_actet.csv
-rw-rw-r--  1 markstro wbeep 9.7G Nov  6 02:29 net_snow.csv
-rw-rw-r--  1 markstro wbeep 9.7G Nov  6 02:29 net_rain.csv
-rw-rw-r--  1 markstro wbeep 9.7G Nov  6 02:29 net_ppt.csv
-rw-rw-r--  1 markstro wbeep 9.7G Nov  6 02:29 infil.csv
-rw-rw-r--  1 markstro wbeep  12G Nov  6 02:29 hru_outflow.csv
-rw-rw-r--  1 markstro wbeep 9.7G Nov  6 02:29 hru_lateral_flow.csv
-rw-rw-r--  1 markstro wbeep 9.7G Nov  6 02:29 hru_intcpstor.csv
-rw-rw-r--  1 markstro wbeep 9.7G Nov  6 02:29 hru_intcpevap.csv
-rw-rw-r--  1 markstro wbeep 9.7G Nov  6 02:29 hru_impervstor.csv
-rw-rw-r--  1 markstro wbeep 9.7G Nov  6 02:29 hru_impervevap.csv
-rw-rw-r--  1 markstro wbeep 9.7G Nov  6 02:29 freeh2o.csv
-rw-rw-r--  1 markstro wbeep 9.7G Nov  6 02:29 contrib_fraction.csv
-rw-rw-r--  1 markstro wbeep 9.7G Nov  6 02:29 cap_waterin.csv
-rw-rw-r--  1 markstro wbeep 9.7G Nov  6 02:29 albedo.csv
-rw-rw-r--  1 markstro wbeep 3.3G Nov  6 02:29 transp_on.csv
-rw-rw-r--  1 markstro wbeep  12G Nov  6 02:29 soil_moist.csv
-rw-rw-r--  1 markstro wbeep 9.7G Nov  6 02:29 slow_stor.csv
-rw-rw-r--  1 markstro wbeep 6.5G Nov  6 02:29 seg_width.csv
-rw-rw-r--  1 markstro wbeep 6.1G Nov  6 02:29 seg_tave_water.csv
-rw-rw-r--  1 markstro wbeep 6.4G Nov  6 02:29 seg_tave_upstream.csv
-rw-rw-r--  1 markstro wbeep 4.9G Nov  6 02:29 seg_tave_sroff.csv
-rw-rw-r--  1 markstro wbeep 6.0G Nov  6 02:29 seg_tave_lat.csv
-rw-rw-r--  1 markstro wbeep 6.3G Nov  6 02:29 seg_tave_gw.csv
-rw-rw-r--  1 markstro wbeep 6.3G Nov  6 02:29 seg_tave_air.csv
-rw-rw-r--  1 markstro wbeep 5.4G Nov  6 02:29 seg_shade.csv
-rw-rw-r--  1 markstro wbeep 7.2G Nov  6 02:29 seg_outflow.csv
-rw-rw-r--  1 markstro wbeep 9.7G Nov  6 02:29 recharge.csv
-rw-rw-r--  1 markstro wbeep  11G Nov  6 02:29 prmx.csv
-rw-rw-r--  1 markstro wbeep 9.7G Nov  6 02:29 pref_flow_stor.csv
-rw-rw-r--  1 markstro wbeep 3.3G Nov  6 02:29 newsnow.csv
-rw-rw-r--  1 markstro wbeep 3.7K Nov  6 02:29 model.out
-rw-rw-r--  1 markstro wbeep 3.3G Nov  6 02:29 intcp_on.csv
-rw-rw-r--  1 markstro wbeep  11G Nov  6 02:29 gwres_stor.csv
-rw-rw-r--  1 markstro wbeep  13G Nov  6 02:29 gwres_in.csv
-rw-rw-r--  1 markstro wbeep 193M Nov  6 02:29 PRMS_RESTART_2019-12-31.nc.init
mhines@denali-login1:/caldera/projects/usgs/water/wbeep/NHM/gf_v11/gridmet/output>

[lindsayplatt]

Oh sorry! I missed that subtle, but key point - the file name is not what you would expect. We are looking for a pkwater_equiv.csv that has the historical dataset, so one that is just Nov 6 wouldn't work. It looks like that location may contain data from each day's model run, too. Yeah, seems like hearing from Steve would be best.

[mhines-usgs]

Ok, thanks for your eyes! :) sooner or later, argh. I may have missed getting the file, I didn't realize he would remove it.

[mhines-usgs]

wrote steve an email and he replied right away! yay. he's going to get the data back up again on denali (or parker is) and it should be there by tomorrow.

[jenniferRapp]

I discussed the Pipeline Failures. Do they need to get fixed before we switch to the new GF? - No What Rich MacDonald said was that the viz team should pull from the development pipeline rather than the old pipeline. Once we have our GF quantile updates then we should pull from dev. Essentially they updated the dev pipeline that we are currently using for Temp to work for the updated Water Storage. When we are ready and have everything updated and approved Dev may become Prod when we are ready to flip the switch. Apparently that is what happened last year too? Megan I wonder what Ivan might think about this, but it sounds like they are not planning to have working Dev and Prod areas for the pipeline. Our process could be to work to get all three apps running, then tighten up expectations for what will happen when we go public. Dec 2 meeting. They emphasized that we should make sure Emily is in the know about the plans.

What do you guys remember from last year dev to prod? Will this include any issues with public or internal facing security settings? @mhines-usgs @mwernimont @lindsayplatt

[mhines-usgs]

it seems odd to me that they would break the current production pipeline for some reason. I can try and do a hotfix so we can get that updating again once I figure out where the new data are going. I was operating under the impression that we would coordinate the switch at the same time once everything was ready, not that the current one would just be abandoned for a new version. i don't think ivan has been involved in the project this year at all, so i don't know what would make sense to ask him about? I'm sure he could offer them some best practices in promoting their work from dev to production in the future?

[lindsayplatt]

Last year going from dev to prod was just on our end, so when they said "yes it is approved", we hit build on prod. It is different this go-around because prod is already up and running and we need to switch out the data that it is using. So, it sounds like they stopped or changed the pipeline where we were grabbing data from on prod out for the new GF version. I was under the impression that both would continue to run in parallel until we actually pushed the viz with GF updates out to prod. I guess having it be broken for a bit is OK since no one uses this for decision making. I don't think we have any security issues we need to be concerned about.

[jenniferRapp]

Makes sense to me. We are not actively developing after we put this to ‘production’ so maintaining both dev and prod might just be overkill.

Jennifer L. Rapp, Physical Scientist Chief, Decision Support Branch USGS Water Mission Area Richmond, VA 23228 804-615-6868 (mobile and home office) jrapp@usgs.govmailto:jrapp@usgs.gov

From: Lindsay (Carr) Platt notifications@github.com Sent: Thursday, November 19, 2020 4:38 PM To: usgs-makerspace/makerspace-sandbox makerspace-sandbox@noreply.github.com Cc: Rapp, Jennifer L jrapp@usgs.gov; Author author@noreply.github.com Subject: [EXTERNAL] Re: [usgs-makerspace/makerspace-sandbox] update quantiles for water storage GF (#655)

This email has been received from outside of DOI - Use caution before clicking on links, opening attachments, or responding.

Last year going from dev to prod was just on our end, so when they said "yes it is approved", we hit build on prod. It is different this go-around because prod is already up and running and we need to switch out the data that it is using. So, it sounds like they stopped or changed the pipeline where we were grabbing data from on prod out for the new GF version. I was under the impression that both would continue to run in parallel until we actually pushed the viz with GF updates out to prod. I guess having it be broken for a bit is OK since no one uses this for decision making. I don't think we have any security issues we need to be concerned about.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fusgs-makerspace%2Fmakerspace-sandbox%2Fissues%2F655%23issuecomment-730652127&data=04%7C01%7Cjrapp%40usgs.gov%7C9015a1f62d16429b151908d88cd3813c%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637414187319886778%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=8FvpUgnQPQjss2dxzB5hOQlbn%2FTcXvm5Hr%2B68dYxYzU%3D&reserved=0, or unsubscribehttps://gcc02.safelinks.pr%20otection%20.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAIBB4BIWYYN5ZHJXDIDKWHDSQWF3BANCNFSM4SEUPN5A&data=04%7C01%7Cjrapp%40usgs.gov%7Cb6b13482355e4e16bd6208d88cd35f4b%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637414186753671287%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=I9xNX5Tutyl4paIoCJIRdqZ8z9jOzVhX5I0oZwVa9%2FE%3D&reserved=0.