water temp: explore adding locations layer to map

[mhines-usgs]

in response to people wanting to get more details about actual temperatures, explore how complicated it would be to add station locations in a new layer that links to the actual water temperature for stations either through an api request to some external station data we can leverage, or even populate a pop up with 7 day water temp graph from the WDFN graph api?

Perhaps show Calibration gages? Steve Markstrom may know.

• [x] talk to Steve Markstrom about what might be appropriate for attempting to meet this need [Jen]
• [x] prototype what this might look like on the map (dots on top of streams?)
• [x] prototype what the data pop up might look like (either a value or an image from the graph api?)

[jenniferRapp]

QW2019_national_GIS.zip I pulled these by HUC 02 with a python script to loop through the waterservices URL generation tool and the It uses the USGS Egistoolbox to make the ULR into a gdb layer. It should give us a perspective for all places where continuous temperature is collected. It was early 2019, but we could re-run as needed.

[jenniferRapp]

Using https://github.com/usgs/wdfn-graph-server it would: Returns a png containing a hydrograph for parameterCode(xxxxxx) for the NWIS site(siteid) for the last seven days. P00011 or P00010 would give us temperature.

[jenniferRapp]

[jenniferRapp]

I spoke with modelers and Pat tonight. They like the idea of adding the temp locations on the map. Steve Markstrom suggested plotting Mean Daily Value from Water Watch (or NWIS) with symbology to match the modeled values for QW points. Would provide some validation verification.. Get a sense for uncertainty. Essentially grabbing the previous daily value and symbolizing it. No popups etc. That sounds more like what Marty was suggesting.

[jenniferRapp]

I think trying out what this looks like and whether we can color based on the previous day's temperature would be a nice addition if you have some time to work on it after the Q&A. @mwernimont

[mhines-usgs]

I converted the shapefile of stations with temperature sensors locations to geojson and uploaded it to http://maptiles-prod-website.s3-us-west-2.amazonaws.com/geojson/QW_Temp_2019.json to play with

just plotting those locally, here is what it looks like with some questionable style choices, for example:

going to see if i can get a dot click to open up a graph api image response, like https://waterdata.usgs.gov/nwisweb/graph?agency_cd=USGS&site_no=09380000&parm_cd=00060&period=7

[mhines-usgs]

thanks to some Marty help ™️ this morning, we got this up on test. the style isn't really well thought out at the moment (dot color and pop up itself), but this was the 'low cost' idea i was babbling about monday.

we might want to decide if this is useful to bring some additional detail to the viz, if we want all the station points or maybe we filter the layer first by those that 1) only intersect the GF segment layer and/or 2) select specific station representatives where we have large clumps. the user can't really discern when they are so heavily clumped. or we could allow more zooming in too, to help with that issue.

one thing too Marty pointed out was that we sometimes don't actually get an accurate response for the last 7 day request, that's just because the server doesn't have the data to share, you might see that if you click around long enough

http://wbeep-test-website.s3-website-us-west-2.amazonaws.com/estimated-availability/#/water-temperature

working on getting it out to beta too to see how mobile looks 😬

[jenniferRapp]

@mhines-usgs Casey Lee pointed me to a national app where I downloaded the Temp sites they say are continuous for 2018 -2020. http://kswsc.cr.usgs.gov:3838/cjlee/qw_inventory/ Continuous_2018-2020.txt

We can take a look and compare the sites that we have with what they have. I agree that many of the sites in the south in my data layer didn't have values for the last week.

[jenniferRapp]

A second option would be to highlight the water quality monitoring Network of 110 sites across the Nation. https://nrtwq.usgs.gov/nwqn/#/

[jenniferRapp]

Third option which I'm leaning toward is a download from QW watch. I extracted only those that are Site Type Stream.
currentQW.xlsx

[mhines-usgs]

Ok, Working now on exchanging the GeoJSON for this last suggested one instead so we can see how that looks.

[jenniferRapp]

Thanks Megan. I think it should be Alright. The sites in that list should only be streams, no canals, ditches, or groundwater sites. Since they are reported in QW Watch real-time page they should be live. Even the waterwatch site has disclaimers for missing data: "Site operated on a seasonal basis or currently is not operating. No values are available for the last 6 hours." Which we could adapt for our app to say "Some sites are operated on a seasonal basis or currently is not operating. No values are available for the previous week. The data used to produce these graphs are provisional.".

[mhines-usgs]

sounds good. I loaded the new data in and updated the test tier now with that data. I need to take a closer look tomorrow and make sure the data are correct, initially when I converted from xlsx->csv, the site_no field converted to numeric and dropped the leading zeroes, hopefully I got them all back where needed and did not add any where I didn't need, but will take another gander tomorrow and redo the conversion more programmatically to confirm. I will also look at a few ideas for display options then too and make sure the layer shows up for toggling in the layer list.

in regards to the text addition, I would guess that text should be present on the first ? page for 'what does this map show me' -- and maybe we could also include a short mention in the About this map text. Thinking something like

It shows the latest available daily estimates of natural water temperature for approximately 58,000 stream reaches across the lower forty-eight states. The point markers visible upon zooming in depict actual sensor locations and can be clicked to gain further details along side the modeled temperature estimates.

or something like that! suggestions welcome!

--suggestion: It shows the latest available daily estimates of natural water temperature for approximately 58,000 stream reaches across the lower forty-eight states. The real-time temperature monitoring sites are visible at finer zoom levels, and clicking will open a graph of the past 7 days of temperature monitoring data along-side the modeled temperature estimates.

[jenniferRapp]

I think these look good. I found a few that don't have data pulls for 00010 in Montana. I think we should increase the zoom level 1-2 times more to allow people to see what is going on in the dense city areas like Washington DC or Philly, PA. We can do it and then demo it for the modeling group tomorrow. Def need to try on Mobile.

[jenniferRapp]

I'm getting errors like this: I checked this site and it has continuous data but it didn't have any since mid-day on Dec 1. but Nov 24 - 30 it had data....

I got an error for 11312685 that "site/Parameter-code combination not found. But there are two forms of temperature there, YSI and ADCP.

073816503 says graph request failed. But there are data in NWIS. Still have a lot of graph request fails near New Orleans.

02098206 in SC combination not found, but there def are data.

We can ask Jim about the functionality of this tool. Maybe certain kinds of water temp are harder to pick up.. ADCP etc. We could switch to the 110 site list of the NQWM network points if we wanted to keep it simpler...

[mhines-usgs]

Yeah, I saw that too and that's why I want to revisit what I did and make sure that the site numbers are actually correct, too.

[jenniferRapp]

The ones I checked in NWIS were all correct station numbers. I could not see a reason except for one site that didn't have data for yesterday. The rest all seemed to have data.

[jenniferRapp]

I have one more option for us. Jordan pointed me to the next-gen graphs. How hard would it be to switch to these graph renderings? https://labs.waterdata.usgs.gov/about-graph-image-api/index.html @lindsayplatt @mhines-usgs @mwernimont

[mwernimont]

I'll give it a look

[mwernimont]

So its possible

A couple caveats I am seeing though: When a site does not meet the parameters instead of just sending a quick image saying the data does not exist it waits for the timeout to cut off the call. One call lasted for 3 minutes, which is forever in user time just to get a broken img tag. Also even when it does come back with an image it does take a bit of waiting.

The other is there doesn't seem to be a combined temp C and F parameter code for water temp? https://help.waterdata.usgs.gov/parameter_cd?group_cd=%

So would we just show C or F? or am I missing the code to do both in my searches?

[mwernimont]

Nevermind, Megan pointed out the F is on the right. My eyes are horrible apparently

[mwernimont]

I am also just getting lots and lots of failed calls even on sites that had worked previously. Even on their own api call testing area it failed.

So not sure if we want to put this in or bring someone in to help figure out why so many fails would be happening?

[jenniferRapp]

The old figure is working well, then I don't think we should try something new, esp if it's not working well.
Thanks for trying it.

[jenniferRapp]

Well, are there more errors in the new service? We certainly could ask. Still, if the old one works, let's leave it.

[mwernimont]

All I can really say for certainty is I adjusted our call parameters to the old service to call to the new one and was get timeout errors on most every call. Maybe the way we call the old service isn't the best for the new service etc etc. So it is nice to have currently working option, but maybe in the future if we want the new service for sure in the app we bring in outside help to make sure we are doing everything perfectly.

[jenniferRapp]

let's keep the old graphs.

I like them.

[jenniferRapp]

@mwernimont @mhines-usgs @lindsayplatt From Joe Nielsen's review: • in the application itself, I am getting some oddities when I look at the monitoring sites. Many come up with errors, and most of the pop-up boxes are so large as to run off the screen. See these screen shots:

I (Jen) have experienced this too. Can we make them smaller? Can we center the graph in the window that is open?

Alternatively, would we want to try the new graphs again? Emily would much rather we use those since they are in development now and the old ones are deprecated. I don't know if the time-our errors have been fixed?

[mwernimont]

Hi Jen,

The sizing is adjustable, but the pop up is designated to be where the mouse click is as that is where the site is. If you want the pop-up center you could make the site center of the screen and it should work, but I don't really like the idea of having the pop up not right where the site is as it creates detachment in my mind.

For the new sites of course, we can use that service, but as Mary said they don't have a current way of dealing with a request that the site can't meet, so it will sit there until the timeout window is hit and then fail. So, I doubt they have fixed that as they are busy with all the current development going on and this seemed like an outside request of their current focus. So, it's up to you which you want to use. The timeouts and timing of request returns are out of our hands so will just head that warning one more time.

As for the error images with the old sites, I know that isn't ideal, but it is better than a long timeout wait. Did we ever go through the data and make sure all the sites have temperature data? Do we have a better time range we think these sites would have more consistently than 7 days? I can adjust all that easily, but am not the one to figure out what is the best window or what is the most appealing scientifically comparing a 7-day window to say a 3?

So just let me know what you decide for the service. It isn't a hard switch.

Thanks,

Marty

---

From: JenniferRapp notifications@github.com Sent: Tuesday, December 15, 2020 8:06 AM To: usgs-makerspace/makerspace-sandbox makerspace-sandbox@noreply.github.com Cc: Wernimont, Martin (Contractor) R mwernimont@contractor.usgs.gov; Mention mention@noreply.github.com Subject: [EXTERNAL] Re: [usgs-makerspace/makerspace-sandbox] water temp: explore adding locations layer to map (#684)

This email has been received from outside of DOI - Use caution before clicking on links, opening attachments, or responding.

@mwernimonthttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmwernimont&data=04%7C01%7Cmwernimont%40contractor.usgs.gov%7C3e8af34677e848b4eb2808d8a102aa41%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637436380110790348%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=c0%2Bb00nsFCUQFQiYYVH9HQOU1up4fru7kbcEEFRshR4%3D&reserved=0 @mhines-usgshttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmhines-usgs&data=04%7C01%7Cmwernimont%40contractor.usgs.gov%7C3e8af34677e848b4eb2808d8a102aa41%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637436380110800304%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=5n0ecP8QY8XTledRBnlIrgpl031EVmLt6gxSUqGqJUQ%3D&reserved=0 @lindsayplatthttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Flindsayplatt&data=04%7C01%7Cmwernimont%40contractor.usgs.gov%7C3e8af34677e848b4eb2808d8a102aa41%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637436380110800304%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=jG2rxNWz1YWh8e55A8x%2FjUO9IBvuzZUnu%2FjARpzJdTE%3D&reserved=0 From Joe Nielsen's review: • in the application itself, I am getting some oddities when I look at the monitoring sites. Many come up with errors, and most of the pop-up boxes are so large as to run off the screen. See these screen shots:

[error2]https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fuser-images.githubusercontent.com%2F33693189%2F102224908-89b04c80-3eb4-11eb-98c1-e6d3f0aee9d5.png&data=04%7C01%7Cmwernimont%40contractor.usgs.gov%7C3e8af34677e848b4eb2808d8a102aa41%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637436380110810262%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=YVAa80KoiOH6WKmKcpQJ7dJ2yupDCrIVpDJlT3zAZQ8%3D&reserved=0 [error]https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fuser-images.githubusercontent.com%2F33693189%2F102224915-8d43d380-3eb4-11eb-9e88-fa63b0a213d9.png&data=04%7C01%7Cmwernimont%40contractor.usgs.gov%7C3e8af34677e848b4eb2808d8a102aa41%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637436380110810262%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=xURNL3imGgDYIUw2RUlNZwUqHGUk6A8IOvaS%2BAZiWoM%3D&reserved=0

I (Jen) have experienced this too. Can we make them smaller? Can we center the graph in the window that is open?

Alternatively, would we want to try the new graphs again? Emily would much rather we use those since they are in development now and the old ones are deprecated. I don't know if the time-our errors have been fixed?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fusgs-makerspace%2Fmakerspace-sandbox%2Fissues%2F684%23issuecomment-745306648&data=04%7C01%7Cmwernimont%40contractor.usgs.gov%7C3e8af34677e848b4eb2808d8a102aa41%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637436380110810262%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=tn2Dg4LiKgghMr%2BDg7paUHa3mNgu4%2BjWuFj9%2BkUj9wQ%3D&reserved=0, or unsubscribehttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FABQYCUDU7Z6ORE2HIKXZ4ODSU5UOJANCNFSM4TRB6UIQ&data=04%7C01%7Cmwernimont%40contractor.usgs.gov%7C3e8af34677e848b4eb2808d8a102aa41%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637436380110820220%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=Bxd8jEfwqh9TAU%2BmCgYrZh44368vRw3K2S3UxqcaP7k%3D&reserved=0.

[jenniferRapp]

@mwernimont Thanks Marty. Let's keep it with the original graphs. I agree, we want the graph pointing at the site and mouse click. Could the map data move, shift or pan so that the graph is centered? I think we could shift to 4 days to make the graphs smaller. Looks like they do 4 full days, plus today in the graphs as they are now. They actually plot 8 days. Hopefully we will have model predictions for some time in the past 4 days.

[jenniferRapp]

@mwernimont I need to think about the best way to tes

t all the sites to see if they have data. These are water watch sites, so we should be able to create a request to their service and extract the values for a certain day or range of days. Clicking to see which sites have data is not a good option. Many sites will have ice, but others I couldn't tell why they were not reporting.

@lindsayplatt I have a script that works with waterwatch streamflow percentiles. I don't know what it would take to switch to qw conditions. Just to be able to pull the values that waterwatch in QWwatch is reporting. Then we would have a list of sites we could test or potentially remove from our application.

Is there an easier way to grab stream temp from our full list of sites to see which ones don't have data right now?

[mwernimont]

@jenniferRapp I'll look to see if there is a way to get the popup to avoid being cutoff. Outside of the sizing adjustment.

[jenniferRapp]

I looked at the QW Watch page and can pull up a list of all sites and what data they are displaying right now. There are a few more remarks than I saw when I created the list of stations to use. I'll send an updated list if I find any that are discontinued, or only seasonally monitored.

Do we know if there must be degress C monitored for the graphs to plot?

[mwernimont]

That is unknown to me.

[jenniferRapp]

There are 1895 sites now. I learned that some a discontinued, some are only seasonal, and while it would be nice to include them, it will give us problems most of the year. Also numerous were internal only. Please use this updated list as the set of stations for the monitoring graphs. Let me know if you rather have a list of those that I deleted.
CurrentQW_Dec15_2020.xlsx

[mwernimont]

@jenniferRapp a list of deleted would be nice!

[jenniferRapp]

QWwatch_toDELETE.xlsx here you go

[jenniferRapp]

I'll be in meetings most of the day now. Feel free to teams chat me if necessary

[mwernimont]

I'll retest the new service when the sites are updated.

[mhines-usgs]

the stations are updated in the s3 geojson file!

[mwernimont]

Map panning when adjusting to the popups is up on test. There are easing functions to play with if anyone thinks its too abrupt of a pan. Thought it was alright as is, but let me know! This lets us keep larger graphs without them being cut off during the dynamic content switch that happens when we receive our graph request image.

[mhines-usgs]

fyi we still see those 'site/param-code combination not found' errors even with this updated list. (CurrentQW_Dec15_2020.xlsx)

here's an example where we still see a 400 response instead of 200 response request: https://waterdata.usgs.gov/nwisweb/graph?agency_cd=USGS&site_no=05457000&parm_cd=00010&period=7 response image:

yet if we look at this page https://waterdata.usgs.gov/nwis/uv?site_no=05400760, the data we seek show up very easily, but it is using a completely different service altogether, not the one we are using or the new one we were suggested to use.

we can confirm the correct pcode is also being used based on the metadata there, but not sure if there is a way to know from the waterdata service which timeseries id it is? or maybe there is an assumption made on the service endpoint we are using that is just not obvious to us. I tried fiddling around with the request and it just seems fundamentally 'missing'. Maybe somewhere along the line of queries, some stations are just missing and the chain just fails.

[jenniferRapp]

The panning is a little abrupt, but keeps them centered. Joe was having difficulty getting the map display to move at all when the graph was half all the page. The graph quality looks fuzzy now???

I'll finalize the text and let you know for the About and Q&A..

@mhines-usgs I think we should share the error code issues we are having with the graph maintainers to get their take...

[mwernimont]

Idk my eyes are sorta bad, but it doesn't look fuzzy to me?

[jenniferRapp]

Yours looks good. Mine I can't read the names well.
might depend on the monitor?

[mwernimont]

@jenniferRapp Megan and I did our best to team work a less fuzzy graph, we think we hit a compromise? I am still looking into a possible more constant solution, but what we have now is on test.

[jenniferRapp]

Ok. I think the graphs look less fuzzy. Thank you. The panning is good too. Any luck with figuring out the reason for so many errors IF there is data?

[jenniferRapp]

@mhines-usgs @mwernimont @lindsayplatt I think we can move to Production. What implications would that have IF we are still tweaking the station list for the Graphs some? Otherwise I think we are good to go.

[mhines-usgs]

as long as we replace the json file in s3 with a file by the same name, we can update that anytime even after the prod release.

[mhines-usgs]

the errors are not anything we are able to fix, unfortunately. it should be something that team can look into and addresses when they decide it is a priority.

[jenniferRapp]

ok