Error in strsplit(df$site_name, "\\(") : non-character argument

[dfosterhill]

Hi. I have used snotelr for a few years. It worked great for me last week. This morning, all of my scripts are now throwing errors. When I do something as simple as:

library(snotelr) meta_data <- snotel_info() head(meta_data)

I get the error:

Error in strsplit(df$site_name, "\(") : non-character argument

I would appreciate any information about how to resolve this. I didn't change anything from last week to this week.

[sageebel]

Hi! I am having the same issue. Any tips would be appreciated!

[dfosterhill]

Sage, I have tested all my SNOTEL pulls with Python, and they are all fine....so it is not something with USDA changing URLs, etc. I'll be curious to see what the developer here has to say re: the R package issues.

[AndyBunn]

I think it's busting here:

rvest::html_table(rvest::html_nodes(rvest::read_html(httr::GET(url = url, 
                                                               query = query)), "h5~ table+ table"))

The url parses OK. I think it's the The "h5~ table+ table" is returning an empty list. Perhaps something changed in the underlying css? This might do the trick. But I'm terrible at scraping data.

rvest::html_table(rvest::html_nodes(rvest::read_html(httr::GET(url = url, 
                                                               query = query)), "table"))

[dfosterhill]

@AndyBunn you are correct!! I simply took the package source files, made that simple modification, and that fixed it. @sageebel I hope this helps you out, as well.

[AndyBunn]

This is the first, and doubtlessly the last, time I've been useful in a scraping problem.

[dfosterhill]

Super appreciated...We wrote a R shiny app (at snowwaterstorage.org) to let people explore snotel data in a particular way (related to a recent paper). I announced it to the world, and the app promptly broke. Embarrassing, but all fixed now.

[dfosterhill]

Now my bigger issue is figuring out how to take the edited source code and get snotelr reinstalled as a correctly working package. For the time being, I just took the function and put it directly into my code rather than using a package at all...

[andypicke]

Now my bigger issue is figuring out how to take the edited source code and get snotelr reinstalled as a correctly working package. For the time being, I just took the function and put it directly into my code rather than using a package at all...

@dfosterhill @AndyBunn I've been having the same issue. Which function needed to be modified? I think the best solution would be to fork the repo and create a pull request with the modified code that addresses the issue.

[dfosterhill]

The original code in snotel_info.r (lines 33-40) is this:

query the data table

df <- httr::GET( url = url, query = query) |> rvest::read_html() |> rvest::html_nodes('h5~ table+ table') |> rvest::html_table() |> data.frame()

And, what you need to do is change it to this:

query the data table

df <- httr::GET( url = url, query = query) |> rvest::read_html() |> rvest::html_nodes('table') |> rvest::html_table() |> data.frame()

That is the ONLY thing that has to be changed. I will try to get around to forking it and putting in the pull request, unless one of your wants to beat me to it. :)

[andypicke]

Thanks @dfosterhill , I will attempt to create a pull request for it.

Nevermind, looks like you already did it. Thanks.

[dfosterhill]

I just tried to do so. I’m a bit rusty with GitHub, so may have screwed it up.

Dave.

On May 22, 2024, at 10:07 AM, Andy @.**@.>> wrote:

[This email originated from outside of OSU. Use caution with links and attachments.]

Thanks @dfosterhillhttps://github.com/dfosterhill , I will attempt to create a pull request for it.

— Reply to this email directly, view it on GitHubhttps://github.com/bluegreen-labs/snotelr/issues/27#issuecomment-2125347814, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AEZ5RTXJH66HNGVHGSVNGETZDTGGBAVCNFSM6AAAAABIABWVSOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMRVGM2DOOBRGQ. You are receiving this because you were mentioned.Message ID: @.***>

[khufkens]

Hi @dfosterhill, @AndyBunn, @sageebel,

Thanks for picking this up, and finding a fix among you all. I've had a few busy days and little time to respond.

Also, I did not know there was an active community using the package. Hope it serves you all well.

I'll tend to the pull request and issue a new release.

Cheers, K

[khufkens]

Fixed with #28, thanks

[khufkens]

New release is on its way to CRAN #29

[dfosterhill]

awesome, thank you! And thanks to others here who chimed in with all the helpful information. Great package…My research group mostly uses python tools to pull snotel data from CUAHSI directly…but the R stuff was necessary for some shiny app developments.

On May 22, 2024, at 10:50 PM, Koen Hufkens @.**@.>> wrote:

[This email originated from outside of OSU. Use caution with links and attachments.]

Hi @dfosterhillhttps://github.com/dfosterhill, @AndyBunnhttps://github.com/AndyBunn, @sageebelhttps://github.com/sageebel,

Thanks for picking this up, and finding a fix among you all. I've had a few busy days and little time to respond.

Also, I did not know there was an active community using the package. Hope it serves you all well.

I'll tend to the pull request and issue a new release.

Cheers, K

— Reply to this email directly, view it on GitHubhttps://github.com/bluegreen-labs/snotelr/issues/27#issuecomment-2126286683, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AEZ5RTXMYQ6KDQAW7HZIAQTZDV7SXAVCNFSM6AAAAABIABWVSOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMRWGI4DMNRYGM. You are receiving this because you were mentioned.Message ID: @.***>

[jpadilla-spu]

Hi @dfosterhill, @AndyBunn, @sageebel,

Thanks for picking this up, and finding a fix among you all. I've had a few busy days and little time to respond.

Also, I did not know there was an active community using the package. Hope it serves you all well.

I'll tend to the pull request and issue a new release.

Cheers, K

There are dozens of us!

[khufkens]

Hi @jpadilla-spu, would you be able to share some use case?

It is hard for me to assess the reach of some of my packages in real world terms (download numbers are not saying much). Having some examples would go a long way of gathering some financial support to continue OSS development and maintenance.

[dfosterhill]

I have used it to build several R shiny apps related to my publications on snow. One example is at snowwaterstorage.orghttp://snowwaterstorage.org/.

Your package is great!

On May 30, 2024, at 4:15 AM, Koen Hufkens @.**@.>> wrote:

[This email originated from outside of OSU. Use caution with links and attachments.]

Hi @jpadilla-spuhttps://github.com/jpadilla-spu, would you be able to share some use case?

It is hard for me to assess the reach of some of my packages in real world terms (download numbers are not saying much). Having some examples would go a long way of gathering some financial support to continue OSS development and maintenance.

— Reply to this email directly, view it on GitHubhttps://github.com/bluegreen-labs/snotelr/issues/27#issuecomment-2139328537, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AEZ5RTXUJZIWW6NR3KPXLVLZE4C3PAVCNFSM6AAAAABIABWVSOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMZZGMZDQNJTG4. You are receiving this because you were mentioned.Message ID: @.***>

[jpadilla-spu]

Oh sure! In fact, I have two:

1. Seattle Public Utilities

I am currently on the real-time reservoir management team for the City of Seattle. We have two mountain reservoirs that provide drinking water for the city and wholesale customers. I use your package in two ways: (1) to support my operational modeling work, and (2) to support regular internal reporting on the water system.

Operational modeling: we use a watershed model to make short-term (10-days) and medium-term (~1-2 months) decisions about reservoir operations. As part of that work, I have been building tools that will help me review scenarios faster (in a private repo). Your package allows me to quickly grab the snotel data to compare against my model which is a key comparison that helps us get the instream flows right in the winter when we build the snowpack and the spring when we melt it.

Right now I am working on the figures, but I'll eventually build this out into an automated thing using your package and officer. They're a little rough right now because they're really just meant for me at the moment (snotel = blue, model = purple):

Regular internal reporting: We use a mix of external data (USGS/SNOTEL) and internal data (SCADA system) to create a weekly synopsis report that gets distributed around the utility - a snow pack summary is one of the 4 big plots we make. Access to our SCADA system is a bit clunky (we have to use an excel add-in), so being able to directly access the external data from the source saves a ton of time. Having access to snotelr lets me quickly nab and wrangle the data for plotting.

Having packages like snotelr and dataRetrieval has probably helped me cut the time needed to generate these reports in half. It has also cut down on the number of mistakes since I have been able to cut out most of the manual work in the spreadsheet (We have to stick with excel for a variety of reasons). There are a few other nice things: (1) my colleague is also able to run this workflow, and (2) now we get to spend more time thinking about the content of the report rather than triple checking that we got the data wrangling right.

2. US Geological Survey

My former colleagues at the USGS VizLab have used this package in at least two data visualizations: (1) From Snow to Flow and (2) I'm pretty sure that @aaarcher-usgs has also used snotelr in a few visualizations, but I haven't been able to quickly dig them up.

[AndyBunn]

I use it in teaching mostly. Time series and spatial stats for grad students. —- This device necessitates brevity

On May 30, 2024, at 8:18 AM, Julie Padilla @.***> wrote:



Oh sure! In fact, I have two:

1. Seattle Public Utilities

I am currently on the real-time reservoir management team for the City of Seattle. We have two mountain reservoirs that provide drinking water for the city and wholesale customers. I use your package in two ways: (1) to support my operational modeling work, and (2) to support regular internal reporting on the water system.

Operational modeling: we use a watershed model to make short-term (10-days) and medium-term (~1-2 months) decisions about reservoir operations. As part of that work, I have been building tools that will help me review scenarios faster (in a private repo). Your package allows me to quickly grab the snotel data to compare against my model which is a key comparison that helps us get the instream flows right in the winter when we build the snowpack and the spring when we melt it.

Right now I am working on the figures, but I'll eventually build this out into an automated thing using your package and officerhttps://github.com/davidgohel/officer. They're a little rough right now because they're really just meant for me at the moment (snotel = blue, model = purple):

image.png (view on web)https://github.com/bluegreen-labs/snotelr/assets/153234500/cf63ecd4-50be-4949-a8a5-397c168454db

Regular internal reporting: We use a mix of external data (USGS/SNOTEL) and internal data (SCADA system) to create a weekly synopsis report that gets distributed around the utility - a snow pack summary is one of the 4 big plots we make. Access to our SCADA system is a bit clunky (we have to use an excel add-in), so being able to directly access the external data from the source saves a ton of time. Having access to snotelr lets me quickly nab and wrangle the data for plotting.

Having packages like snotelr and dataRetrieval has probably helped me cut the time needed to generate these reports in half. It has also cut down on the number of mistakes since I have been able to cut out most of the manual work in the spreadsheet (We have to stick with excel for a variety of reasons). There are a few other nice things: (1) my colleague is also able to run this workflow, and (2) now we get to spend more time thinking about the content of the report rather than triple checking that we got the data wrangling right.

2. US Geological Survey

My former colleagues at the USGS VizLabhttps://labs.waterdata.usgs.gov/visualizations/index.html#/ have used this package in at least two data visualizations: (1) From Snow to Flowhttps://labs.waterdata.usgs.gov/visualizations/snow-to-flow/index.html#/ and (2) I'm pretty sure that @aaarcher-usgshttps://github.com/aaarcher-usgs has also used snotelr in a few visualizations, but I haven't been able to quickly dig them up.

— Reply to this email directly, view it on GitHubhttps://github.com/bluegreen-labs/snotelr/issues/27#issuecomment-2139906256, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AC7UCXKCHAIVZARZ34FADQ3ZE47LDAVCNFSM6AAAAABIABWVSOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMZZHEYDMMRVGY. You are receiving this because you were mentioned.Message ID: @.***>

[andypicke]

I use snotelr in a Shiny app I made to track streamflow and snowpack in near where I live in Golden, CO : https://andypicke.shinyapps.io/GoldenStreamGauge/

[khufkens]

Hi everyone, thanks for sharing these examples. Good to see this is getting used in such diverse settings, including operational components. Shameless plug, but if you ever need an extra hand on anything I'm up for hire as well.

:smile: :point_down:

https://bluegreenlabs.org/