search

bluegreen-labs / snotelr

a snow data network (SNOTEL) R package
https://bluegreen-labs.github.io/snotelr/
GNU Affero General Public License v3.0
15 stars 10 forks source link

alternative cache of site metadata for computers without port access #17

Closed potterzot closed 2 years ago

potterzot commented 2 years ago

Some users (e.g. myself) don't have sufficient privileges to open a port with RSelenium as done here. Would there be interest in an alternative caching method that doesn't require the memoise and wdman packages but rather saves the site metadata as a tempfile? I'm not familiar with memoise but an alternative that doesn't require opening a port would be helpful.

khufkens commented 2 years ago

Historically the base table would not load because it was rendered in java(script) after the page was send. In short, when using e.g. httr roughly the default for doing things download related these days, it would not work as the page would be flagged as loaded even before all content was there. Hence, the wdman setup which emulates browser usage, and which waits for all components to load.

A quick test shows that either httr has been updated to account for this, or the snotel page has changed (I guess the former). In any case, I'll implement this later tonight.

# test transfer
library(tidyverse)
library(httr)
library(rvest)

df <- httr::GET("https://wcc.sc.egov.usda.gov/nwcc/yearcount?network=sntl&counttype=listwithdiscontinued") %>%
  rvest::read_html() %>%
  rvest::html_table() %>%
  data.frame()
khufkens commented 2 years ago

Fixed: https://github.com/bluegreen-labs/snotelr/commit/c00b8aaf6afe045162ee92e3b81fa9fd4ebc8651

Install from github for now