search

ropensci / rnoaa

R interface to many NOAA data APIs
https://docs.ropensci.org/rnoaa
Other
330 stars 84 forks source link

Error in using ghcnd() function #172

Closed sckott closed 7 years ago

sckott commented 8 years ago

See https://ropensci.org/blog/2014/03/13/rnoaa/#comment-2949486345

sckott commented 8 years ago

I haven't seen the problem before. There shouldn't be a limit on how many requests you can make with ghcnd() as it's just pulling from an FTP server. (whereas the ncdc functions do have limits)

can you share your devtools::session_info() OR sessionInfo() with me so I can see what versions of packages you have?

catherinehulshof commented 8 years ago

Thank you for responding. This is my first time reaching out to the R forum community. Still learning the standards and etiquette (and how to share session_info, simply pasted below).

I am running RStudio AMI. I suspect this error might be the result of a memory issue. I'm running something like:

stationpre <- try(ghcnd(stationid=station[i,1]))

inside of a loop, getting precipitation data, running some analyses on the precipitation data and storing the results in a matrix. But when I surpass ~5000 iterations I receive the error message:

error

Does anything pop out as unusual?

Session info --------------------------------------------------------------------------------------------- setting value version R version 3.3.0 (2016-05-03) system x86_64, linux-gnu ui RStudio (0.99.896) language (EN) collate en_US.UTF-8 tz Zulu date 2016-10-14

Packages ------------------------------------------------------------------------------------------------- package * version date source
assertthat 0.1 2013-12-06 CRAN (R 3.3.0) colorspace 1.2-6 2015-03-11 CRAN (R 3.3.0) curl 2.1 2016-09-22 CRAN (R 3.3.0) DBI 0.5-1 2016-09-10 CRAN (R 3.3.0) devtools * 1.12.0 2016-06-24 CRAN (R 3.3.0) digest 0.6.9 2016-01-08 CRAN (R 3.3.0) dplyr * 0.5.0 2016-06-24 CRAN (R 3.3.0) forecast * 7.3 2016-10-13 CRAN (R 3.3.0) fracdiff * 1.4-2 2012-12-02 CRAN (R 3.3.0) ggplot2 2.1.0 2016-03-01 CRAN (R 3.3.0) gridExtra 2.2.1 2016-02-29 CRAN (R 3.3.0) gtable 0.2.0 2016-02-26 CRAN (R 3.3.0) httr 1.2.1 2016-07-03 CRAN (R 3.3.0) jsonlite 0.9.19 2015-11-28 CRAN (R 3.3.0) lattice 0.20-33 2015-07-14 CRAN (R 3.2.1) lazyeval 0.2.0 2016-06-12 CRAN (R 3.3.0) lubridate 1.6.0 2016-09-13 CRAN (R 3.3.0) magrittr 1.5 2014-11-22 CRAN (R 3.3.0) Matrix 1.2-6 2016-05-02 CRAN (R 3.3.0) memoise 1.0.0 2016-01-29 CRAN (R 3.3.0) mgcv * 1.8-12 2016-03-03 CRAN (R 3.2.5) mime 0.4 2015-09-03 CRAN (R 3.3.0) munsell 0.4.3 2016-02-13 CRAN (R 3.3.0) nlme * 3.1-127 2016-04-16 CRAN (R 3.2.5) nnet 7.3-12 2016-02-02 CRAN (R 3.2.5) plyr 1.8.3 2015-06-12 CRAN (R 3.3.0) quadprog 1.5-5 2013-04-17 CRAN (R 3.3.0) R6 2.1.2 2016-01-26 CRAN (R 3.3.0) rappdirs 0.3.1 2016-03-28 CRAN (R 3.3.0) Rcpp 0.12.4 2016-03-26 CRAN (R 3.3.0) reshape2 * 1.4.1 2014-12-06 CRAN (R 3.3.0) rnoaa * 0.6.4 2016-10-07 CRAN (R 3.3.0) RStudioAMI * 0.2 2016-05-08 local
scales 0.4.0 2016-02-26 CRAN (R 3.3.0) stringi 1.0-1 2015-10-22 CRAN (R 3.3.0) stringr 1.0.0 2015-04-30 CRAN (R 3.3.0) taRifx * 1.0.6 2014-05-29 CRAN (R 3.3.0) tibble 1.2 2016-08-26 CRAN (R 3.3.0) tidyr 0.6.0 2016-08-12 CRAN (R 3.3.0) timeDate * 3012.100 2015-01-23 CRAN (R 3.3.0) tseries * 0.10-35 2016-05-02 CRAN (R 3.3.0) withr 1.0.2 2016-06-20 CRAN (R 3.3.0) XML 3.98-1.4 2016-03-01 CRAN (R 3.3.0) xml2 1.0.0 2016-06-24 CRAN (R 3.3.0) zoo * 1.7-13 2016-05-03 CRAN (R 3.3.0) `

sckott commented 8 years ago

thanks for the report @catherinehulshof - i'll have a look and get back to you soon

sckott commented 7 years ago

@catherinehulshof Can you give me any ghcnd call that leads to that error, curious if it has to do with certain stations

sckott commented 7 years ago

@catherinehulshof it's possible your problem is related to https://github.com/ropensci/rnoaa/issues/173 - which we're fixing now

catherinehulshof commented 7 years ago

@sckott I am just catching up to this now. In response to your last comment I initially thought the error was related to certain stations but once I received the error I could no longer pull data from any station even those that I had already called, which made me think it was an error related to device storage or cache like one user suggested.

Reading through #173 sounds like you were able to identify the issue. Thank you so much for looking into this!

sckott commented 7 years ago

@catherinehulshof to be sure, the error in that issue only happened when you installed binary version of the package. Is that how you installed it?

catherinehulshof commented 7 years ago

@sckott The first time I installed the binary version of the package and received the error after ~5,000 iterations of a loop that calls precipitation data for station after station. After troubleshooting, I uninstalled and re-installed the pacakage instead using the source version and received the same error, again after ~5,000 iterations. The only other thing I can think of is that I'm running a clunky for loop. Do you think it might be related to using a loop as opposed to something more efficient like dplyr? I am working on optimizing now.

sckott commented 7 years ago

Not sure what it could be. If it happened with the binary and source version , then it's not related to the other I think.

You could be running out of space on the disk you have possibly, seems possible since it seems to happen after a lot of iterations

I think for loops are fine in this case, makes it easier to reason about I think

catherinehulshof commented 7 years ago

Thank you! I'll keep experimenting.

sckott commented 7 years ago

closing due to inactivity - can reopen if needed