getSymbols.yahoo warnings

helgasoft commented 4 years ago

Description

warnings about failed downloads when they shouldn't be failing. It happens when getSymbols.yahoo gets multiple symbols, some of them invalid.

Expected behavior

getSymbols.yahoo to skip an invalid symbol and succeed getting the next valid one the first time

Minimal, reproducible example

> library(quantmod)
> myEnv <- new.env()
> getSymbols.yahoo(c('IBM','dummy1','GE'), from='2020-01-01', env=myEnv, verbose=TRUE)
downloading  IBM .....

done.
downloading  dummy1 .....

Warning: dummy1 download failed; trying again.
Warning: Unable to import “dummy1”.
dummy1 download failed after two attempts. Error message:
HTTP error 404.
downloading  GE .....

Warning: GE download failed; trying again.      <<==  should not happen
done.
[1] "IBM" "GE"

Solution

close and reopen the same connection, instead of abandon it and generate a new handle with a new connection.

# changes to getSymbols.yahoo code
  for (i in seq_along(Symbols)) {
...
      yahoo.URL <- quantmod:::.yahooURL(Symbols.name, from.posix, to.posix, interval, "history", handle)
      conn <- curl::curl(yahoo.URL, handle = handle$ch)         # new insert
      fr <- try(read.csv(conn, na.strings = "null"), silent = TRUE) # changed
      if (inherits(fr, "try-error")) {
        warning(Symbols.name, " download failed; trying again.", call. = FALSE, immediate. = TRUE)
        # handle <- .getHandle(curl.options, force.new = TRUE)
        # yahoo.URL <- .yahooURL(Symbols.name, from.posix, to.posix, interval, "history", handle)
        close(conn)                             # new insert
        conn <- curl::curl(yahoo.URL, handle = handle$ch)       # new insert
        fr <- try(read.csv(conn, na.strings = "null"), silent = TRUE)   # changed
        if (inherits(fr, "try-error")) {
          close(conn)                           # new insert
          stop(Symbols.name, " download failed after two attempts. Error", 
...

It's a small change to consider for the next version of quantmod. Prefer to leave it at that, too tiny for a pull request IMHO.

thesnowkeepsfalling commented 4 years ago

Hi - very glad to find this as am having a similar issue..

> myEnv <- new.env()
> getSymbols.yahoo(c('BHP.AX','RIO.AX','SXY.AX'), from='2020-01-01', env=myEnv, verbose=TRUE)
downloading  BHP.AX .....

done.
downloading  RIO.AX .....

done.
downloading  SXY.AX .....

done.
[1] "BHP.AX" "RIO.AX" "SXY.AX"
> getSymbols.yahoo(c('BHP.AX','Oops','SXY.AX'), from='2020-01-01', env=myEnv, verbose=TRUE)
downloading  BHP.AX .....

done.
downloading  Oops .....

Warning: Oops download failed; trying again.
Warning: Unable to import “Oops”.
Oops download failed after two attempts. Error message:
HTTP error 404.
downloading  SXY.AX .....

Warning: SXY.AX download failed; trying again.
done.

helgasoft commented 4 years ago

well, you still get SXY.AX, but on the second attempt, when it says "done". Even so, I think, the fix is needed. Tests with large lists of symbols show improved stability. The idea is to imitate browser requests, so just one handle/crumb is sufficient to do the job.

joshuaulrich commented 4 years ago

Thanks for the report and patch! I left the code that creates a new handle, URL, and connection. We may need a new handle if the download fails because of some issue with the existing handle. Other than that, your patch was perfect!

joshuaulrich / quantmod