getOptionChain() working discretely "Error in open.connection(con, "rb") : HTTP error 500."

jrburl commented 4 years ago

Description

thanks for your work Unable to pull option chains for a few high profile companies. With the same code, am able to pull for other companies. Does work for 'F', doesn't work for 'MSFT', 'AAPL'.

Expected behavior

Expecting a response.

Minimal, reproducible example

library(ggplot2)
library(reshape2)
library(lubridate)
library(xlsx)
library(quantmod)
library(readxl)
library(purrr)
library(dplyr)
library(plotly)
library(BatchGetSymbols)
library(derivmkts)

test_ticker <- 'AAPL'
test_total = data.frame()
spot_check <- 
  tryCatch(
  test <- as.matrix(quantmod::getOptionChain(test_ticker,Exp = NULL)),
      error=function(e) e)
  mylist = do.call(rbind,do.call(rbind,test)) %>%
    tibble::rownames_to_column(., "Name")

  test_total <- rbind(test_total,mylist)

  View(test_total)

joshuaulrich commented 4 years ago

Thanks for the report. I cannot replicate with a simple, minimal example:

x <- quantmod::getOptionChain("AAPL", Exp = NULL)
length(x)
# [1] 18
str(x[1:2])
# List of 2
#  $ Feb.21.2020:List of 2
#   ..$ calls:'data.frame': 78 obs. of  7 variables:
#   .. ..$ Strike: num [1:78] 100 105 110 120 130 135 140 150 155 160 ...
#   .. ..$ Last  : num [1:78] 224 219 214 205 194 ...
#   .. ..$ Chg   : int [1:78] 0 0 0 0 0 0 0 0 0 0 ...
#   .. ..$ Bid   : num [1:78] 0 0 0 0 0 0 0 0 0 0 ...
#   .. ..$ Ask   : int [1:78] 0 0 0 0 0 0 0 0 0 0 ...
#   .. ..$ Vol   : int [1:78] 2 NA 5 NA NA 3 7 180 NA 31 ...
#   .. ..$ OI    : int [1:78] 0 0 0 NA 0 0 0 0 0 0 ...
#   ..$ puts :'data.frame': 74 obs. of  7 variables:
#   .. ..$ Strike: num [1:74] 100 105 110 115 120 125 130 135 140 145 ...
#   .. ..$ Last  : num [1:74] 0.01 0.01 0.01 0.01 0.01 0.02 0.03 0.03 0.03 0.04 ...
#   .. ..$ Chg   : int [1:74] 0 0 0 0 0 0 0 0 0 0 ...
#   .. ..$ Bid   : num [1:74] 0 0 0 0 0 0 0 0 0 0 ...
#   .. ..$ Ask   : num [1:74] 0.01 0.04 0.01 0 0 0.08 0.08 0.01 0 0 ...
#   .. ..$ Vol   : int [1:74] 8 96 124 270 32 25 15 45 35 1 ...
#   .. ..$ OI    : int [1:74] 1724 0 193 0 0 85 89 110 0 0 ...
#  $ Feb.28.2020:List of 2
#   ..$ calls:'data.frame': 60 obs. of  7 variables:
#   .. ..$ Strike: num [1:60] 200 205 210 215 220 225 230 235 240 245 ...
#   .. ..$ Last  : num [1:60] 127 112 114 103 104 ...
#   .. ..$ Chg   : int [1:60] 0 0 0 0 0 0 0 0 0 0 ...
#   .. ..$ Bid   : num [1:60] 0 0 0 0 0 0 0 0 0 0 ...
#   .. ..$ Ask   : int [1:60] 0 0 0 0 0 0 0 0 0 0 ...
#   .. ..$ Vol   : int [1:60] 15 24 45 10 67 28 1 1 2 15 ...
#   .. ..$ OI    : int [1:60] 0 0 0 0 0 0 0 0 0 0 ...
#   ..$ puts :'data.frame': 58 obs. of  7 variables:
#   .. ..$ Strike: num [1:58] 200 205 210 215 220 225 230 235 240 245 ...
#   .. ..$ Last  : num [1:58] 0.01 0.01 0.01 0.01 0.02 0.01 0.01 0.02 0.01 0.02 ...
#   .. ..$ Chg   : int [1:58] 0 0 0 0 0 0 0 0 0 0 ...
#   .. ..$ Bid   : num [1:58] 0 0 0 0 0.01 0 0 0.03 0 0 ...
#   .. ..$ Ask   : int [1:58] 0 0 0 0 0 0 0 0 0 0 ...
#   .. ..$ Vol   : int [1:58] 9 168 602 402 402 49 29 49 7 113 ...
#   .. ..$ OI    : int [1:58] 0 0 0 0 0 0 0 0 0 0 ...

A HTTP 500 error is a generic error that suggests something went wrong with the Yahoo Finance server(s). I suspect your problem will be resolved once your request goes to a functioning Yahoo Finance server.

So I'm going to close this because I don't think there's anything I can do to help resolve your issue. Please feel free to re-open it if you think I've missed something.

Also, I'd appreciate if your minimal reproducible example were more minimal. I hope you would agree that it's not minimal to attach 11 packages (and however many their Depends also attach) that require installation of nearly 100 packages. It probably didn't matter in this case, but interactions between attached packages can potentially cause issues that are very hard to track down.

jrburl commented 4 years ago

Sincerely appreciate the second look. Thank you.

Sent from my iPhone

On Feb 20, 2020, at 7:04 AM, Joshua Ulrich notifications@github.com wrote: Thanks for the report. I cannot replicate with a simple, minimal example:

x <- quantmod::getOptionChain("AAPL", Exp = NULL) length(x)

[1] 18

str(x[1:2])

List of 2

$ Feb.21.2020:List of 2

..$ calls:'data.frame': 78 obs. of 7 variables:

.. ..$ Strike: num [1:78] 100 105 110 120 130 135 140 150 155 160 ...

.. ..$ Last : num [1:78] 224 219 214 205 194 ...

.. ..$ Chg : int [1:78] 0 0 0 0 0 0 0 0 0 0 ...

.. ..$ Bid : num [1:78] 0 0 0 0 0 0 0 0 0 0 ...

.. ..$ Ask : int [1:78] 0 0 0 0 0 0 0 0 0 0 ...

.. ..$ Vol : int [1:78] 2 NA 5 NA NA 3 7 180 NA 31 ...

.. ..$ OI : int [1:78] 0 0 0 NA 0 0 0 0 0 0 ...

..$ puts :'data.frame': 74 obs. of 7 variables:

.. ..$ Strike: num [1:74] 100 105 110 115 120 125 130 135 140 145 ...

.. ..$ Last : num [1:74] 0.01 0.01 0.01 0.01 0.01 0.02 0.03 0.03 0.03 0.04 ...

.. ..$ Chg : int [1:74] 0 0 0 0 0 0 0 0 0 0 ...

.. ..$ Bid : num [1:74] 0 0 0 0 0 0 0 0 0 0 ...

.. ..$ Ask : num [1:74] 0.01 0.04 0.01 0 0 0.08 0.08 0.01 0 0 ...

.. ..$ Vol : int [1:74] 8 96 124 270 32 25 15 45 35 1 ...

.. ..$ OI : int [1:74] 1724 0 193 0 0 85 89 110 0 0 ...

$ Feb.28.2020:List of 2

..$ calls:'data.frame': 60 obs. of 7 variables:

.. ..$ Strike: num [1:60] 200 205 210 215 220 225 230 235 240 245 ...

.. ..$ Last : num [1:60] 127 112 114 103 104 ...

.. ..$ Chg : int [1:60] 0 0 0 0 0 0 0 0 0 0 ...

.. ..$ Bid : num [1:60] 0 0 0 0 0 0 0 0 0 0 ...

.. ..$ Ask : int [1:60] 0 0 0 0 0 0 0 0 0 0 ...

.. ..$ Vol : int [1:60] 15 24 45 10 67 28 1 1 2 15 ...

.. ..$ OI : int [1:60] 0 0 0 0 0 0 0 0 0 0 ...

..$ puts :'data.frame': 58 obs. of 7 variables:

.. ..$ Strike: num [1:58] 200 205 210 215 220 225 230 235 240 245 ...

.. ..$ Last : num [1:58] 0.01 0.01 0.01 0.01 0.02 0.01 0.01 0.02 0.01 0.02 ...

.. ..$ Chg : int [1:58] 0 0 0 0 0 0 0 0 0 0 ...

.. ..$ Bid : num [1:58] 0 0 0 0 0.01 0 0 0.03 0 0 ...

.. ..$ Ask : int [1:58] 0 0 0 0 0 0 0 0 0 0 ...

.. ..$ Vol : int [1:58] 9 168 602 402 402 49 29 49 7 113 ...

.. ..$ OI : int [1:58] 0 0 0 0 0 0 0 0 0 0 ...

A HTTP 500 error is a generic error that suggests something went wrong with the Yahoo Finance server(s). I suspect your problem will be resolved once your request goes to a functioning Yahoo Finance server.

So I'm going to close this because I don't think there's anything I can do to help resolve your issue. Please feel free to re-open it if you think I've missed something.

Also, I'd appreciate if your minimal reproducible example were more minimal. I hope you would agree that it's not minimal to attach 11 packages (and however many their Depends also attach) that require installation of nearly 100 packages. It probably didn't matter in this case, but interactions between attached packages can potentially cause issues that are very hard to track down.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

joshuaulrich commented 4 years ago

Here's another example of an error caused by the Yahoo Finance server response.

I ran this code 2 days ago and got an error:

R> x <- quantmod::getOptionChain("MSFT", Exp = NULL)
Error in data.frame(Strike = strike, Last = lastprice, Chg = change, Bid = bid,  : 
  object 'volume' not found

I ran it again just now and it worked. It might have something to do with the weekly options. We might be able to address that in quantmod, if we can find the root cause of the error. I'd appreciate it if you're able to investigate. If not, no worries.

jrburl commented 4 years ago

@joshuaulrich quantmod::getOptionChain('PG', NULL) is failing for the same reasons. I went and checked out the option chain on Yahoo Finance and if you check out the 4/9/2020 expiry it's got near zilch for volume. No volume on calls, 1 on puts. I believe this is the culprit.

quantmod::getOptionChain('PG', Exp = "2020-04-09") fails w/ the same error & adds to this idea.

jrburl commented 4 years ago

Have submitted a fork request. I think this is what you're looking for after your NewToOld function:

                        Vol= if("volume" %in% names(x))
                        {
                          volume
                        }
                        else
                        {
                          NA
                        }

Also. The same needs to be done for open interest. I've seen that fail for the same reason as well.

joshuaulrich commented 4 years ago

I see your patch, thanks! It looks ready to merge to me, but you need to click the [Create Pull Request] button for me to be able to merge. Thank you very much for taking the initiative to fix that issue!

joshuaulrich commented 4 years ago

As @jrburl mentioned in the commit comment, this error occurred again. Thankfully, I can replicate, so I'm reopening.

Here's some relevant information from stepping through the debug call that fails.

debugonce(quantmod:::getOptionChain.yahoo)
oc <- quantmod::getOptionChain("FB", Exp = "2020-04-09")
...
# Browse[2]> all.expiries.posix
 [1] "2020-03-13 UTC" "2020-03-20 UTC" "2020-03-27 UTC" "2020-04-03 UTC"
 [5] "2020-04-09 UTC" "2020-04-17 UTC" "2020-04-24 UTC" "2020-05-01 UTC"
 [9] "2020-05-15 UTC" "2020-06-19 UTC" "2020-09-18 UTC" "2020-11-20 UTC"
[13] "2021-01-15 UTC" "2021-03-19 UTC" "2021-06-18 UTC" "2022-01-21 UTC"
[17] "2026-02-21 UTC"
...
Browse[2]> 
debug: return(getOptionChain.yahoo(Symbols, expiry.subset, .expiry.known = TRUE))
Browse[2]> expiry.subset
[1] 1586390400
Browse[2]> .POSIXct(expiry.subset, tz = "UTC")
[1] "2020-04-09 UTC"
Browse[2]> 
Error in open.connection(con, "rb") : HTTP error 500.

So here's some relevant information from stepping through a debugging call to getOptionChain.yahoo()

quantmod:::getOptionChain.yahoo("FB", 1586390400, .expiry.known = TRUE)
# Error in open.connection(con, "rb") : HTTP error 500.
debugonce(quantmod:::getOptionChain.yahoo)
...
Browse[2]> 
# debug: tbl <- jsonlite::fromJSON(urlExp)
Browse[2]> urlExp
# [1] "https://query2.finance.yahoo.com/v7/finance/options/FB?&date=1586390400"
Browse[2]> 
# Error in open.connection(con, "rb") : HTTP error 500.

joshuaulrich commented 4 years ago

Something is odd with the site. I'm able to select the 2020-04-09 expiry from the drop-down, but all the contracts listed in the output do not change. The values are for whatever expiry was last selected. In the screenshot below, that's 2020-04-03.

Screenshot from 2020-03-12 21-05-50

jrburl commented 4 years ago

Good evening. The flavor of the day is GOOG200501. Appears to be on yahoo's end.

Screen Shot 2020-03-27 at 8 53 18 PM Screen Shot 2020-03-27 at 8 33 21 PM

joshuaulrich commented 4 years ago

Hey @jrburl, what do you think of the patch I just pushed?

Here's an example of the output when Exp = NULL and there are expiry dates that do not have data. In this case, oc will not have list elements named "Jun.15.2020", "Jun.17.2020", "Jun.26.2020", or "Mar.18.2022".

oc <- quantmod::getOptionChain("SPY", NULL)
Warning: no data for 'SPY' expiry 2020-06-15, omitting
    (server response: HTTP error 500.)
Warning: no data for 'SPY' expiry 2020-06-17, omitting
    (server response: HTTP error 500.)
Warning: no data for 'SPY' expiry 2020-06-26, omitting
    (server response: HTTP error 500.)
Warning: no data for 'SPY' expiry 2022-03-18, omitting
    (server response: HTTP error 500.)

ekuutan commented 4 years ago

It stopped working altogether for me "Error in open.connection(con, "rb") : HTTP error 500"

joshuaulrich / quantmod