getSymbols.yahoo() always throws "Error in new.session()"

abernal30 commented 2 years ago

Description

When applying the function getSymbols(), it errors and does not return a result.

Minimal, reproducible example

getSymbols("AAPL")
## Error in new.session() : Could not establish session after 5 attempts.

joshuaulrich commented 2 years ago

Hi @zlfang00. That's very odd and it's not clear why remotes::install_github() thinks the packages it just installed are not available. I could be missing something, or it may be a bug in the function. Either way, I'm afraid I can't help.

joshuaulrich commented 2 years ago

'package ‘quantmod’ is in use and will not be installed

@yw6010, that means you have an R session that already has quantmod loaded, so you can't change anything in the directory the package is installed in. You need to close all your R sessions on that computer, install quantmod, and then you can load and use quantmod.

ethanbsmith commented 2 years ago

rstudio can load and keep links to packages active for intellisense parsing and such, even after a Session -> Restart R. best to shut down everything and do installs from a clean vanilla r session.

yw6010 commented 2 years ago

'package ‘quantmod’ is in use and will not be installed

@yw6010, that means you have an R session that already has quantmod loaded, so you can't change anything in the directory the package is installed in. You need to close all your R sessions on that computer, install quantmod, and then you can load and use quantmod.

it works now!!! very appreciate. time to resume my project haha

rhamo commented 2 years ago

Thank you for the quick fix @joshuaulrich Once installing "remotes::install_github("joshuaulrich/quantmod@358-getsymbols-new.session")" this worked for me once when requesting a small group of tickers, however all new attempts since I now constantly receive "Not Cached - Error in download.." errors.

Example:

I have tried restarting R but now I cannot even receive one successful return. Each ticker states "Error in download" once again. Any ideas?

Thank you

edwinhung commented 2 years ago

Hi, Thank you very much for the fix and comments here. getSymols.yahoo() now works for me, but I have a different problem. When I run getSymbols.yahoo() successfully in a loop for more than about 300-400 tickers, I started to get "HTTP error 401" for all following downloads. There are some failed downloads of invalid tickers in between though. Does anyone know what the issue is? Maybe I have the same problem with @rhamo.

Here is an example of the subsequent download: > getSymbols.yahoo("AAPL",from="2022-04-27",to="2022-04-29",auto.assign=FALSE) Warning: AAPL download failed; trying again. Warning: Unable to import “AAPL”. AAPL download failed after two attempts. Error message: HTTP error 401. [1] "Error in open.connection(file, \"rt\") : HTTP error 401.\n" attr(,"class") [1] "try-error" attr(,"condition") <simpleError in open.connection(file, "rt"): HTTP error 401.>

Thank you!

zpio commented 2 years ago

> install.packages("quantmod")
> library(quantmod)
> remotes::install_github("joshuaulrich/xts", force = TRUE)

* installing *source* package 'xts' ...
ERROR: cannot remove earlier installation, is it in use?
* removing 'C:/Users/Tama/Documents/R/win-library/3.6/xts'
* restoring previous 'C:/Users/Tama/Documents/R/win-library/3.6/xts'
Error in file.copy(lp, dirname(pkgdir), recursive = TRUE, copy.date = TRUE) : 
  (converted from warning) problem copying C:\Users\Tama\Documents\R\win-library\3.6\00LOCK-xts\xts\libs\x64\xts.dll to C:\Users\Tama\Documents\R\win-library\3.6\xts\libs\x64\xts.dll: Permission denied
* removing 'C:/Users/Tama/Documents/R/win-library/3.6/xts'
* restoring previous 'C:/Users/Tama/Documents/R/win-library/3.6/xts'
Error in file.copy(lp, dirname(pkgdir), recursive = TRUE, copy.date = TRUE) : 
  (converted from warning) problem copying C:\Users\Tama\Documents\R\win-library\3.6\00LOCK-xts\xts\libs\x64\xts.dll to C:\Users\Tama\Documents\R\win-library\3.6\xts\libs\x64\xts.dll: Permission denied
Execution halted
Error: Failed to install 'xts' from GitHub:
  (converted from warning) installation of package ‘C:/Users/Tama/AppData/Local/Temp/RtmpAnQpkC/file449050346c2/xts_0.12.1.2.tar.gz’ had non-zero exit status

> remotes::install_github("joshuaulrich/quantmod@358-getsymbols-new.session")

 installing *source* package 'quantmod' ...
   ** using staged installation
   ** R
   ** demo
   ** byte-compile and prepare package for lazy loading
   Error: package 'xts' required by 'quantmod' could not be found
   Execution halted
   ERROR: lazy loading failed for package 'quantmod'
-  removing 'C:/Users/Tama/AppData/Local/Temp/RtmpqubKOV/Rinst31ac26a1569f/quantmod'
         -----------------------------------
   ERROR: package installation failed
Error: Failed to install 'quantmod' from GitHub:
  System command 'Rcmd.exe' failed, exit status: 1, stdout + stderr (last 10 lines):
E> ** using staged installation
E> ** R
E> ** demo
E> ** byte-compile and prepare package for lazy loading
E> Error: package 'xts' required by 'quantmod' could not be found
E> Execution halted
E> ERROR: lazy loading failed for package 'quantmod'
E> * removing 'C:/Users/Tama/AppData/Local/Temp/RtmpqubKOV/Rinst31ac26a1569f/quantmod'
E>       -----------------------------------
E> ERROR: package installation failed

I don't know what else to do. Help

BathosEconTrade commented 2 years ago

get this as well when apply patch: "Warning: package ‘quantmod’ is in use and will not be installed" this after I removed quantmod and Quandl and then new session then new install. get this: Error in new.session() : Could not establish session after 5 attempts.Error in new.session() : Could not establish session after 5 attempts. Error in new.session() : Could not establish session after 5 attempts. I am so grateful for quantmod and really dependent upon it

rpfreitasxyz commented 2 years ago

Hi, Thank you very much for the fix and comments here. getSymols.yahoo() now works for me, but I have a different problem. When I run getSymbols.yahoo() successfully in a loop for more than about 300-400 tickers, I started to get "HTTP error 401" for all following downloads. There are some failed downloads of invalid tickers in between though. Does anyone know what the issue is? Maybe I have the same problem with @rhamo.

Here is an example of the subsequent download: > getSymbols.yahoo("AAPL",from="2022-04-27",to="2022-04-29",auto.assign=FALSE) Warning: AAPL download failed; trying again. Warning: Unable to import “AAPL”. AAPL download failed after two attempts. Error message: HTTP error 401. [1] "Error in open.connection(file, \"rt\") : HTTP error 401.\n" attr(,"class") [1] "try-error" attr(,"condition") <simpleError in open.connection(file, "rt"): HTTP error 401.>

Thank you!

I've been having the same issue!

zlfang00 commented 2 years ago

Hi, @joshuaulrich, One thing I noticed is zoo is already installed in the public R library, while xts and TTR are not. My guess is this version quantmod only searches for its dependencies in the public library path? Could you please confirm of that? Thanks very much!

Hi @zlfang00. That's very odd and it's not clear why remotes::install_github() thinks the packages it just installed are not available. I could be missing something, or it may be a bug in the function. Either way, I'm afraid I can't help.

BathosEconTrade commented 2 years ago

ok remote is working then use getSymbols.yahoo("AAPL) and still get: Error in new.session() : Could not establish session after 5 attempts. rpfreitasxyz you are doing better than I at least working but this this other problem. any insights would be great. all my models are based on getSymbols and yahoo, and am sure everyone notices is a very interesting time now. many thanks

joshuaulrich commented 2 years ago

Patched quantmod on CRAN now

The source package for quantmod 0.4.20 has been released to CRAN! It may take another day or two for CRAN to build the Windows and macOS binaries, so users on those systems can install from source or wait for the binaries to be built.

Yahoo Finance alternative

I also want to mention an alternative to Yahoo Finance data. I've found the end of day data from tiingo to be higher quality and the endpoint is more stable than Yahoo Finance. You need to sign up to get a free API key, and that gives individuals free access with 500 requests/hour and 20,000 requests/day, up to 5GB of data/month.

Disclaimer: While I don't receive any referral compensation for pointing individuals to the Free Plan or Power Plan, I may receive compensation (I honestly don't remember how much) from them for new Commercial Plan customers who learned of tiingo through me. But you have to mention me when you join, I don't have an affiliate link or anything like that.

joshuaulrich commented 2 years ago

@rhamo, @edwinhung, and @rpfreitasxyz A HTTP 401 response means the request is "unauthorized". That can happen for various reasons, but my guess is that you're hitting rate limits imposed by the Yahoo Finance servers. getSymbols() adds pauses when you request a lot of symbols at once to try and avoid hitting these rate limits. @msperlin (author of BatchGetSymbols) may have a better idea.

Let's move this conversation to #360.

@zpio you shouldn't need to install xts from GitHub. You only need the version on CRAN. Can you try to install quantmod via remotes::install_github("joshuaulrich/quantmod") and see if that works?

@zlfang00 The new version of quantmod uses zoo, xts, and TTR just like prior versions. Nothing has changed in that regard, so I'm stumped about what's causing your issue.

Error in new.session() : Could not establish session after 5 attempts.

@BathosEconTrade That looks like version 0.4.18 is being loaded instead of 0.4.19 or 0.4.20. Check packageVersion("quantmod") to see which version you're loading. Double check your library paths in case you are using more than one library...

BathosEconTrade commented 2 years ago

Many thanks all! Great help. Must have version 4.2.

rhamo commented 2 years ago

Thank you for the quick fix @joshuaulrich Once installing "remotes::install_github("joshuaulrich/quantmod@358-getsymbols-new.session")" this worked for me once when requesting a small group of tickers, however all new attempts since I now constantly receive "Not Cached - Error in download.." errors.

Example:

EMN | yahoo (161|506) | Found cache file - Got 100% of valid prices | Feels good! EMR | yahoo (162|506) | Found cache file - Got 100% of valid prices | Got it! ENPH | yahoo (163|506) | Found cache file - Got 100% of valid prices | Boa! EOG | yahoo (164|506) | Not Cached - Error in download.. EPAM | yahoo (165|506) | Not Cached - Error in download.. EQIX | yahoo (166|506) | Not Cached - Error in download.. EQR | yahoo (167|506) | Not Cached - Error in download.. ES | yahoo (168|506) | Not Cached - Error in download..

I have tried restarting R but now I cannot even receive one successful return. Each ticker states "Error in download" once again. Any ideas?

Thank you

Have tried with new quantmod 0.4.20 update but still no success. Every ticker states "Not Cached - Error in download" error as above.

Any ideas on how to solve this issue? Thanks again

samiranvekar commented 2 years ago

Thank you@joshuaulrich. The issue is resolved

msperlin commented 2 years ago

It seems that yahoo finance is now restricting access to the data. Once you download too much data, you get the "HTTP 401" error. This is controled by Yahoo and there is nothing myself or @joshuaulrich can do about it.

As with any webscrapping script, you can try to overcome the restriction with:

changing your ip one you reach the limit (use VPN)
use a caching system to save data locally and download data over the course of days. Usually the restricted ips are released in 24 hours. A caching system is already available in yfR/BatchGetSymbols).

Overall, if the limits are too restrictive, it might be best to find other source of price data.

NilssonHedge commented 2 years ago

Is there any indication of what the Yahoo Rate limit is?

I can also confirm that the VPN fix is working, thanks for putting some serious effort into this. :)

msperlin commented 2 years ago

@NilssonHedge Im looking into it right now.. It doenst seem too restrictive. After a 401 error, I can download again in a few minutes.

zlfang00 commented 2 years ago

are you sure it is the yahoo restriction? I tried Python package yfinance to grab the same size of date, no issues.

joshuaulrich commented 2 years ago

are you sure it is the yahoo restriction? I tried Python package yfinance to grab the same size of date, no issues.

The yfinance python library uses a different endpoint that returns JSON instead of a CSV. I'll look into adding something to getSymbols() that tries the other endpoint if one of them fails.

rhamo commented 2 years ago

Changing ip has worked wonders. Once again BatchGetSymbols now retrieves data without errors.

Thanks so much for all your help @msperlin and @joshuaulrich

mif2000 commented 2 years ago

Hi, Really appreciate your dedication. I'm a new user to R (v 3.5.3) through Bert for Excel. I finally had everything working to call R functionality and getSymbols (with a single ticker) through an Excel VBA macro and then it abruptly stopped as noted in prior comments. It appears that the problem with getSymbols has been resolved and that I can now download quantmod v 4.20 zip file for Windows (10), which I did from the quantmod.zip link on the CRAN page dtd 2022-04-29, copying the unzipped quantmod directory to the BERT2/lib folder. The command "library (quantmod)" ouputs an error: "package or namespace load failed for 'quantmod' in rbind(info, getNamespaceInfo(env,"S3methods")). packageVersion("quantmod") outputs '0.4.20'. I must be doing something wrong! What is the proper way to install the new version of quantmod?

Thanks for your help, mif2000

joshuaulrich / quantmod