search

sstrickx / yahoofinance-api

Java Client API for Yahoo Finance
MIT License
498 stars 219 forks source link

Server returned HTTP response code: 429 for URL: https://query1.finance.yahoo.com/v1/test/getcrumb #211

Open zytan89 opened 1 year ago

zytan89 commented 1 year ago

I am having issue when run code line CrumbManager.getCrumb();

210

java.io.IOException: Server returned HTTP response code: 429 for URL: https://query1.finance.yahoo.com/v1/test/getcrumb at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) at java.lang.reflect.Constructor.newInstance(Unknown Source) at sun.net.www.protocol.http.HttpURLConnection$10.run(Unknown Source) at sun.net.www.protocol.http.HttpURLConnection$10.run(Unknown Source) at java.security.AccessController.doPrivileged(Native Method) at sun.net.www.protocol.http.HttpURLConnection.getChainedException(Unknown Source) at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(Unknown Source) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source) at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(Unknown Source) at yahoofinance.histquotes2.CrumbManager.setCrumb(CrumbManager.java:167) at yahoofinance.histquotes2.CrumbManager.refresh(CrumbManager.java:182) at yahoofinance.histquotes2.CrumbManager.getCrumb(CrumbManager.java:187) at yahoofinance.quotes.query1v7.QuotesRequest.getResult(QuotesRequest.java:65) at yahoofinance.YahooFinance.getQuotes(YahooFinance.java:381) at yahoofinance.YahooFinance.get(YahooFinance.java:98) at yahoofinance.YahooFinance.get(YahooFinance.java:82) at stock.api.StockApi.getHistoryData(StockApi.java:21) at monitor.Screener.getData(Screener.java:194) at monitor.Screener.screen(Screener.java:103) at monitor.Screener.run(Screener.java:77) at monitor.MonitorPanel.lambda$1(MonitorPanel.java:285) at java.lang.Thread.run(Unknown Source)

lumirodri commented 1 year ago

@zytan89 Getting the same issue as of 2 days ago.

Apparently, they're now applying a rate limit, throttling the number of requests - which in turn results in the error code 429 - "Too Many Requests".

This seems like it's device and geo-based. I have users reporting that they don't get any data, and others saying it's working as before, although if I do test https://query1.finance.yahoo.com/v1/test/getcrumb on my browser, I always get a crumb.

If I fire up Android Studio and do the same request through the emulator, I get the error you described.

I'm not sure how, or if it's even possible to fix this issue.

pktczwd commented 1 year ago

You guys can try to add a "user-agent" header to the getcrumb request, then it can work as before.

lumirodri commented 1 year ago

You guys can try to add a "user-agent" header to the getcrumb request, then it can work as before.

I've wasted a couple hours fiddling with the code and I can say this indeed works (if you also change some other stuff). Here are the steps I've used:

Thanks for the tip! @pktczwd

DISCLAIMER: this should only be used for testing purposes, obviously. You shouldn't spoof user-agent!

pktczwd commented 1 year ago

You guys can try to add a "user-agent" header to the getcrumb request, then it can work as before.

I've wasted a couple hours fiddling with the code and I can say this indeed works (if you also change some other stuff). Here are the steps I've used:

  • Step A:

    • FILE - CookieAndCrumbManager.java

    • Get a specific user-agent (e.g. access the website via your PC https://www.whatsmyua.info/ and retrieve the value.

    • On cookie request, add:

    • connection.addRequestProperty("User-Agent", YOUR_VALUE_HERE)

    • On crumb request, add:

    • connection.setRequestProperty("Cookie", cookie);

    • connection.addRequestProperty("Cookie", cookie);

    • connection.addRequestProperty("User-Agent", YOUR_VALUE_HERE)

    • connection.setRequestMethod("GET");

  • Step B:

    • FILE - QuotesRequest.java

    • Change the type of connection from RedirectableRequest to HttpURLConnection (as per CookieAndCrumbManager)

    • On get result, add

    • connection.setRequestProperty("Cookie", cookie);

    • connection.addRequestProperty("Cookie", cookie);

    • connection.addRequestProperty("User-Agent", YOUR_VALUE_HERE)

Thanks for the tip! @pktczwd

DISCLAIMER: this should only be used for testing purposes, obviously. You shouldn't spoof user-agent!

Yes, I also wasted many hours on it before.

  1. No need to add user-agent to the cookie request, only the cookie and user-agent are mandatory.
  2. Remove the CookieHandler in yahoofinance.util.RedirectableRequest (and manage the cookie mannually in yahoofinance.histquotes2.CrumbManager).
  3. Add a Accept header with value text/html to the get cookie request.
  4. Remove the pattern limitation in setCookie method, and just store all the cookies.
  5. Do not forget include the crumb parameter in your query reqeust.
playinlab commented 1 year ago

Hi,

I have the same issue (HTTP response code: 429).

Is it possible to download a working YahooFinanceAPI-3.18.0-SNAPSHOT.jar somewhere?

Best regards, Primož

playinlab commented 1 year ago

Hi,

Can you at least attach the files which were changed in your working solution please (CrumbManager.java, QuoteRequest.java...).

I tried many different things but I can not get data from yahoo. I always get an error:

java.io.IOException: Server returned HTTP response code: 429 for URL: https://query1.finance.yahoo.com/v1/test/getcrumb at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:na] at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[na:na] at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:na] at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490) ~[na:na]

Best regards

playinlab commented 1 year ago

Hi guys,

I solved the problem. Thanks @pktczwd for your hint.

Best regards, Primož

SamSambuka commented 1 year ago

Hi, unfortunately i can't get it to work. can please someone write a step by step tutorial. or create a pull request with code changes. thanks!

kpavan536 commented 1 year ago

@pktczwd and @playinlab followed same steps but getting java.lang.IllegalStateException: Already connected at java.base/sun.net.www.protocol.http.HttpURLConnection.setRequestProperty(HttpURLConnection.java:3316) at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.setRequestProperty(HttpsURLConnectionImpl.java:286) at yahoofinance.histquotes2.CrumbManager.setCookie(CrumbManager.java:51) at yahoofinance.histquotes2.CrumbManager.refresh(CrumbManager.java:216) at yahoofinance.histquotes2.CrumbManager.getCrumb(CrumbManager.java:222) at yahoofinance.quotes.query1v7.QuotesRequest.getResult(QuotesRequest.java:63) at yahoofinance.YahooFinance.getQuotes(YahooFinance.java:381)

kpavan536 commented 1 year ago

with below peace of code in get result. it is working //URLConnection connection = redirectableRequest.openConnection(); HttpURLConnection connection = (HttpURLConnection) request.openConnection(); connection.setRequestProperty("User-Agent", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.5.2 Safari/605.1.15"); connection.setRequestProperty("Cookie",CrumbManager.getCookie()); InputStreamReader is = new InputStreamReader(connection.getInputStream());

SamSambuka commented 1 year ago

@kpavan536 can you please provide code from CrumbManager.java and QuotesRequest.java ?

code-monkey-101 commented 1 year ago

@SamSambuka This is how I did it. Did not have to change QuotesRequest to make it work. image

Big thanks to @pktczwd and everybody else who provided assistance!

ghost commented 1 year ago

@code-monkey-101 could you show your code for the cookie portion as well?

code-monkey-101 commented 1 year ago

@code-monkey-101 could you show your code for the cookie portion as well?

I didn't (have to) make any changes to setCookie. This is the only change I made for error 429.

However, there was a similar issue (error 401) not too long ago for which I posted a fix here: https://github.com/sstrickx/yahoofinance-api/issues/206

SamSambuka commented 10 months ago

Hi, works for me with this changes: https://github.com/sstrickx/yahoofinance-api/compare/develop...v3rtumnus:yahoofinance-api:develop

moritz31 commented 6 months ago

@SamSambuka i tried your code but i am still stuck at 401. Based in eu and crumb on webpage gets me something, while the application it is always empty