AbhigyanBafna commented 5 months ago

Bug Report

Describe the bug

Quote and Quote summary stopped working recently due to invalid crumbs. First encountered it on my NextJS server and was able to reproduce the bug in CLI as well.

Minimal Reproduction

npx yahoo-finance2 quote AAPL

Environment

Node Node version (if applicable): v21.6.1 Npm version: 9.5.1

Additional Context

NarendranKT commented 5 months ago

I thought, I was only one to get this error. If someone has an solution to this drop it here. It will be useful for us.

th3l0g4n commented 5 months ago

From the Link in the yahoo error message: We’re sorry for the inconvenience, but API-level access to Yahoo Finance quotes data has been disabled.

Inofficial API Access might be over...

gadicc commented 5 months ago

I would strongly encourage everyone to go to that link and fill out the form. https://bit.ly/yahoo-finance-api-feedback

This project has always done its best to respect Yahoo's policies and while many efforts have been made to keep up-to-date with their technical requirements, we will never blatantly help to side-step their policies. Both for legal reasons but also out of great respect and appreciation for the data they have been providing us at no cost (!) until now.

The above form is an amazing step by Yahoo to try better understand developers who are using their API. Many serious users of this project (myself included) have looked at alternative (paid) options and found them inferior in many areas. Whereas in the past, Yahoo has mostly just ignored developers using their platform, it now looks like they're looking to seriously engage us and (perhaps) better address our needs.

The possibility of having a official API support from Yahoo is a big deal, and would result in much more stable use without us having to constantly ship updates for the frequent changes they make to their existing and basically internal API. I think having a small free plan for personal use with sensible limits would be amazing, as would a paid option which opens up the possibility for developers to monetize their hard work (alongside any local regulatory requirements).

So, again, I really encourage everyone to complete the form and see where this process leads. As users of this library, you may have enjoyed a reasonably stable experience (if you keep the library up-to-date), but as developers, keeping things working (and working correctly) against a constantly changing service is a huge effort and massive energy drain. Something official from Yahoo would be absolutely incredibly for all involved, so again, I encourage you take advantage of this opportunity and keep your fingers crossed.

gadicc commented 5 months ago

Thanks also to everyone above for reporting the issue, and for your input and feedback. I know this is a huge pain and inconvenience, but I really believe that Yahoo asking for developer feedback rather than just blocking us is a very positive step that could lead somewhere good for all of us.

programmishka commented 5 months ago

From the Link in the yahoo error message: _We’re sorry for the inconvenience, but API-level access to Yahoo Finance

This message from yahoo has been provided since many months. The API was still accessible using the node-yahoo-finance2 library, because it deals with cookies. However, the "Invalid Crumb" error is new, as of last week. The API responded last Tuesday, using the version 2.9.0 of the library.

programmishka commented 5 months ago

I am able to get a positive response from the API https://query2.finance.yahoo.com/v7/finance/quote?symbols=AAPL&crumb=XXXXXXX Using Postman, with my A1 cookie set.

programmishka commented 5 months ago

The library (v.2.11.1) works from my console: npx yahoo-finance2 quote AAPL Storing cookies in /Users/xxxxxx/.yf2-cookies.json Retrieved crumb from cookie store: ELG7IOJ9t.1 { language: 'en-US', region: 'US', ... For my case I need to find out, why the error "Invalid Crumb" occurs when I use the library (v.2.9.0 or v.2.11.1) in my Google Cloud Function. This problem I have after last Tuesday. But this is not the problem of the library, I think.

watadarkstar commented 5 months ago

Just started getting this error today. Sad to see this library not working anymore due to Yahoo disabling their API. Hopefully they open their API up even if its not free. I'd pay for it if needed.

almeidapaulooliveira commented 5 months ago

Confirmed not working on my side too. Hope Yahoo reopens as a paid service.

programmishka commented 5 months ago

As mentioned above, the library works from the console. I've modified my Google Cloud function to use axios and make direct requests to the quote API. The API works fine. You need to retrieve a crumb using your YF cookie and use it as an API parameter.

anupanup2001 commented 5 months ago

As mentioned above, the library works from the console. I've modified my Google Cloud function to use axios and make direct requests to the quote API. The API works fine. You need to retrieve a crumb using your YF cookie and use it as an API parameter.

I tried it. doesnt work for me. The library doesn't even work from the console for me.

npx yahoo-finance2 quote AAPL
Storing cookies in /home/neptune/.yf2-cookies.json
Retrieved crumb from cookie store: yRG6Yj2wkbt
Exiting with Error: Invalid Crumb. For Developers - https://bit.ly/yahoo-finance-api-feedback

Nathan-Franck commented 5 months ago

Welp, I filled out the form... Back to Selenium I go 😔

If anybody has any other simple stock/crypto price feeds you know about, let me know

Yhprum commented 5 months ago

I would strongly encourage everyone to go to that link and fill out the form.

This form first appeared last year around May or so, so it's definitely not a new thing they just made and set out - but who knows about any new API since the new Yahoo Finance interface is public now.

I assume this is just a different cookie/crumb/useragent thing to fix - for me it's working fine on my deployed site but not in dev (maybe because prod has an older crumb?) so I'll be looking into the issue on my end as well.

watadarkstar commented 5 months ago

As mentioned above, the library works from the console. I've modified my Google Cloud function to use axios and make direct requests to the quote API. The API works fine. You need to retrieve a crumb using your YF cookie and use it as an API parameter.

Can you explain this in more detail? Specifically how you retrieved a crumb?

Finalet commented 5 months ago

Getting same issues. Additionally, yahooFinance.options also returns Error: Invalid Crumb

mnsrulz commented 5 months ago

https://github.com/achannarasappa/ticker this package is still standing strong. It's not exactly the same but provide stock relation info using yahoo finance api's only. Any effort to incorporate the implementation detail from that package into this one?

nocodehummel commented 5 months ago

I just ran the tests and all test cases passed. Does it actually access the Yahoo API for any request in DEVEL mode?

vmetnev commented 5 months ago

As mentioned above, the library works from the console. I've modified my Google Cloud function to use axios and make direct requests to the quote API. The API works fine. You need to retrieve a crumb using your YF cookie and use it as an API parameter.

Can you provide the code for you solution? It would be very much appreciated!

nocodehummel commented 5 months ago

When removing some cached results the test cases are passed. Access to the API is working with fetching new results in the test cases.

programmishka commented 5 months ago

Can you explain this in more detail? Specifically how you retrieved a crumb?

Using the URL https://query2.finance.yahoo.com/v1/test/getcrumb You need your cookie for YF, named A1 or A3 for both - getcrumb and quote requests (they expire in 6 months, so need to update them sometimes). For the quote request you need to set the crumb as an URL parameter: https://query2.finance.yahoo.com/v7/finance/quote?crumb=XXXXX

I've just analyzed the network activity when searching for AAPL on the YF site and made the same requests. The API works.

UPD: for getcrumb request you also need to set the User-Agent header, otherwise it will respond with "too many requests".

nocodehummel commented 5 months ago

@programmishka in my logs i see that the crumb is retrieved which indicates that getCrumb is working.

gadicc commented 5 months ago

Hey all. Firstly, thanks everyone for chiming in. And wow, I personally had never seen the form until now, so thanks for letting me know it's been around for some time.

A few comments:

1) Also confirming that my old cookie and crumb, stored by the CLI in ~/.yf2-cookies.json, works great, but if I delete that file, I get the invalid crumb error for the newly retrieved cookie / crumb.

I guess that's encouraging that probably this issue could be worked around, but a) I unfortunately don't have any time to look at this now, and b) we need to be very careful of any legal implications (I'm thinking at least, requiring an extra step from the developer to acknowledge the situation with a link to yahoo's form).

2) @nocodehummel, as you noticed, all the tests used local disk cache for speed, however, if you set env var FETCH_DEVEL="nocache" it will instead make fresh network requests (link to docs). Note, this is very slow, can sometimes trigger a yahoo rate limiting ban (although I think we added our own rate-limiting code to avoid such bans), and also, a lot of tests that pass in the cache can sometimes fail when they added/removed fields we didn't accommodate yet.

Anyway, let's see what we can do as a community. As I said, I won't have a chance to investigate this now, but maybe others in the community might have some luck in the meantime. Thanks again to everyone above who has posted their findings so far - you guys are awesome! - and also to all users for your understanding of the situation and our stance.

P.S. Probably quickest workaround would be to access and accept all terms from your web browser, and then for us to have a good way to import the cookie/crumb for yf2. With the appropriate warnings, acknowledgements and links I mentioned above. Lastly, one big thanks again to this awesome community! :pray: :heart:

nocodehummel commented 5 months ago

@gadicc thanks for the feedback. The tests are also passed when deleting a cached file (quoteSummary-summaryProfile-AAPL.json). The YF API returns the data and it is saved again including the content, but formatted with escaped quotes. How can the tests be passed when the actual API is not working with invalid crumb?

gadicc commented 5 months ago

@nocodehummel, my pleasure, thanks for your interest in this :)

Ah yes, true, deleting a specific test file will re-fetch just for that test.
The fact that it's saving as quoted text is mildly worrying, it probably means that Yahoo isn't sending the correct content-type header, but I'd need to look through the code again to be sure.

How can the tests be passed when the actual API is not working with invalid crumb?

That sounds impossible `:) In short, I don't have an answer at this stage, until I'll have a chance to look at in more detail :/

nocodehummel commented 5 months ago

Some modules (endpoints) are actually working (e.g. calendar and time-series data), while others are not (e.g. quoteSummary).

my assumption how it is possible that all test passed, is because the cached crumb is valid.

gadicc commented 5 months ago

Ah yeah sorry, I thought you meant for new fetches (non-cached). Thanks.

Quick update: had a quick chance to look at this now. Using their /v1/test/getcrumb endpoint we can actually get a different crumb that is valid. It's a slightly different flow than what we used before. I'm on track but ran out of time, presumably I can get something working later this week, BUT, it requires changing the User-Agent, which goes against the guiding policies of this library, so, I need to think more about this.

nocodehummel commented 5 months ago

@gadicc I was on the same path. When calling that crumb endpoint with a non-browser user-agent it responds with Too Many Requests. The endpoint does work with Mozilla/5.0 (compatible; yahoo-finance2/0.0.1).

Yhprum commented 5 months ago

I've pushed a commit to my fork that fixed the issue for me - the only thing to consider is your stance on the user-agent thing. Everyone (and this library) using yahoo finance APIs has been using the yahoo finance APIs long after they had been unsupported and after they introduced this cookie+crumb thing last year so I have no qualms with this change, but ultimately it's up to you

https://github.com/Yhprum/yahoo-finance/commit/51b1e9ec6faa0b967e3166bcf4b8e6954cb57525

DDushkin commented 5 months ago

Having the same error, Shoot I need to set the notifications to my service, last successfully fetched quote was 2 days ago Is there anything I can help or do about it?

NarendranKT commented 5 months ago

Can you explain this in more detail? Specifically how you retrieved a crumb?

Using the URL https://query2.finance.yahoo.com/v1/test/getcrumb You need your cookie for YF, named A1 or A3 for both - getcrumb and quote requests (they expire in 6 months, so need to update them sometimes). For the quote request you need to set the crumb as an URL parameter: https://query2.finance.yahoo.com/v7/finance/quote?crumb=XXXXX

I've just analyzed the network activity when searching for AAPL on the YF site and made the same requests. The API works.

UPD: for getcrumb request you also need to set the User-Agent header, otherwise it will respond with "too many requests".

Thanks for the help. After tons of errors finally I got an response as a text and finally I parsed it as JSON.

gadicc commented 5 months ago

Hi all, a fix is building now and there will be an automated message here when the release is published.

Thanks to @AbhigyanBafna who first reported the issue and @nocodehummel, @programmishka, @Yhprum who helped to investigate and work towards a fix.

The endpoint does work with Mozilla/5.0 (compatible; yahoo-finance2/0.0.1).

@nocodehummel, this is a great find! And a great compromise that I'm happy to publish. I wasn't willing to ship something that falsified the user-agent, and had intended to provide some explanations and general directions on manually overriding the user-agent by hand. So, on behalf of all our users, thanks for figuring out something that works out the box!

Thanks again to everyone who contributed above. The magic of open source remains awesome and keeps helping everyone :pray: :heart:

gadicc commented 5 months ago

:tada: This issue has been resolved in version 2.11.2 :tada:

The release is available on:

Your semantic-release bot :package::rocket:

miliu99 commented 4 months ago

It appears it's broken again since this morning (5/30/2024):

Storing cookies in C:\Users\mingl\.yf2-cookies.json
Retrieved crumb from cookie store: yRG6Yj2wkbt
Exiting with Error: Invalid Crumb. For Developers - https://bit.ly/yahoo-finance-api-feedback

DDushkin commented 4 months ago

It appears it's broken again since this morning (5/30/2024):


Storing cookies in C:\Users\mingl\.yf2-cookies.json

Retrieved crumb from cookie store: yRG6Yj2wkbt

Exiting with Error: Invalid Crumb. For Developers - https://bit.ly/yahoo-finance-api-feedback

Try upgrade to the latest version, it helped me

miliu99 commented 4 months ago

OK, the latest version (2.11.3) works. I was using 2.11.2 and it worked until yesterday. Thank you very much, DDushkin.

raalarcon9705 commented 4 months ago

version 2.11.3 is not working for me

Fleker commented 4 months ago

I updated the package in the package.json, npm install'd, and now it does seem to work.

miliu99 commented 4 months ago

Yes, it started working for me again after I upgraded from 2.11.2 to 2.11.3 in package.json.

However, I just realized I'm still getting the same error when running command line npx yahoo-finance2 quote AAPL. Even with npx yahoo-finance2@2.11.3 quote AAPL, and it asks me to confirm to install the package first, it still gave me the same error. Any body can explain why?

gadicc commented 4 months ago

Thanks everyone for your comments and helping each other out 🙏

@miliu99, you might have a bad cookie stored in your .yf2-cookies.json file (used only by the CLI) from the old version. Can you try deleting this file and run the command again and see if it works then?

miliu99 commented 4 months ago

After deleting .yf2-cookies.json, I got this:

> npx yahoo-finance2 quote AAPL
Storing cookies in C:\Users\mingl\.yf2-cookies.json
Fetching crumb and cookies from https://finance.yahoo.com/quote/AAPL...
Exiting with Error: Unsupported redirect to https://finance.yahoo.com/quote/AAPL/, please report.

gadicc commented 4 months ago

@miliu99 are you sure you're definitely on 2.11.3 ? Because this is exactly what we fixed in that release, https://github.com/gadicc/node-yahoo-finance2/issues/777... actually the string of the error you showed doesn't even exist in this release 😅

what does the version field say in node_modules/yahoo-finance2/package.json ? If you only upped the version in your project's package.json you still need to tell your package manager to do the update.

miliu99 commented 3 months ago

I just took a closer look at this... Strangely, I found two folders in C:\Users\{UserName}\AppData\Local\npm-cache\_npx\ for yahoo-finance2, one is version "2.11.1" (installed 4/15/2024) and another "2.11.3" (installed 5/30/2024). In this state, I still get same error. So it appears it's still using the cache for 2.11.1. I then manually deleted the folder for 2.11.1, run the command again, it reinstalls yahoo-finance2 again (I can see 2.11.3 is already in cache, why reinstall?) and it started working as shown here:

>npx yahoo-finance2 quote AAPL
Need to install the following packages:
  yahoo-finance2@2.11.3
Ok to proceed? (y) y
Storing cookies in C:\Users\mingl\.yf2-cookies.json
Please consider completing the survey at https://bit.ly/yahoo-finance-api-feedback if you haven't already; for more info see https://github.com/gadicc/node-yahoo-finance2/issues/764#issuecomment-2056623851.
Fetching crumb and cookies from https://finance.yahoo.com/quote/AAPL...
We expected a redirect to guce.yahoo.com, but got https://finance.yahoo.com/quote/AAPL/
We'll try to continue anyway - you can safely ignore this if the request succeeds
Success. Cookie expires on Sun Jun 01 2025 16:43:27 GMT-0500 (Central Daylight Time)
fetch https://query1.finance.yahoo.com/v1/test/getcrumb
New crumb: MKI8vq023i9
{
  language: 'en-US',
  region: 'US',
  quoteType: 'EQUITY',
...

When I check the folder at C:\Users\{UserName}\AppData\Local\npm-cache\_npx\ again and see a new folders for yahoo-finance2 (installed today). So, somehow, the one installed on 5/30/2024 is not good but I couldn't see anything wrong when I compare two folders.

AbhigyanBafna commented 3 months ago

@miliu99 What about performing a clean install of yahoo-finance2 entirely with the latest flag?

npm i yahoo-finance2@latest

Or if you are using this in a project, you could just modify the package.json -

"yahoo-finance2": "^2.11.3"

miliu99 commented 3 months ago

@AbhigyanBafna As I mentioned earlier, it works fine in my project, but I was having trouble with npx command. Thanks.

gadicc / node-yahoo-finance2

Invalid Crumb / Yahoo disables API access to quotes data #764

Bug Report

Describe the bug

Minimal Reproduction

Environment

Additional Context