search

ranaroussi / yfinance

Download market data from Yahoo! Finance's API
https://aroussi.com/post/python-yahoo-finance
Apache License 2.0
12.9k stars 2.29k forks source link

.info returns 404 on all tickers #1729

Closed ssvetli closed 9 months ago

ssvetli commented 10 months ago

@ValueRaider hijacking top post

Describe bug

Call to .info returns 404 error on all tickers. Example url: https://query2.finance.yahoo.com/v6/finance/quoteSummary/SPY?modules=financialData&modules=quoteType&modules=defaultKeyStatistics&modules=assetProfile&modules=summaryDetail&ssl=true Using browsers on multiple connections/devices, yahoo output "Will be right back... Our engineers are working quickly to resolve the issue."

Data could have moved to new link

Simple code that reproduces your problem

import yfinance as yf yf_ticker=yf.Ticker("SPY") yf_ticker.info

Debug log

HTTPError: 404 Client Error: Not Found for url: https://query2.finance.yahoo.com/v6/finance/quoteSummary/SPY?modules=financialData&modules=quoteType&modules=defaultKeyStatistics&modules=assetProfile&modules=summaryDetail&ssl=true

Bad data proof

No response

yfinance version

0.2.31

Python version

No response

Operating system

No response

vdahmane commented 10 months ago

The yahoo finance API looks down.

ssvetli commented 10 months ago

For reference some of the methods still working: .history .fast_info .news .earnings_dates .history_metadata .option_chain

parin-2002 commented 10 months ago

Try running on Google colab it works for me but still it is not working on local environment. Bit weird 🤔

ValueRaider commented 10 months ago

Firefox -> https://finance.yahoo.com/quote/AMZN -> F12 -> Network tab -> F5 -> compare requests vs yfinance That's essence of how to find fix.

ShaadyEmad commented 10 months ago

same issue but when i open USA vpn it works

andy60 commented 10 months ago

Try running on Google colab it works for me but still it is not working on local environment. Bit weird 🤔

i cant use my code on colab, i hope in resolution. I have error also in pyfinvis, its a strange period:)

bot-unit commented 10 months ago

May be it will help to switch to v7 or v10 api: https://query2.finance.yahoo.com/v1/test/getcrumb

ValueRaider commented 10 months ago

@bot-unit That reminds me, does #1657 help anyone?

RudyNL commented 10 months ago

yahooquery has the same problems

MrH32 commented 10 months ago

same issue but when i open USA vpn it works

You save my life.

USA VPN does work!

pedrosimao commented 10 months ago

I can confirm using USA VPN fixes the issue. Thanks for sharing @MrH32 !

junsyw commented 10 months ago

In my case using threads=False partially solves it

ShaadyEmad commented 10 months ago

In my case using threads=False partially solves it

can you explain more

junsyw commented 10 months ago

In my case using threads=False partially solves it

can you explain more It is a yfinace option(yd.download(share, threads=False))

centic9 commented 10 months ago

for me it is unrelated to threads, download seems to work always, others not:

#!/bin/python3

import yfinance as yf

print('Quotes:\n')
quotes = yf.download('ORCL', period = '1d')
print(quotes)

print('\n\nQuote:\n')
quote = yf.Ticker('ORCL')
print(quote.info['currentPrice'])
Quotes:

[*********************100%%**********************]  1 of 1 completed
                  Open        High         Low       Close   Adj Close    Volume
Date                                                                            
2023-10-20  106.989998  107.035004  100.660004  101.849998  101.849998  21718467

Quote:

Traceback (most recent call last):
  File "/opt/svn/DynatraceFinance/./test.py", line 11, in <module>
    print(quote.info['currentPrice'])
  File "/home/dstadler/.local/lib/python3.10/site-packages/yfinance/ticker.py", line 142, in info
    return self.get_info()
  File "/home/dstadler/.local/lib/python3.10/site-packages/yfinance/base.py", line 1736, in get_info
    data = self._quote.info
  File "/home/dstadler/.local/lib/python3.10/site-packages/yfinance/scrapers/quote.py", line 571, in info
    self._fetch(self.proxy)
  File "/home/dstadler/.local/lib/python3.10/site-packages/yfinance/scrapers/quote.py", line 600, in _fetch
    result = self._data.get_raw_json(
  File "/home/dstadler/.local/lib/python3.10/site-packages/yfinance/data.py", line 75, in get_raw_json
    response.raise_for_status()
  File "/home/dstadler/.local/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://query2.finance.yahoo.com/v6/finance/quoteSummary/ORCL?modules=financialData&modules=quoteType&modules=defaultKeyStatistics&modules=assetProfile&modules=summaryDetail&ssl=true
bot-unit commented 10 months ago

> @bot-unit That reminds me, does #1657 help anyone?

I don't known. We just need to fix url to use api v7,v8 and v10 and include crumb= in requests.

Crumb we can take something like this:

        self.session = requests.Session()
        response = self.session.get('https://guce.yahoo.com/consent', headers=self.user_agent_headers)
        soup = BeautifulSoup(response.content, 'html.parser')
        csrfTokenInput = soup.find('input', attrs={'name': 'csrfToken'})
        csrfToken = csrfTokenInput['value']
        sessionIdInput = soup.find('input', attrs={'name': 'sessionId'})
        sessionId = sessionIdInput['value']
        originalDoneUrl = 'https://finance.yahoo.com/'
        namespace = 'yahoo'
        data = {
            'agree': ['agree', 'agree'],
            'consentUUID': 'default',
            'sessionId': sessionId,
            'csrfToken': csrfToken,
            'originalDoneUrl': originalDoneUrl,
            'namespace': namespace,
        }
        self.session.post(f'https://consent.yahoo.com/v2/collectConsent?sessionId={sessionId}', data=data, headers=self.user_agent_headers)
        self.session.get(f'https://guce.yahoo.com/copyConsent?sessionId={sessionId}', headers=self.user_agent_headers)
        self.crumb = self.session.get('https://query2.finance.yahoo.com/v1/test/getcrumb', headers=self.user_agent_headers).text
ibart commented 10 months ago

I am not really into programming at this level but i think the answers is understanding why it works with an american IP...

bot-unit commented 10 months ago

I am not really into programming at this level but i think the answers is understanding why it works with an american IP...

Because Yahoo is banning not-American IP ˆ)

ibart commented 10 months ago

I am not really into programming at this level but i think the answers is understanding why it works with an american IP...

Because Yahoo is banning not-American IP ˆ)

Right now I am trying with this american VPN ip 143.244.44.169 and I am getting 404. =)

fredrik-corneliusson commented 10 months ago

I've observed different IP addresses for query2.finance.yahoo.com when running nslookup on AWS EC2 instances in the US and EU.

Using the following curl command works in the US but not in the EU: curl -v 'https://query2.finance.yahoo.com/v6/finance/quoteSummary/SPY?modules=financialData&modules=quoteType&modules=defaultKeyStatistics&modules=assetProfile&modules=summaryDetail&ssl=true' However, when I hardcode the IP address returned by the DNS lookup from the US instance, the request works outside of the US: curl -v -k -H "Host: query2.finance.yahoo.com" 'https://69.147.92.11/v6/finance/quoteSummary/SPY?modules=financialData&modules=quoteType&modules=defaultKeyStatistics&modules=assetProfile&modules=summaryDetail&ssl=true' Please note that this is not a long-term solution, as the IP address might change dynamically. It's just an observation that might help diagnose the issue.

lucas03 commented 10 months ago

hmm, interesting. What about this?

curl -v -k -H "Host: query2.finance.yahoo.com" 'https://e1.ycpi.vip.dca.yahoo.com/v6/finance/quoteSummary/SPY?modules=financialData&modules=quoteType&modules=defaultKeyStatistics&modules=assetProfile&modules=summaryDetail&ssl=true'
fredrik-corneliusson commented 10 months ago

I use yahooquery and for me hardcoding the IP for "query2.finance.yahoo.com" in /etc/hosts works as a temporary solution for my script. @lucas03 using the subdomain is probably better but is there a quick way to fix existing scripts with subdomain as for IP adresses?

elementace commented 10 months ago

I've observed different IP addresses for query2.finance.yahoo.com when running nslookup on AWS EC2 instances in the US and EU.

Using the following curl command works in the US but not in the EU: curl -v 'https://query2.finance.yahoo.com/v6/finance/quoteSummary/SPY?modules=financialData&modules=quoteType&modules=defaultKeyStatistics&modules=assetProfile&modules=summaryDetail&ssl=true' However, when I hardcode the IP address returned by the DNS lookup from the US instance, the request works outside of the US: curl -v -k -H "Host: query2.finance.yahoo.com" 'https://69.147.92.11/v6/finance/quoteSummary/SPY?modules=financialData&modules=quoteType&modules=defaultKeyStatistics&modules=assetProfile&modules=summaryDetail&ssl=true' Please note that this is not a long-term solution, as the IP address might change dynamically. It's just an observation that might help diagnose the issue.

I wonder if you're bypassing the location block by hitting a server directly. Maybe the location block is attached to the load-balancer somehow.

elementace commented 10 months ago

I am not really into programming at this level but i think the answers is understanding why it works with an american IP...

Because Yahoo is banning not-American IP ˆ)

Right now I am trying with this american VPN ip 143.244.44.169 and I am getting 404. =)

I'm using Browsec VPN. Using my Australian IP: 404 Using Browsec German VPN: 404 Using Browsec US VPN: I get results. Using Browsec US (west) VPN: 404.

ibart commented 10 months ago

I am not really into programming at this level but i think the answers is understanding why it works with an american IP...

Because Yahoo is banning not-American IP ˆ)

Right now I am trying with this american VPN ip 143.244.44.169 and I am getting 404. =)

I'm using Browsec VPN. Using my Australian IP: 404 Using Browsec German VPN: 404 Using Browsec US VPN: I get results. Using Browsec US (west) VPN: 404.

Yes, from a Western US' IP I got 404, but switching to a Californian one, as far as i am concerned, I got results yesterday.

dioni-dev commented 10 months ago

I am having the same problem, yahoo wants to deny for not US ip address?

james-stevens commented 10 months ago

hmm, interesting. What about this?

curl -v -k -H "Host: query2.finance.yahoo.com" 'https://e1.ycpi.vip.dca.yahoo.com/v6/finance/quoteSummary/SPY?modules=financialData&modules=quoteType&modules=defaultKeyStatistics&modules=assetProfile&modules=summaryDetail&ssl=true'

both this curl & the one with the IP work for me from the UK

jfmatth commented 10 months ago

I'm using PIA VPN and the only one that works for me is US Houston (191.96.67.107).

That works on my desktop, but my code is running on a server in the DMZ, so can't fix it with VPN.

codingwizzard4 commented 10 months ago

I live in the united states and even trying to use my local ip dosent work for me.

wilhelmagren commented 10 months ago

I might (unfortunately) be able to add some extra complexity to this issue. I'm running automated tests in my github workflows using yfinance, one test which tries and fetch information using the get_info() method on the yf.Ticker object.

This workflow is run on [ubuntu-latest, macos-latest, windows-latest] for Python3.10.x. Interestingly, the tests pass when running on ubuntu and macos, but failing when running on windows.

The request to the following url is getting a 404: https://query2.finance.yahoo.com/v6/finance/quoteSummary/ABB.ST?modules=financialData&modules=quoteType&modules=defaultKeyStatistics&modules=assetProfile&modules=summaryDetail&ssl=true

The runner version are the following:

os version image provisioner fails due to 404
Ubuntu 22.04.3 LTS 20231016.1.0 2.0.312.1 No
macOS 12.6.9 21G726 20230921.1 2.0.312.1 No
Microsoft Windows Server 2022 10.0.20348 Datacenter 20231023.1.0 2.0.312.1 Yes

Link to the entire failing test output due to 404.

Could it be that GitHub's ubuntu and macOS workers are hosted in the US, whereas the windows worker is hosted in the EU?

Looking forward to follow this issue and hopefully help narrow the problem down :)

//w

lucas03 commented 10 months ago

I tried to apply curl changes to yfinance, as a temporary hotfix for testing

https://github.com/ranaroussi/yfinance/compare/main...lucas03:yfinance:lukas/yahoo_domain

0zd3m1r commented 10 months ago

How can I solve the problem if I'm working from Spain?

Is there any free solution?

Temporary solution: sed -i '0 i 69.147.92.11 query2.finance.yahoo.com' /etc/hosts

ValueRaider commented 10 months ago

@lukas How did you find the e1.ycpi.vip.dca.yahoo.com subdomain?

lucas03 commented 10 months ago

"Reverse IP Lookup"

ms82494 commented 10 months ago

I think @zq99 is on to something. His approach works for me. After appending the crumb from a request to: https://query2.finance.yahoo.com/v1/test/getcrumb to this URL: https://query2.finance.yahoo.com/v10/finance/quoteSummary/AAPL?module=summaryDetail&ssl=true&crumb=<crumbfrompreviouscall> I get a valid result. Without having any modifications in /etc/hosts. I should say that I am in the US, but if I use the v6 API and an unmodified hosts file I still get 404 errors.

bagcivan commented 10 months ago

This link, which appears in the library's error, doesn't work https://query2.finance.yahoo.com/v6/finance/quoteSummary/MSFT?modules=financialData&modules=quoteType&modules=defaultKeyStatistics&modules=assetProfile&modules=summaryDetail&ssl=true

But it only works when query1,v8 and chart changes

https://query1.finance.yahoo.com/v8/finance/chart/MSFT?modules=financialData&modules=quoteType&modules=defaultKeyStatistics&modules=assetProfile&modules=summaryDetail&ssl=true

fredrik-corneliusson commented 10 months ago

Not an immediate solution to the problem, but roic.ai has API access for only $67 per month. https://roic.ai/pricing

This thread may not be the ideal place to discuss this, but I, for one, would like to have a contingency plan in place for when Yahoo decides to close their API, as I believe some of Yahoo's APIs have been deprecated for years. I’m okay with paying a reasonable amount for a robust data source. There are a variety of API providers available, and I imagine there are many potential pitfalls.

What’s your stance on this, @ranaroussi and @ValueRaider? Would it be okay to start a discussion on backup alternatives?

ValueRaider commented 10 months ago

@fredrik-corneliusson Discussing alternatives is fine just not in this thread, but probably already blogposts exist, I've seen a few. My feeling is Yahoo moving to subscription not closure. They can't close it, their website needs it.

jirisarri10 commented 10 months ago

Can someone tell me when this is going to be fixed? Is it dependent on Yahoo? Is it an issue with the API? Is it because Yahoo wants to charge for the service? Is the current state of Yfinance irreversible? Thank you!

ValueRaider commented 10 months ago

@jirisarri10 most questions answered by reading top post then #1084

ValueRaider commented 10 months ago

@bot-unit What symbol did you test crumbs on, and where are you? I am in Europe. https://github.com/ranaroussi/yfinance/issues/1729#issuecomment-1775575210 I get more fails than successes, either with your crumb fetch or mine #1657.

edit: retracting my investigation, smells like a bug

RudyNL commented 10 months ago

@ValueRaider Due to my experience with yahooquery you can try a different capitalization instead in case of failure amzn instead of AMZN. It seems Yahoo is changing the capitalization rules.

bot-unit commented 10 months ago

@bot-unit What symbol did you test crumbs on, and where are you? I am in Europe. #1729 (comment) I get more fails than successes, either with your crumb fetch or mine #1657. Symbol #1657 crumb bot-unit crumb MSFT works 401 Client Error: Unauthorized for url AMZN 401 Client Error: Unauthorized for url works AZN.L 401 Client Error: Unauthorized for url 401 Client Error: Unauthorized for url DBK.DE 401 Client Error: Unauthorized for url 401 Client Error: Unauthorized for url

I am in Europe and my fix is working for ALL symbols. With dot notated symbols, just do simple replace dot to '-': symbol_name = symbol_name.replace('.', '-')

bzynpb commented 10 months ago

Try running on Google colab it works for me but still it is not working on local environment. Bit weird 🤔

In my case, it's the exact opposite. It works locally but doesn't work on Colab

ValueRaider commented 10 months ago

@bot-unit I'm having more success now, maybe my caching was problem. How long can crumb be cached for? Takes 4 requests to get crumb, need to maximise crumb reuse. update: I've ported @bot-unit 's crumb fetch into #1657 and fixed reuse. Works great except _fetch_complementary() failing.

prabhakaran2812 commented 10 months ago

I've same issue in colab. When I try delete runtime and reconnect multiple times, it works. Its very random, should be based on IP I guess.

ricardodacosta commented 10 months ago

yesterday it was working fine with a US VPN, today is not. Locally it is not working either and am based in the US/EAST

prabhakaran2812 commented 10 months ago

yesterday it was working fine with a US VPN, today is not. Locally it is not working either and am based in the US/EAST

The page is genuinely under maintenance I guess https://query2.finance.yahoo.com/v6/finance/quoteSummary/AAPL?modules=financialData&modules=quoteType&modules=defaultKeyStatistics&modules=assetProfile&modules=summaryDetail&ssl=true

ibart commented 10 months ago

yesterday it was working fine with a US VPN, today is not. Locally it is not working either and am based in the US/EAST

The page is genuinely under maintenance I guess https://query2.finance.yahoo.com/v6/finance/quoteSummary/AAPL?modules=financialData&modules=quoteType&modules=defaultKeyStatistics&modules=assetProfile&modules=summaryDetail&ssl=true

That message is the same as 10 days ago.

patfla commented 10 months ago

https://imgur.com/a/YVJNyGQ

For a while VPNing outside the US worked. No longer. I'd guess that they're not going to be right back.