Closed ssvetli closed 9 months ago
The yahoo finance API looks down.
For reference some of the methods still working: .history .fast_info .news .earnings_dates .history_metadata .option_chain
Try running on Google colab it works for me but still it is not working on local environment. Bit weird 🤔
Firefox -> https://finance.yahoo.com/quote/AMZN -> F12 -> Network tab -> F5 -> compare requests vs yfinance That's essence of how to find fix.
same issue but when i open USA vpn it works
Try running on Google colab it works for me but still it is not working on local environment. Bit weird 🤔
i cant use my code on colab, i hope in resolution. I have error also in pyfinvis, its a strange period:)
May be it will help to switch to v7 or v10 api: https://query2.finance.yahoo.com/v1/test/getcrumb
@bot-unit That reminds me, does #1657 help anyone?
yahooquery has the same problems
same issue but when i open USA vpn it works
You save my life.
USA VPN does work!
I can confirm using USA VPN fixes the issue. Thanks for sharing @MrH32 !
In my case using threads=False partially solves it
In my case using threads=False partially solves it
can you explain more
In my case using threads=False partially solves it
can you explain more It is a yfinace option(yd.download(share, threads=False))
for me it is unrelated to threads
, download
seems to work always, others not:
#!/bin/python3
import yfinance as yf
print('Quotes:\n')
quotes = yf.download('ORCL', period = '1d')
print(quotes)
print('\n\nQuote:\n')
quote = yf.Ticker('ORCL')
print(quote.info['currentPrice'])
Quotes:
[*********************100%%**********************] 1 of 1 completed
Open High Low Close Adj Close Volume
Date
2023-10-20 106.989998 107.035004 100.660004 101.849998 101.849998 21718467
Quote:
Traceback (most recent call last):
File "/opt/svn/DynatraceFinance/./test.py", line 11, in <module>
print(quote.info['currentPrice'])
File "/home/dstadler/.local/lib/python3.10/site-packages/yfinance/ticker.py", line 142, in info
return self.get_info()
File "/home/dstadler/.local/lib/python3.10/site-packages/yfinance/base.py", line 1736, in get_info
data = self._quote.info
File "/home/dstadler/.local/lib/python3.10/site-packages/yfinance/scrapers/quote.py", line 571, in info
self._fetch(self.proxy)
File "/home/dstadler/.local/lib/python3.10/site-packages/yfinance/scrapers/quote.py", line 600, in _fetch
result = self._data.get_raw_json(
File "/home/dstadler/.local/lib/python3.10/site-packages/yfinance/data.py", line 75, in get_raw_json
response.raise_for_status()
File "/home/dstadler/.local/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://query2.finance.yahoo.com/v6/finance/quoteSummary/ORCL?modules=financialData&modules=quoteType&modules=defaultKeyStatistics&modules=assetProfile&modules=summaryDetail&ssl=true
>
@bot-unit That reminds me, does #1657 help anyone?
I don't known. We just need to fix url to use api v7,v8 and v10 and include crumb= in requests.
Crumb we can take something like this:
self.session = requests.Session()
response = self.session.get('https://guce.yahoo.com/consent', headers=self.user_agent_headers)
soup = BeautifulSoup(response.content, 'html.parser')
csrfTokenInput = soup.find('input', attrs={'name': 'csrfToken'})
csrfToken = csrfTokenInput['value']
sessionIdInput = soup.find('input', attrs={'name': 'sessionId'})
sessionId = sessionIdInput['value']
originalDoneUrl = 'https://finance.yahoo.com/'
namespace = 'yahoo'
data = {
'agree': ['agree', 'agree'],
'consentUUID': 'default',
'sessionId': sessionId,
'csrfToken': csrfToken,
'originalDoneUrl': originalDoneUrl,
'namespace': namespace,
}
self.session.post(f'https://consent.yahoo.com/v2/collectConsent?sessionId={sessionId}', data=data, headers=self.user_agent_headers)
self.session.get(f'https://guce.yahoo.com/copyConsent?sessionId={sessionId}', headers=self.user_agent_headers)
self.crumb = self.session.get('https://query2.finance.yahoo.com/v1/test/getcrumb', headers=self.user_agent_headers).text
I am not really into programming at this level but i think the answers is understanding why it works with an american IP...
I am not really into programming at this level but i think the answers is understanding why it works with an american IP...
Because Yahoo is banning not-American IP ˆ)
I am not really into programming at this level but i think the answers is understanding why it works with an american IP...
Because Yahoo is banning not-American IP ˆ)
Right now I am trying with this american VPN ip 143.244.44.169 and I am getting 404. =)
I've observed different IP addresses for query2.finance.yahoo.com when running nslookup on AWS EC2 instances in the US and EU.
Using the following curl command works in the US but not in the EU:
curl -v 'https://query2.finance.yahoo.com/v6/finance/quoteSummary/SPY?modules=financialData&modules=quoteType&modules=defaultKeyStatistics&modules=assetProfile&modules=summaryDetail&ssl=true'
However, when I hardcode the IP address returned by the DNS lookup from the US instance, the request works outside of the US:
curl -v -k -H "Host: query2.finance.yahoo.com" 'https://69.147.92.11/v6/finance/quoteSummary/SPY?modules=financialData&modules=quoteType&modules=defaultKeyStatistics&modules=assetProfile&modules=summaryDetail&ssl=true'
Please note that this is not a long-term solution, as the IP address might change dynamically. It's just an observation that might help diagnose the issue.
hmm, interesting. What about this?
curl -v -k -H "Host: query2.finance.yahoo.com" 'https://e1.ycpi.vip.dca.yahoo.com/v6/finance/quoteSummary/SPY?modules=financialData&modules=quoteType&modules=defaultKeyStatistics&modules=assetProfile&modules=summaryDetail&ssl=true'
I use yahooquery and for me hardcoding the IP for "query2.finance.yahoo.com" in /etc/hosts
works as a temporary solution for my script.
@lucas03 using the subdomain is probably better but is there a quick way to fix existing scripts with subdomain as for IP adresses?
I've observed different IP addresses for query2.finance.yahoo.com when running nslookup on AWS EC2 instances in the US and EU.
Using the following curl command works in the US but not in the EU:
curl -v 'https://query2.finance.yahoo.com/v6/finance/quoteSummary/SPY?modules=financialData&modules=quoteType&modules=defaultKeyStatistics&modules=assetProfile&modules=summaryDetail&ssl=true'
However, when I hardcode the IP address returned by the DNS lookup from the US instance, the request works outside of the US:curl -v -k -H "Host: query2.finance.yahoo.com" 'https://69.147.92.11/v6/finance/quoteSummary/SPY?modules=financialData&modules=quoteType&modules=defaultKeyStatistics&modules=assetProfile&modules=summaryDetail&ssl=true'
Please note that this is not a long-term solution, as the IP address might change dynamically. It's just an observation that might help diagnose the issue.
I wonder if you're bypassing the location block by hitting a server directly. Maybe the location block is attached to the load-balancer somehow.
I am not really into programming at this level but i think the answers is understanding why it works with an american IP...
Because Yahoo is banning not-American IP ˆ)
Right now I am trying with this american VPN ip 143.244.44.169 and I am getting 404. =)
I'm using Browsec VPN. Using my Australian IP: 404 Using Browsec German VPN: 404 Using Browsec US VPN: I get results. Using Browsec US (west) VPN: 404.
I am not really into programming at this level but i think the answers is understanding why it works with an american IP...
Because Yahoo is banning not-American IP ˆ)
Right now I am trying with this american VPN ip 143.244.44.169 and I am getting 404. =)
I'm using Browsec VPN. Using my Australian IP: 404 Using Browsec German VPN: 404 Using Browsec US VPN: I get results. Using Browsec US (west) VPN: 404.
Yes, from a Western US' IP I got 404, but switching to a Californian one, as far as i am concerned, I got results yesterday.
I am having the same problem, yahoo wants to deny for not US ip address?
hmm, interesting. What about this?
curl -v -k -H "Host: query2.finance.yahoo.com" 'https://e1.ycpi.vip.dca.yahoo.com/v6/finance/quoteSummary/SPY?modules=financialData&modules=quoteType&modules=defaultKeyStatistics&modules=assetProfile&modules=summaryDetail&ssl=true'
both this curl
& the one with the IP work for me from the UK
I'm using PIA VPN and the only one that works for me is US Houston (191.96.67.107).
That works on my desktop, but my code is running on a server in the DMZ, so can't fix it with VPN.
I live in the united states and even trying to use my local ip dosent work for me.
I might (unfortunately) be able to add some extra complexity to this issue. I'm running automated tests in my github workflows using yfinance, one test which tries and fetch information using the get_info()
method on the yf.Ticker
object.
This workflow is run on [ubuntu-latest, macos-latest, windows-latest]
for Python3.10.x
. Interestingly, the tests pass when running on ubuntu and macos, but failing when running on windows.
The request to the following url is getting a 404: https://query2.finance.yahoo.com/v6/finance/quoteSummary/ABB.ST?modules=financialData&modules=quoteType&modules=defaultKeyStatistics&modules=assetProfile&modules=summaryDetail&ssl=true
The runner version are the following:
os | version | image provisioner | fails due to 404 |
---|---|---|---|
Ubuntu 22.04.3 LTS | 20231016.1.0 | 2.0.312.1 | No |
macOS 12.6.9 21G726 | 20230921.1 | 2.0.312.1 | No |
Microsoft Windows Server 2022 10.0.20348 Datacenter | 20231023.1.0 | 2.0.312.1 | Yes |
Link to the entire failing test output due to 404.
Could it be that GitHub's ubuntu and macOS workers are hosted in the US, whereas the windows worker is hosted in the EU?
Looking forward to follow this issue and hopefully help narrow the problem down :)
//w
I tried to apply curl changes to yfinance, as a temporary hotfix for testing
https://github.com/ranaroussi/yfinance/compare/main...lucas03:yfinance:lukas/yahoo_domain
How can I solve the problem if I'm working from Spain?
Is there any free solution?
Temporary solution: sed -i '0 i 69.147.92.11 query2.finance.yahoo.com' /etc/hosts
@lukas How did you find the e1.ycpi.vip.dca.yahoo.com
subdomain?
"Reverse IP Lookup"
I think @zq99 is on to something. His approach works for me. After appending the crumb from a request to:
https://query2.finance.yahoo.com/v1/test/getcrumb
to this URL:
https://query2.finance.yahoo.com/v10/finance/quoteSummary/AAPL?module=summaryDetail&ssl=true&crumb=<crumbfrompreviouscall>
I get a valid result. Without having any modifications in /etc/hosts
. I should say that I am in the US, but if I use the v6 API and an unmodified hosts file I still get 404 errors.
This link, which appears in the library's error, doesn't work
https://query2.finance.yahoo.com/v6/finance/quoteSummary/MSFT?modules=financialData&modules=quoteType&modules=defaultKeyStatistics&modules=assetProfile&modules=summaryDetail&ssl=true
But it only works when query1,v8 and chart changes
https://query1.finance.yahoo.com/v8/finance/chart/MSFT?modules=financialData&modules=quoteType&modules=defaultKeyStatistics&modules=assetProfile&modules=summaryDetail&ssl=true
Not an immediate solution to the problem, but roic.ai has API access for only $67 per month. https://roic.ai/pricing
This thread may not be the ideal place to discuss this, but I, for one, would like to have a contingency plan in place for when Yahoo decides to close their API, as I believe some of Yahoo's APIs have been deprecated for years. I’m okay with paying a reasonable amount for a robust data source. There are a variety of API providers available, and I imagine there are many potential pitfalls.
What’s your stance on this, @ranaroussi and @ValueRaider? Would it be okay to start a discussion on backup alternatives?
@fredrik-corneliusson Discussing alternatives is fine just not in this thread, but probably already blogposts exist, I've seen a few. My feeling is Yahoo moving to subscription not closure. They can't close it, their website needs it.
Can someone tell me when this is going to be fixed? Is it dependent on Yahoo? Is it an issue with the API? Is it because Yahoo wants to charge for the service? Is the current state of Yfinance irreversible? Thank you!
@jirisarri10 most questions answered by reading top post then #1084
@bot-unit What symbol did you test crumbs on, and where are you? I am in Europe. https://github.com/ranaroussi/yfinance/issues/1729#issuecomment-1775575210 I get more fails than successes, either with your crumb fetch or mine #1657.
edit: retracting my investigation, smells like a bug
@ValueRaider Due to my experience with yahooquery
you can try a different capitalization instead in case of failure amzn
instead of AMZN
. It seems Yahoo is changing the capitalization rules.
@bot-unit What symbol did you test crumbs on, and where are you? I am in Europe. #1729 (comment) I get more fails than successes, either with your crumb fetch or mine #1657. Symbol #1657 crumb bot-unit crumb MSFT works 401 Client Error: Unauthorized for url AMZN 401 Client Error: Unauthorized for url works AZN.L 401 Client Error: Unauthorized for url 401 Client Error: Unauthorized for url DBK.DE 401 Client Error: Unauthorized for url 401 Client Error: Unauthorized for url
I am in Europe and my fix is working for ALL symbols. With dot notated symbols, just do simple replace dot to '-': symbol_name = symbol_name.replace('.', '-')
Try running on Google colab it works for me but still it is not working on local environment. Bit weird 🤔
In my case, it's the exact opposite. It works locally but doesn't work on Colab
@bot-unit I'm having more success now, maybe my caching was problem. How long can crumb be cached for? Takes 4 requests to get crumb, need to maximise crumb reuse.
update:
I've ported @bot-unit 's crumb fetch into #1657 and fixed reuse. Works great except _fetch_complementary()
failing.
I've same issue in colab. When I try delete runtime and reconnect multiple times, it works. Its very random, should be based on IP I guess.
yesterday it was working fine with a US VPN, today is not. Locally it is not working either and am based in the US/EAST
yesterday it was working fine with a US VPN, today is not. Locally it is not working either and am based in the US/EAST
The page is genuinely under maintenance I guess https://query2.finance.yahoo.com/v6/finance/quoteSummary/AAPL?modules=financialData&modules=quoteType&modules=defaultKeyStatistics&modules=assetProfile&modules=summaryDetail&ssl=true
yesterday it was working fine with a US VPN, today is not. Locally it is not working either and am based in the US/EAST
The page is genuinely under maintenance I guess https://query2.finance.yahoo.com/v6/finance/quoteSummary/AAPL?modules=financialData&modules=quoteType&modules=defaultKeyStatistics&modules=assetProfile&modules=summaryDetail&ssl=true
That message is the same as 10 days ago.
For a while VPNing outside the US worked. No longer. I'd guess that they're not going to be right back.
@ValueRaider hijacking top post
Doesn't affect USATicker.fast_info
has subset ofTicker.info
USA VPN is a workaround https://github.com/ranaroussi/yfinance/issues/1729#issuecomment-1773032475changing endpoint might help https://github.com/ranaroussi/yfinance/issues/1729#issuecomment-1776096509Describe bug
Call to .info returns 404 error on all tickers. Example url: https://query2.finance.yahoo.com/v6/finance/quoteSummary/SPY?modules=financialData&modules=quoteType&modules=defaultKeyStatistics&modules=assetProfile&modules=summaryDetail&ssl=true Using browsers on multiple connections/devices, yahoo output "Will be right back... Our engineers are working quickly to resolve the issue."
Data could have moved to new link
Simple code that reproduces your problem
import yfinance as yf yf_ticker=yf.Ticker("SPY") yf_ticker.info
Debug log
HTTPError: 404 Client Error: Not Found for url: https://query2.finance.yahoo.com/v6/finance/quoteSummary/SPY?modules=financialData&modules=quoteType&modules=defaultKeyStatistics&modules=assetProfile&modules=summaryDetail&ssl=true
Bad data proof
No response
yfinance
version0.2.31
Python version
No response
Operating system
No response