search

ranaroussi / yfinance

Download market data from Yahoo! Finance's API
https://aroussi.com/post/python-yahoo-finance
Apache License 2.0
14.89k stars 2.44k forks source link

There is json cannot load error consistently #2135

Closed solomonszh closed 2 days ago

solomonszh commented 4 days ago

Describe bug

There is json cannot load error consistently

Simple code that reproduces your problem

yf.download()

Debug log

Initialized DB. [ 0% ] 5 of 8565 completedFailed to get ticker 'HWEL' reason: Expecting value: line 1 column 1 (char 0) [ 0% ] 18 of 8565 completedFailed to get ticker 'ELIQ' reason: Expecting value: line 1 column 1 (char 0) [ 0% ] 30 of 8565 completedFailed to get ticker 'PRTK' reason: Expecting value: line 1 column 1 (char 0) [ 1% ] 73 of 8565 completedFailed to get ticker 'IPVF' reason: Expecting value: line 1 column 1 (char 0) [ 1% ] 88 of 8565 completedFailed to get ticker 'TWNK' reason: Expecting value: line 1 column 1 (char 0) [ 1% ] 94 of 8565 completedFailed to get ticker 'CSTA' reason: Expecting value: line 1 column 1 (char 0) [ 2% ] 137 of 8565 completedFailed to get ticker 'INPX' reason: Expecting value: line 1 column 1 (char 0) [ 2% ] 138 of 8565 completedFailed to get ticker 'SOVO' reason: Expecting value: line 1 column 1 (char 0) [ 2% ] 171 of 8565 completedFailed to get ticker 'ETAO' reason: Expecting value: line 1 column 1 (char 0) [ 2% ] 181 of 8565 completedFailed to get ticker 'AMEH' reason: Expecting value: line 1 column 1 (char 0) [ 3% ] 229 of 8565 completedFailed to get ticker 'ALYA' reason: Expecting value: line 1 column 1 (char 0) [ 3% ] 230 of 8565 completedFailed to get ticker 'CCLP' reason: Expecting value: line 1 column 1 (char 0) [ 3% ] 238 of 8565 completedFailed to get ticker 'UTAA' reason: Expecting value: line 1 column 1 (char 0) [ 3% ] 292 of 8565 completedFailed to get ticker 'SSU' reason: Expecting value: line 1 column 1 (char 0) [ 4% ] 313 of 8565 completedFailed to get ticker 'MOXC' reason: Expecting value: line 1 column 1 (char 0) [ 4% ] 347 of 8565 completedFailed to get ticker 'MDWT' reason: Expecting value: line 1 column 1 (char 0) [ 5% ] 421 of 8565 completedFailed to get ticker 'SCTL' reason: Expecting value: line 1 column 1 (char 0) [ 5% ] 459 of 8565 completedFailed to get ticker 'BRDS' reason: Expecting value: line 1 column 1 (char 0) [ 5% ] 463 of 8565 completedFailed to get ticker 'THRX' reason: Expecting value: line 1 column 1 (char 0) [ 5% ] 471 of 8565 completedFailed to get ticker 'CCVI' reason: Expecting value: line 1 column 1 (char 0) [ 6% ] 503 of 8565 completedFailed to get ticker 'HARP' reason: Expecting value: line 1 column 1 (char 0) [ 7% ] 568 of 8565 completedFailed to get ticker 'PRPC' reason: Expecting value: line 1 column 1 (char 0) [ 7% ] 578 of 8565 completedFailed to get ticker 'HHLA' reason: Expecting value: line 1 column 1 (char 0) [ 7% ] 581 of 8565 completedFailed to get ticker 'WE' reason: Expecting value: line 1 column 1 (char 0) [ 8% ] 671 of 8565 completedFailed to get ticker 'BVH' reason: Expecting value: line 1 column 1 (char 0) [ 8% ] 676 of 8565 completedFailed to get ticker 'NM' reason: Expecting value: line 1 column 1 (char 0) [ 8% ] 680 of 8565 completedFailed to get ticker 'CORR' reason: Expecting value: line 1 column 1 (char 0) [ 8% ] 681 of 8565 completedFailed to get ticker 'GSD' reason: Expecting value: line 1 column 1 (char 0) [ 8% ] 691 of 8565 completedFailed to get ticker 'PEAK' reason: Expecting value: line 1 column 1 (char 0) [ 8% ] 723 of 8565 completedFailed to get ticker 'IMPL' reason: Expecting value: line 1 column 1 (char 0) [ 9% ] 743 of 8565 completedFailed to get ticker 'PACI' reason: Expecting value: line 1 column 1 (char 0) [ 9% ] 772 of 8565 completedFailed to get ticker 'APAC' reason: Expecting value: line 1 column 1 (char 0) [ 9% ] 789 of 8565 completedFailed to get ticker 'PTHR' reason: Expecting value: line 1 column 1 (char 0) [ 9% ] 810 of 8565 completedFailed to get ticker 'AAIC' reason: Expecting value: line 1 column 1 (char 0) [ 10% ] 870 of 8565 completedFailed to get ticker 'UPTD' reason: Expecting value: line 1 column 1 (char 0) [ 10% ] 881 of 8565 completedFailed to get ticker 'NFNT' reason: Expecting value: line 1 column 1 (char 0) [ 11% ] 914 of 8565 completedFailed to get ticker 'DWAC' reason: Expecting value: line 1 column 1 (char 0) [ 11% ] 916 of 8565 completedFailed to get ticker 'TSP' reason: Expecting value: line 1 column 1 (char 0) [ 11% ] 928 of 8565 completedFailed to get ticker 'CIR' reason: Expecting value: line 1 column 1 (char 0) [ 11% ] 933 of 8565 completedFailed to get ticker 'EAC' reason: Expecting value: line 1 column 1 (char 0) [* 11% ] 963 of 8565 completedFailed to get ticker 'OXSQL' reason: Expecting value: line 1 column 1 (char 0) [** 12% ] 1033 of 8565 completed DEBUG Entering download() DEBUG Disabling multithreading because DEBUG logging enabled DEBUG Entering history() DEBUG Entering history() DEBUG AAPL: Yahoo GET parameters: {'range': '1mo', 'interval': '1d', 'includePrePost': False, 'events': 'div,splits,capitalGains'} DEBUG Entering get() DEBUG Entering _make_request() DEBUG url=https://query2.finance.yahoo.com/v8/finance/chart/AAPL DEBUG params={'range': '1mo', 'interval': '1d', 'includePrePost': False, 'events': 'div,splits,capitalGains'} DEBUG Entering _get_cookie_and_crumb() DEBUG cookie_mode = 'basic' DEBUG Entering _get_cookie_and_crumb_basic() DEBUG reusing cookie DEBUG reusing crumb DEBUG Exiting _get_cookie_and_crumb_basic() DEBUG Exiting _get_cookie_and_crumb() DEBUG response code=429 DEBUG toggling cookie strategy basic -> csrf DEBUG Entering _get_cookie_and_crumb() DEBUG cookie_mode = 'csrf' DEBUG Entering _get_crumb_csrf() DEBUG Failed to find "csrfToken" in response DEBUG Exiting _get_crumb_csrf() DEBUG toggling cookie strategy csrf -> basic DEBUG Entering _get_cookie_and_crumb_basic() DEBUG loaded persistent cookie DEBUG reusing cookie DEBUG crumb = 'Edge: Too Many Requests' DEBUG Exiting _get_cookie_and_crumb_basic() DEBUG Exiting _get_cookie_and_crumb() DEBUG response code=429

Bad data proof

[ 8% ] 680 of 8565 completedFailed to get ticker 'CORR' reason: Expecting value: line 1 column 1 (char 0) [ 8% ] 681 of 8565 completedFailed to get ticker 'GSD' reason: Expecting value: line 1 column 1 (char 0) [ 8% ] 691 of 8565 completedFailed to get ticker 'PEAK' reason: Expecting value: line 1 column 1 (char 0) [ 8% ] 723 of 8565 completedFailed to get ticker 'IMPL' reason: Expecting value: line 1 column 1 (char 0) [ 9% ] 743 of 8565 completedFailed to get ticker 'PACI' reason: Expecting value: line 1 column 1 (char 0) [ 9% ] 772 of 8565 completedFailed to get ticker 'APAC' reason: Expecting value: line 1 column 1 (char 0) [ 9% ] 789 of 8565 completedFailed to get ticker 'PTHR' reason: Expecting value: line 1 column 1 (char 0) [ 9% ] 810 of 8565 completedFailed to get ticker 'AAIC' reason: Expecting value: line 1 column 1 (char 0) [ 10% ] 870 of 8565 completedFailed to get ticker 'UPTD' reason: Expecting value: line 1 column 1 (char 0) [ 10% ] 881 of 8565 completedFailed to get ticker 'NFNT' reason: Expecting value: line 1 column 1 (char 0) [ 11% ] 914 of 8565 completedFailed to get ticker 'DWAC' reason: Expecting value: line 1 column 1 (char 0) [ 11% ] 916 of 8565 completedFailed to get ticker 'TSP' reason: Expecting value: line 1 column 1 (char 0) [ 11% ] 928 of 8565 completedFailed to get ticker 'CIR' reason: Expecting value: line 1 column 1 (char 0) [ 11% ] 933 of 8565 completedFailed to get ticker 'EAC' reason: Expecting value: line 1 column 1 (char 0) [***** 11% ] 963 of 8565 completedFailed to get ticker 'OXSQL' reason: Expecting value: line 1 column 1 (char 0)

yfinance version

latest

Python version

No response

Operating system

No response

riverland01 commented 4 days ago

did you bulk download? if so maybe yahoo finance is blocking you. Try to parse multiple tickers in one api call and wait a bit to see.

solomonszh commented 4 days ago

yup why is it blocking there is JSON error

solomonszh commented 4 days ago

even yf.download('AAPL') could not work

riverland01 commented 3 days ago

even yf.download('AAPL') could not work

It works perfectly for me with yfinance 0.2.49. I think you need to wait a bit before resuming the download. Try not to use a loop to download individual ticker. I think if you are rate limited by yahoo, it may be feeding the api call something else than what yfinance expects.

MartinGuimera commented 3 days ago

I'm having the exact same problem:

['J']: %ticker%SONDecodeError('Expecting value: line 1 column 1 (char 0)') ['SO']: J%ticker%NDecodeError('Expecting value: line 1 column 1 (char 0)') ['O']: JS%ticker%NDecodeError('Expecting value: line 1 column 1 (char 0)')

and i do use bulk download to download the data of around 1500 tickers but never had this problem, been running this code several times a day for more than a year now. I'm in 0.2.37. I tried in the 0.2.49 but dind't work either. 

            data = yf.download(
                " ".join(all_tickers),
                start=start_date_str,
                end=end_date_close_str,
                interval="1d",
                group_by="tickers",
                threads=8,
                progress=False,
            )

Is there documentation of how many api calls we can do or how much can we download?

SphereSK commented 3 days ago

Describe bug

There is json cannot load error consistently

Simple code that reproduces your problem

yf.download()

Debug log

Initialized DB. [ 0% ] 5 of 8565 completedFailed to get ticker 'HWEL' reason: Expecting value: line 1 column 1 (char 0) [ 0% ] 18 of 8565 completedFailed to get ticker 'ELIQ' reason: Expecting value: line 1 column 1 (char 0) [ 0% ] 30 of 8565 completedFailed to get ticker 'PRTK' reason: Expecting value: line 1 column 1 (char 0) [ 1% ] 73 of 8565 completedFailed to get ticker 'IPVF' reason: Expecting value: line 1 column 1 (char 0) [ 1% ] 88 of 8565 completedFailed to get ticker 'TWNK' reason: Expecting value: line 1 column 1 (char 0) [ 1% ] 94 of 8565 completedFailed to get ticker 'CSTA' reason: Expecting value: line 1 column 1 (char 0) [ 2% ] 137 of 8565 completedFailed to get ticker 'INPX' reason: Expecting value: line 1 column 1 (char 0) [ 2% ] 138 of 8565 completedFailed to get ticker 'SOVO' reason: Expecting value: line 1 column 1 (char 0) [ 2% ] 171 of 8565 completedFailed to get ticker 'ETAO' reason: Expecting value: line 1 column 1 (char 0) [ 2% ] 181 of 8565 completedFailed to get ticker 'AMEH' reason: Expecting value: line 1 column 1 (char 0) [ 3% ] 229 of 8565 completedFailed to get ticker 'ALYA' reason: Expecting value: line 1 column 1 (char 0) [ 3% ] 230 of 8565 completedFailed to get ticker 'CCLP' reason: Expecting value: line 1 column 1 (char 0) [ 3% ] 238 of 8565 completedFailed to get ticker 'UTAA' reason: Expecting value: line 1 column 1 (char 0) [ 3% ] 292 of 8565 completedFailed to get ticker 'SSU' reason: Expecting value: line 1 column 1 (char 0) [ 4% ] 313 of 8565 completedFailed to get ticker 'MOXC' reason: Expecting value: line 1 column 1 (char 0) [ 4% ] 347 of 8565 completedFailed to get ticker 'MDWT' reason: Expecting value: line 1 column 1 (char 0) [ 5% ] 421 of 8565 completedFailed to get ticker 'SCTL' reason: Expecting value: line 1 column 1 (char 0) [ 5% ] 459 of 8565 completedFailed to get ticker 'BRDS' reason: Expecting value: line 1 column 1 (char 0) [ 5% ] 463 of 8565 completedFailed to get ticker 'THRX' reason: Expecting value: line 1 column 1 (char 0) [ 5% ] 471 of 8565 completedFailed to get ticker 'CCVI' reason: Expecting value: line 1 column 1 (char 0) [ 6% ] 503 of 8565 completedFailed to get ticker 'HARP' reason: Expecting value: line 1 column 1 (char 0) [ 7% ] 568 of 8565 completedFailed to get ticker 'PRPC' reason: Expecting value: line 1 column 1 (char 0) [ 7% ] 578 of 8565 completedFailed to get ticker 'HHLA' reason: Expecting value: line 1 column 1 (char 0) [ 7% ] 581 of 8565 completedFailed to get ticker 'WE' reason: Expecting value: line 1 column 1 (char 0) [ 8% ] 671 of 8565 completedFailed to get ticker 'BVH' reason: Expecting value: line 1 column 1 (char 0) [ 8% ] 676 of 8565 completedFailed to get ticker 'NM' reason: Expecting value: line 1 column 1 (char 0) [ 8% ] 680 of 8565 completedFailed to get ticker 'CORR' reason: Expecting value: line 1 column 1 (char 0) [ 8% ] 681 of 8565 completedFailed to get ticker 'GSD' reason: Expecting value: line 1 column 1 (char 0) [ 8% ] 691 of 8565 completedFailed to get ticker 'PEAK' reason: Expecting value: line 1 column 1 (char 0) [ 8% ] 723 of 8565 completedFailed to get ticker 'IMPL' reason: Expecting value: line 1 column 1 (char 0) [ 9% ] 743 of 8565 completedFailed to get ticker 'PACI' reason: Expecting value: line 1 column 1 (char 0) [ 9% ] 772 of 8565 completedFailed to get ticker 'APAC' reason: Expecting value: line 1 column 1 (char 0) [ 9% ] 789 of 8565 completedFailed to get ticker 'PTHR' reason: Expecting value: line 1 column 1 (char 0) [ 9% ] 810 of 8565 completedFailed to get ticker 'AAIC' reason: Expecting value: line 1 column 1 (char 0) [ 10% ] 870 of 8565 completedFailed to get ticker 'UPTD' reason: Expecting value: line 1 column 1 (char 0) [ 10% ] 881 of 8565 completedFailed to get ticker 'NFNT' reason: Expecting value: line 1 column 1 (char 0) [ 11% ] 914 of 8565 completedFailed to get ticker 'DWAC' reason: Expecting value: line 1 column 1 (char 0) [ 11% ] 916 of 8565 completedFailed to get ticker 'TSP' reason: Expecting value: line 1 column 1 (char 0) [ 11% ] 928 of 8565 completedFailed to get ticker 'CIR' reason: Expecting value: line 1 column 1 (char 0) [ 11% ] 933 of 8565 completedFailed to get ticker 'EAC' reason: Expecting value: line 1 column 1 (char 0) [* 11% ] 963 of 8565 completedFailed to get ticker 'OXSQL' reason: Expecting value: line 1 column 1 (char 0) [** 12% ] 1033 of 8565 completed DEBUG Entering download() DEBUG Disabling multithreading because DEBUG logging enabled DEBUG Entering history() DEBUG Entering history() DEBUG AAPL: Yahoo GET parameters: {'range': '1mo', 'interval': '1d', 'includePrePost': False, 'events': 'div,splits,capitalGains'} DEBUG Entering get() DEBUG Entering _make_request() DEBUG url=https://query2.finance.yahoo.com/v8/finance/chart/AAPL DEBUG params={'range': '1mo', 'interval': '1d', 'includePrePost': False, 'events': 'div,splits,capitalGains'} DEBUG Entering _get_cookie_and_crumb() DEBUG cookie_mode = 'basic' DEBUG Entering _get_cookie_and_crumb_basic() DEBUG reusing cookie DEBUG reusing crumb DEBUG Exiting _get_cookie_and_crumb_basic() DEBUG Exiting _get_cookie_and_crumb() DEBUG response code=429 DEBUG toggling cookie strategy basic -> csrf DEBUG Entering _get_cookie_and_crumb() DEBUG cookie_mode = 'csrf' DEBUG Entering _get_crumb_csrf() DEBUG Failed to find "csrfToken" in response DEBUG Exiting _get_crumb_csrf() DEBUG toggling cookie strategy csrf -> basic DEBUG Entering _get_cookie_and_crumb_basic() DEBUG loaded persistent cookie DEBUG reusing cookie DEBUG crumb = 'Edge: Too Many Requests' DEBUG Exiting _get_cookie_and_crumb_basic() DEBUG Exiting _get_cookie_and_crumb() DEBUG response code=429

Bad data proof

[ 8% ] 680 of 8565 completedFailed to get ticker 'CORR' reason: Expecting value: line 1 column 1 (char 0) [ 8% ] 681 of 8565 completedFailed to get ticker 'GSD' reason: Expecting value: line 1 column 1 (char 0) [ 8% ] 691 of 8565 completedFailed to get ticker 'PEAK' reason: Expecting value: line 1 column 1 (char 0) [ 8% ] 723 of 8565 completedFailed to get ticker 'IMPL' reason: Expecting value: line 1 column 1 (char 0) [ 9% ] 743 of 8565 completedFailed to get ticker 'PACI' reason: Expecting value: line 1 column 1 (char 0) [ 9% ] 772 of 8565 completedFailed to get ticker 'APAC' reason: Expecting value: line 1 column 1 (char 0) [ 9% ] 789 of 8565 completedFailed to get ticker 'PTHR' reason: Expecting value: line 1 column 1 (char 0) [ 9% ] 810 of 8565 completedFailed to get ticker 'AAIC' reason: Expecting value: line 1 column 1 (char 0) [ 10% ] 870 of 8565 completedFailed to get ticker 'UPTD' reason: Expecting value: line 1 column 1 (char 0) [ 10% ] 881 of 8565 completedFailed to get ticker 'NFNT' reason: Expecting value: line 1 column 1 (char 0) [ 11% ] 914 of 8565 completedFailed to get ticker 'DWAC' reason: Expecting value: line 1 column 1 (char 0) [ 11% ] 916 of 8565 completedFailed to get ticker 'TSP' reason: Expecting value: line 1 column 1 (char 0) [ 11% ] 928 of 8565 completedFailed to get ticker 'CIR' reason: Expecting value: line 1 column 1 (char 0) [ 11% ] 933 of 8565 completedFailed to get ticker 'EAC' reason: Expecting value: line 1 column 1 (char 0) [***** 11% ] 963 of 8565 completedFailed to get ticker 'OXSQL' reason: Expecting value: line 1 column 1 (char 0)

yfinance version

latest

Python version

No response

Operating system

No response

SphereSK commented 3 days ago

same for me. It works yesterday without problem.

solomonszh commented 3 days ago

Basically, different users have different experience

ValueRaider commented 2 days ago

DEBUG response code=429

duplicate of #2125