All Sensors stopped working

[LuckyTriple7]

HA 2023.4.6 and HA 2023.5.1

All Sensors stopped working since yesterday: Error Log:

2023-05-05 08:32:53.184 ERROR (MainThread) [custom_components.yahoofinance.coordinator] Received status 401 (error=Unauthorized) for https://query1.finance.yahoo.com/v7/finance/quote?symbols=EBK.DE,O2D.DE,LHA.DE,CL=F,EURUSD=X,ERDZ.MU,JPJX.MU,HSIC,0P000151AI.F,LU1019989323.SG,ZZR8.F,FLRG.DE,NRJ.PA,DBK.DE

[eddysteurs]

I have exact the same error since last evening

[sharknoon]

Same here

[AndyDann]

I'm also seeing a problem with an android widget that I presume uses Yahoo Finance to source their data...

[dougle03]

Same here too. Log: This error originated from a custom integration.

Logger: custom_components.yahoofinance.coordinator Source: custom_components/yahoofinance/coordinator.py:307 Integration: Yahoo Finance (documentation, issues) First occurred: 11:51:32 (2 occurrences) Last logged: 11:51:38

Received status 401 (error=Unauthorized) for https://query1.finance.yahoo.com/v7/finance/quote?symbols=RR.L,BT-A.L,ARB.L

[iprak]

This is the same issue as https://github.com/iprak/yahoofinance/issues/93. It seems that restrictions have been added at webserver end which seemed to automatically resolve.

I did however attempt to put in a workaround to try/parsing out the authorization crumb. But the the error log indicates that no crumb was passed. Can you enable debug logging, it might tell what is happening.

logger:
  logs:
    custom_components.yahoofinance: debug

I will also point out that the component will try to reload after 20 seconds in case of failure. So one time entry of error message should be okay.

[drjjr2]

Same issue. Here is the debug log:

2023-05-05 07:39:30.006 DEBUG (MainThread) [custom_components.yahoofinance.coordinator] Getting crumb from from https://finance.yahoo.com/quote/NQ%3DF
2023-05-05 07:39:31.258 DEBUG (MainThread) [custom_components.yahoofinance.coordinator] Crumb=J7juHWu\u002FOPp
2023-05-05 07:39:31.258 DEBUG (MainThread) [custom_components.yahoofinance.coordinator] Requesting data from 'https://query1.finance.yahoo.com/v7/finance/quote?symbols=^GSPC,^IXIC,^DJI,^RUT&crumb=J7juHWu\u002FOPp'
2023-05-05 07:39:31.506 ERROR (MainThread) [custom_components.yahoofinance.coordinator] Received status 401 (error=Unauthorized) for https://query1.finance.yahoo.com/v7/finance/quote?symbols=^GSPC,^IXIC,^DJI,^RUT&crumb=J7juHWu\u002FOPp
2023-05-05 07:39:31.512 DEBUG (MainThread) [custom_components.yahoofinance.coordinator] Finished fetching YahooSymbolUpdateCoordinator data in 1.506 seconds (success: False)
2023-05-05 07:39:31.513 WARNING (MainThread) [custom_components.yahoofinance.coordinator] Error obtaining data, retrying in 20 seconds.
2023-05-05 07:39:31.514 DEBUG (MainThread) [custom_components.yahoofinance.sensor] ^GSPC available=False
2023-05-05 07:39:31.518 DEBUG (MainThread) [custom_components.yahoofinance.sensor] ^IXIC available=False
2023-05-05 07:39:31.518 DEBUG (MainThread) [custom_components.yahoofinance.sensor] ^DJI available=False
2023-05-05 07:39:31.519 DEBUG (MainThread) [custom_components.yahoofinance.sensor] ^RUT available=False

[iprak]

Thanks @drjjr2 The log is what I was hoping to see.

There seems to be some additional authentication involved and maybe this is based on region from where request is being made leading to different data centers being used. I have been able to successfully access data via a crumb from different internet providers. Last time this happened, I noticed that if the path was changed from v7 to v6, then the requests were successful but after a couple hours requests to v6 also started failing. It is as if some internal change in yahoo was propagating to other websites.

[drjjr2]

Thanks @drjjr2 The log is what I was hoping to see.

There seems to be some additional authentication involved and maybe this is based on region from where request is being made leading to different data centers being used. I have been able to successfully access data via a crumb from different internet providers. Last time this happened, I noticed that if the path was changed from v7 to v6, then the requests were successful but after a couple hours requests to v6 also started failing. It is as if some internal change in yahoo was propagating to other websites.

I can confirm that: https://query1.finance.yahoo.com/v7/finance/quote?symbols=^GSPC,^IXIC,^DJI,^RUT&crumb=J7juHWu\u002FOPp returns {"finance":{"result":null,"error":{"code":"Unauthorized","description":"Invalid Crumb"}}} while https://query1.finance.yahoo.com/v6/finance/quote?symbols=^GSPC,^IXIC,^DJI,^RUT&crumb=J7juHWu\u002FOPp returns {"quoteResponse":{"result":[{"language":"en-US","region":"US","quoteType":"INDEX","typeDisp":"Index","quoteSourceName":"Free Realtime Quote","triggerable":true,"customPriceAlertConfidence":"HIGH","tradeable":false,"cryptoTradeable":false,"currency":"USD","exchange":"SNP","shortName":"S&P 500","longName":"S&P 500","messageBoardId":"finmb_INDEXGSPC","marketState":"REGULAR","exchangeTimezoneName":"America/New_York","exchangeTimezoneShortName":"EDT","gmtOffSetMilliseconds":-14400000,"market":"us_market","esgPopulated":false,"regularMarketChangePercent":0.9396169,"regularMarketPrice":4099.38,"firstTradeDateMilliseconds":-1325583000000,"priceHint":2,"regularMarketChange":38.159912,"regularMarketTime":1683293440,"regularMarketDayHigh":4099.43,"regularMarketDayRange":"4084.73 - 4099.43","regularMarketDayLow":4084.73,"regularMarketVolume":66093059,"regularMarketPreviousClose":4061.22,"bid":4094.59,"ask":4104.44,"bidSize":0,"askSize":0,"fullExchangeName":"SNP","regularMarketOpen":4084.73,"averageDailyVolume3Month":4292687741,"averageDailyVolume10Day":3953081000,"fiftyTwoWeekLowChange":607.7998,"fiftyTwoWeekLowChangePercent":0.17407586,"fiftyTwoWeekRange":"3491.58 - 4325.28","fiftyTwoWeekHighChange":-225.8999,"fiftyTwoWeekHighChangePercent":-0.052227814,"fiftyTwoWeekLow":3491.58,"fiftyTwoWeekHigh":4325.28,"fiftyDayAverage":4039.1782,"fiftyDayAverageChange":60.20166,"fiftyDayAverageChangePercent":0.014904433,"twoHundredDayAverage":3969.841,"twoHundredDayAverageChange":129.53882,"twoHundredDayAverageChangePercent":0.03263073,"sourceInterval":15,"exchangeDataDelayedBy":0,"symbol":"^GSPC"},{"language":"en-US","region":"US","quoteType":"INDEX","typeDisp":"Index","quoteSourceName":"Free Realtime Quote","triggerable":true,"customPriceAlertConfidence":"HIGH","tradeable":false,"cryptoTradeable":false,"currency":"USD","exchange":"NIM","shortName":"NASDAQ Composite","longName":"NASDAQ Composite","messageBoardId":"finmb_INDEXIXIC","marketState":"REGULAR","exchangeTimezoneName":"America/New_York","exchangeTimezoneShortName":"EDT","gmtOffSetMilliseconds":-14400000,"market":"us_market","esgPopulated":false,"regularMarketChangePercent":0.92007303,"regularMarketPrice":12076.498,"firstTradeDateMilliseconds":34612200000,"priceHint":2,"regularMarketChange":110.09961,"regularMarketTime":1683293442,"regularMarketDayHigh":12078.509,"regularMarketDayRange":"12072.087 - 12078.509","regularMarketDayLow":12072.087,"regularMarketVolume":162578119,"regularMarketPreviousClose":11966.398,"bid":0.0,"ask":0.0,"bidSize":0,"askSize":0,"fullExchangeName":"Nasdaq GIDS","regularMarketOpen":11997.339,"averageDailyVolume3Month":5037841451,"averageDailyVolume10Day":5159296000,"fiftyTwoWeekLowChange":1987.668,"fiftyTwoWeekLowChangePercent":0.1970167,"fiftyTwoWeekRange":"10088.83 - 13181.09","fiftyTwoWeekHighChange":-1104.5918,"fiftyTwoWeekHighChangePercent":-0.08380125,"fiftyTwoWeekLow":10088.83,"fiftyTwoWeekHigh":13181.09,"fiftyDayAverage":11824.833,"fiftyDayAverageChange":251.66504,"fiftyDayAverageChangePercent":0.021282757,"twoHundredDayAverage":11493.624,"twoHundredDayAverageChange":582.874,"twoHundredDayAverageChangePercent":0.050712813,"sourceInterval":15,"exchangeDataDelayedBy":0,"symbol":"^IXIC"},{"language":"en-US","region":"US","quoteType":"INDEX","typeDisp":"Index","quoteSourceName":"Free Realtime Quote","triggerable":true,"customPriceAlertConfidence":"HIGH","tradeable":false,"cryptoTradeable":false,"currency":"USD","exchange":"DJI","shortName":"Dow Jones Industrial Average","longName":"Dow Jones Industrial Average","messageBoardId":"finmb_INDEXDJI","marketState":"REGULAR","exchangeTimezoneName":"America/New_York","exchangeTimezoneShortName":"EDT","gmtOffSetMilliseconds":-14400000,"market":"us_market","esgPopulated":false,"regularMarketChangePercent":0.9529283,"regularMarketPrice":33443.42,"firstTradeDateMilliseconds":694362600000,"priceHint":2,"regularMarketChange":315.6836,"regularMarketTime":1683293444,"regularMarketDayHigh":33449.46,"regularMarketDayRange":"33248.55 - 33449.46","regularMarketDayLow":33248.55,"regularMarketVolume":11491663,"regularMarketPreviousClose":33127.74,"bid":33420.83,"ask":33461.1,"bidSize":0,"askSize":0,"fullExchangeName":"DJI","regularMarketOpen":33248.55,"averageDailyVolume3Month":313024032,"averageDailyVolume10Day":297041000,"fiftyTwoWeekLowChange":4782.4824,"fiftyTwoWeekLowChangePercent":0.16686411,"fiftyTwoWeekRange":"28660.94 - 34712.28","fiftyTwoWeekHighChange":-1268.8594,"fiftyTwoWeekHighChangePercent":-0.036553614,"fiftyTwoWeekLow":28660.94,"fiftyTwoWeekHigh":34712.28,"fiftyDayAverage":33076.184,"fiftyDayAverageChange":367.23828,"fiftyDayAverageChangePercent":0.011102801,"twoHundredDayAverage":32708.07,"twoHundredDayAverageChange":735.35156,"twoHundredDayAverageChangePercent":0.022482267,"sourceInterval":120,"exchangeDataDelayedBy":0,"symbol":"^DJI"},{"language":"en-US","region":"US","quoteType":"INDEX","typeDisp":"Index","quoteSourceName":"Delayed Quote","triggerable":true,"customPriceAlertConfidence":"HIGH","tradeable":false,"cryptoTradeable":false,"currency":"USD","exchange":"WCB","shortName":"Russell 2000","longName":"Russell 2000","messageBoardId":"finmb_INDEXRUT","marketState":"REGULAR","exchangeTimezoneName":"America/New_York","exchangeTimezoneShortName":"EDT","gmtOffSetMilliseconds":-14400000,"market":"us_market","esgPopulated":false,"regularMarketChangePercent":-1.177074,"regularMarketPrice":1718.811,"firstTradeDateMilliseconds":558279000000,"priceHint":2,"regularMarketChange":-20.472656,"regularMarketTime":1683232208,"regularMarketDayHigh":1728.3916,"regularMarketDayRange":"1704.8181 - 1728.3916","regularMarketDayLow":1704.8181,"regularMarketVolume":0,"regularMarketPreviousClose":1739.2837,"bid":0.0,"ask":0.0,"bidSize":0,"askSize":0,"fullExchangeName":"Chicago Options","regularMarketOpen":1727.2416,"averageDailyVolume3Month":4292687741,"averageDailyVolume10Day":3953081000,"fiftyTwoWeekLowChange":77.341064,"fiftyTwoWeekLowChangePercent":0.047116954,"fiftyTwoWeekRange":"1641.47 - 2030.05","fiftyTwoWeekHighChange":-311.239,"fiftyTwoWeekHighChangePercent":-0.15331593,"fiftyTwoWeekLow":1641.47,"fiftyTwoWeekHigh":2030.05,"fiftyDayAverage":1792.1952,"fiftyDayAverageChange":-73.384155,"fiftyDayAverageChangePercent":-0.04094652,"twoHundredDayAverage":1826.6267,"twoHundredDayAverageChange":-107.815674,"twoHundredDayAverageChangePercent":-0.05902447,"sourceInterval":15,"exchangeDataDelayedBy":15,"symbol":"^RUT"}],"error":null}}

I get data.

[pciduncan76]

Disabled NordVPN - sensors began working again.

[justinodunn]

It works when I access this url on my browser: https://query2.finance.yahoo.com/v1/test/getcrumb

copy the crumb

and paste it at the end of the url like this: https://query1.finance.yahoo.com/v7/finance/quote?symbols=^GSPC,^IXIC,^DJI,^RUT&crumb=YOURCRUMBHERE

Not sure how to translate this in code though

[iprak]

@justinodunn That only seems to work if you have accessed Yahoo Finance website once. https://query2.finance.yahoo.com/v1/test/getcrumb doesn't return anything for a brand new request i.e. incognito instance.

[jeroen84]

Seems like you need to provide the cookie(s) that are provided once you visit the yahoo finance website in the getcrumb request. Otherwise the getcrumb returns nothing, which you observe in an incognito instance. There is however a cookies consent form you need to pass. This can be a challenging step...

[justinodunn]

Don't know much about shell scripts but they seem to have a workaround here.. https://github.com/pstadler/ticker.sh/blob/acquire-yahoo-finance-session/ticker.sh

[iprak]

@jeroen84 , @justinodunn

That ticker.sh script establishes a session by navigating to finance.yahoo.com, then gets the crumb which is later passed to ticker requests. My component does something similar, crumb+cookies are extracted out of a request for Nasdaq symbol.

Logs provided by other folks confirm that crumb is being supplied in the requests. Can you enable debug logging to verify that crumb is being passed?

[drjjr2]

Well it started working again for me and I changed nothing.

[iprak]

That is what happened last time too. It suddenly stopped working and started working by itself. It feels that something might be going on at the Yahoo server backend.

One thing I did notice in my experimentation that sometimes the request would fail with invalid crumb. But if I made the same request again, with same crumb+cookies then it succeeded. Seems like including a re-try would be good idea.

[swiergot]

Still not working here. Looks like it can't get the crumb.

2023-05-07 12:34:17.001 DEBUG (MainThread) [custom_components.yahoofinance.coordinator] Getting crumb from from https://finance.yahoo.com/quote/NQ%3DF
2023-05-07 12:34:17.567 DEBUG (MainThread) [custom_components.yahoofinance.coordinator] Requesting data from 'https://query1.finance.yahoo.com/v7/finance/quote?symbols=^GSPC
2023-05-07 12:34:17.679 ERROR (MainThread) [custom_components.yahoofinance.coordinator] Received status 401 (error=Unauthorized) for https://query1.finance.yahoo.com/v7/finance/quote?symbols=^GSPC

But logging in the code is not perfect, there are three places where it can fail silently. I will try to enhance the logging.

[eddysteurs]

I see the request for getting the crumb but then in the next request the crumb is not added, is this not necessary?

2023-05-07 15:21:00.001 DEBUG (MainThread) [custom_components.yahoofinance.coordinator] Getting crumb from from https://finance.yahoo.com/quote/NQ%3DF 2023-05-07 15:21:00.599 DEBUG (MainThread) [custom_components.yahoofinance.coordinator] Requesting data from 'https://query1.finance.yahoo.com/v7/finance/quote?symbols=STZ.PA,SPYK.DE,ABBN.SW,EW,MSFT,REP.MC,ANTO.L,ACLN.SW,EURUSD=X,EURCHF=X,EURDKK=X,EURGBP=X,EURNOK=X,BAR.BR,BMW.DE,AIR.PA,KBC.BR,KIN.BR,MELE.BR,ACKB.BR,UCB.BR,XFAB.PA,VUSA.AS,MDLZ,COLR.BR,CYAD.BR,GREEN.BR,AD.AS,EUNL.DE,DEME.BR,STK.PA,MMM,ABT,ADBE,ALGN,GOOGL,AMZN,AXP,AMT,AAPL,ASML.AS,AZN,BAC,BLK,SQ,BNP.PA,AVGO,CAT,COP,COST,ACA.PA,BN.PA,DSY.PA,DOCU,LLY,EQNR.OL,EL.PA,HEI,HMC,ISP.MI,INTU,ISRG,JPM,LRCX,OR.PA,MA,MBG.DE,MRK,META,NFLX,NEE,NKE,NSC,NOVO-B.CO,NVDA,ORCL,PYPL,RIO,ROP,CRM,SAN.PA,SLB,NOW,SIE.DE,SONY,TSM,TGT,FTI,SCHW,GS,HD,DIS,TMO,TT,UBER,UNA.AS,VRTX,V,WMT,XZMJ.DE,XYL,ZM,IBE.MC,AAL.L,RMS.PA,DTG.DE,HLT,BKNG,VWS.CO,C,MS,DPWA.DE,AMD' 2023-05-07 15:21:00.792 ERROR (MainThread) [custom_components.yahoofinance.coordinator] Received status 401 (error=Unauthorized) for https://query1.finance.yahoo.com/v7/finance/quote?symbols=STZ.PA,SPYK.DE,ABBN.SW,EW,MSFT,REP.MC,ANTO.L,ACLN.SW,EURUSD=X,EURCHF=X,EURDKK=X,EURGBP=X,EURNOK=X,BAR.BR,BMW.DE,AIR.PA,KBC.BR,KIN.BR,MELE.BR,ACKB.BR,UCB.BR,XFAB.PA,VUSA.AS,MDLZ,COLR.BR,CYAD.BR,GREEN.BR,AD.AS,EUNL.DE,DEME.BR,STK.PA,MMM,ABT,ADBE,ALGN,GOOGL,AMZN,AXP,AMT,AAPL,ASML.AS,AZN,BAC,BLK,SQ,BNP.PA,AVGO,CAT,COP,COST,ACA.PA,BN.PA,DSY.PA,DOCU,LLY,EQNR.OL,EL.PA,HEI,HMC,ISP.MI,INTU,ISRG,JPM,LRCX,OR.PA,MA,MBG.DE,MRK,META,NFLX,NEE,NKE,NSC,NOVO-B.CO,NVDA,ORCL,PYPL,RIO,ROP,CRM,SAN.PA,SLB,NOW,SIE.DE,SONY,TSM,TGT,FTI,SCHW,GS,HD,DIS,TMO,TT,UBER,UNA.AS,VRTX,V,WMT,XZMJ.DE,XYL,ZM,IBE.MC,AAL.L,RMS.PA,DTG.DE,HLT,BKNG,VWS.CO,C,MS,DPWA.DE,AMD 2023-05-07 15:21:00.792 DEBUG (MainThread) [custom_components.yahoofinance.coordinator] Finished fetching YahooSymbolUpdateCoordinator data in 0.792 seconds (success: False)

[swiergot]

Works again for me with changes from #97.

[iprak]

@eddysteurs That will only happen if crumb response did not contain crumb or the parsing failed. Debug log would contain a message about that.

[SzemiSun]

https://github.com/iprak/yahoofinance/pull/98/files

[eddysteurs]

Change to v6 works for me😁