Could not get crumb from Yahoo Finance

[vincenttermaat]

python.exe RecessionPredictor_master.py deploy

Getting primary data from APIs...

Getting data from FRED API as of 2023-01-08...
        |--Getting data for Non-farm_Payrolls(PAYEMS).
        |--Getting data for Civilian_Unemployment_Rate(UNRATE).
        |--Getting data for Effective_Fed_Funds(FEDFUNDS).
        |--Getting data for CPI_All_Items(CPIAUCSL).
        |--Getting data for 10Y_Treasury_Rate(GS10).
        |--Getting data for 5Y_Treasury_Rate(GS5).
        |--Getting data for 3_Month_T-Bill_Rate(TB3MS).
        |--Getting data for IPI(INDPRO).
Finished getting data from FRED API!

Getting data from Yahoo Finance...
        |--Getting data for S&P_500_Index(^GSPC).
Traceback (most recent call last):
  File "RecessionPredictor_master.py", line 48, in <module>
    get_data = mk.MakeDataset().get_all_data()
  File "C:\Users\Vincent\source\repos\Recession-Predictor\Recession-Predictor\src\data\make_dataset.py", line 399, in get_all_data
    self.get_primary_data()
  File "C:\Users\Vincent\source\repos\Recession-Predictor\Recession-Predictor\src\data\make_dataset.py", line 283, in get_primary_data
    self.get_yahoo_data()
  File "C:\Users\Vincent\source\repos\Recession-Predictor\Recession-Predictor\src\data\make_dataset.py", line 202, in get_yahoo_data
    series_data.yahoo_response(series_id)
  File "C:\Users\Vincent\source\repos\Recession-Predictor\Recession-Predictor\src\data\make_dataset.py", line 108, in yahoo_response
    series_dataframe = YahooData(series_id).get_quote()[::-1]
  File "C:\Users\Vincent\source\repos\Recession-Predictor\Recession-Predictor\src\data\make_dataset.py", line 64, in get_quote
    self.get_crumb()
  File "C:\Users\Vincent\source\repos\Recession-Predictor\Recession-Predictor\src\data\make_dataset.py", line 54, in get_crumb
    raise ValueError('Could not get crumb from Yahoo Finance')
ValueError: Could not get crumb from Yahoo Finance

[vincenttermaat]

The response from Yahoo seems to have changed. When calling https://finance.yahoo.com/quote/%5EGSPC/history?p=%5EGSPC with postman. It now returns the crumb without the "Crumbstore" attribute, as such:

"bkt":"finance-US-en-US-def","crumb":"pQyoa9t9AWz",

I think the crumble_regex in make_dataset.py needs to be changed from: crumble_regex = r'CrumbStore":{"crumb":"(.*?)"}' to something like this: crumble_regex = r'"crumb":"(.*?)"}'

I did try some stuff but I do not have the tooling to properly debug.

Grabbing the crumb manually from the response in postman and putting it in the code as below did fix the issue, but I guess this is only temporary:

   def get_crumb(self):
        """
        Original code source: https://stackoverflow.com/questions/44225771/scraping-historical-data-from-yahoo-finance-with-python
        """
        self.crumb = "pQyoa9t9AWz"
        #https://finance.yahoo.com/quote/%5EGSPC/history?p=%5EGSPC

        #response = self.session.get(self.crumb_link.format(self.symbol),
        #                            headers=self.headers,
        #                            timeout=self.timeout)
        #response.raise_for_status()
        #match = re.search(self.crumble_regex, response.text)
        #if not match:
        #    raise ValueError('Could not get crumb from Yahoo Finance')
        #else:
        #    self.crumb = match.group(1)

[KatherineShaver]

I am having the same issue. Will there be a fix for this?

[tzhangwps]

Hi - I was out travelling last weekend, will take a look this weekend.

[tzhangwps]

@vtermaat your suggestion worked perfectly, thank you! I've updated make_dataset.py accordingly, and merged into the master branch.

[KatherineShaver]

Worked for me today. Big thanks to both of you!