Scraper error "TypeError: string indices must be integers" - Yahoo decrypt fail

[SamZhang02]

Updates

2023 January 13

By the time of posting the issue (2023 January 12), the issue only occured sometimes. The library is now (2023 January 13) completely broken and I am unable to retrieve any stock informatio

2023 January 14

Fix has been merged to the branch dev

Info about your system:

yfinance version: 0.2.3 Operating system: macOS Monteray 12.0.1

Snippet that can recreate the error

stock = yf.Ticker("^GSPC")
info = stock.info

Error

Message:TypeError: string indices must be integers It seems to be a problem where the scraper is not scraping the correct information, leading to a crash.

Traceback:

Traceback (most recent call last):
  File "/home/2022/szhang139/.local/lib/python3.10/site-packages/apscheduler/executors/base_py3.py", line 30, in run_coroutine_job
    retval = await job.func(*job.args, **job.kwargs)
  File "/home/2022/szhang139/repos/STONK/src/main.py", line 61, in notify
    market = get_major_index(f'Market Close - {daytime.today_date()}')
  File "/home/2022/szhang139/repos/STONK/src/market_info.py", line 63, in get_major_index
    sp500 = get_stock('^GSPC')
  File "/home/2022/szhang139/repos/STONK/src/market_info.py", line 41, in get_stock
    stock_info = get_stock_info(stock_name)
  File "/home/2022/szhang139/repos/STONK/src/market_info.py", line 8, in get_stock_info
    info = stock.info
  File "/home/2022/szhang139/.local/lib/python3.10/site-packages/yfinance/ticker.py", line 138, in info
    return self.get_info()
  File "/home/2022/szhang139/.local/lib/python3.10/site-packages/yfinance/base.py", line 894, in get_info
    data = self._quote.info
  File "/home/2022/szhang139/.local/lib/python3.10/site-packages/yfinance/scrapers/quote.py", line 27, in info
    self._scrape(self.proxy)
  File "/home/2022/szhang139/.local/lib/python3.10/site-packages/yfinance/scrapers/quote.py", line 58, in _scrape
    quote_summary_store = json_data['QuoteSummaryStore']

Frequency

The error occurs in no apparent pattern. Every time it occurs, it seem to persist for some range of time before it recovers back to normal. n.

[Lokan0py]

I also have the exact same problem as you, good to know that im not alone. The yfinance ticker module seems very unstable at the moment.

[YieldPilot]

Same issue here.

Traceback (most recent call last): File "/home/gulder/PycharmProjects/YPUpdater/YPFunctions.py", line 171, in updatepricedata beta = yf.Ticker(ticker).info.get('beta') File "/home/gulder/PycharmProjects/YPUpdater/venv/lib/python3.8/site-packages/yfinance/ticker.py", line 138, in info return self.get_info() File "/home/gulder/PycharmProjects/YPUpdater/venv/lib/python3.8/site-packages/yfinance/base.py", line 894, in get_info data = self._quote.info File "/home/gulder/PycharmProjects/YPUpdater/venv/lib/python3.8/site-packages/yfinance/scrapers/quote.py", line 27, in info self._scrape(self.proxy) File "/home/gulder/PycharmProjects/YPUpdater/venv/lib/python3.8/site-packages/yfinance/scrapers/quote.py", line 58, in _scrape quote_summary_store = json_data['QuoteSummaryStore'] TypeError: string indices must be integers

[ValueRaider]

Are you accessing info in a loop of many tickers? Spamming info is a great way to get temporarily blocked, which I suspect.

If yes, what key are you accessing? Might be better way to get data.

[andreasklippinge]

Are you accessing info in a loop of many tickers? Spamming info is a great way to get temporarily blocked, which I suspect.

If yes, what key are you accessing? Might be better way to get data.

I have the same error, and it does not matter if I looping through 4 or 503 stocks..

data = yf.download(tickers, period = "max", interval = "1mo", progress=True)
stocks = []
for ticker in tickers:
    df = data.xs(ticker, level=1, axis=1).copy()
    df.dropna(inplace=True)
    if df.empty:
        continue
    info = yf.Ticker(ticker).info
    div = info.get("dividendYield")  
    mcap = info.get("marketCap") 
    PE = info.get("trailingPE")
    if div is not None and div >= 0.04 and mcap is not None and mcap > 1E10 and PE is not None and PE < 15:
        stocks.append(ticker)

Traceback (most recent call last):

  File "C:\Users\untitled0.py", line 18, in <module>
    info = yf.Ticker(ticker).info

  File "C:\Users\Andreas\anaconda3\lib\site-packages\yfinance\ticker.py", line 138, in info
    return self.get_info()

  File "C:\Users\Andreas\anaconda3\lib\site-packages\yfinance\base.py", line 894, in get_info
    data = self._quote.info

  File "C:\Users\Andreas\anaconda3\lib\site-packages\yfinance\scrapers\quote.py", line 27, in info
    self._scrape(self.proxy)

  File "C:\Users\Andreas\anaconda3\lib\site-packages\yfinance\scrapers\quote.py", line 58, in _scrape
    quote_summary_store = json_data['QuoteSummaryStore']

TypeError: string indices must be integers

[CopenhagenToLondon]

Yes me too. It seems it went down hill over the last few moments. I can now only run about 1 at a time. I get the error on financials with some symbols and on balancesheet data with other symbols, but none on price data or dividends

[YieldPilot]

Yes, I am, loop of around 450 tickers. I am accessing: yf.Ticker(ticker).info.get('beta') and yf.Ticker(ticker).info.get('dividendRate')

[galashour]

Indeed, I also just encountered the same thing. In the specific tool, I'm suing it for a single ticker, and I get this issue as well

Retrieve fundamental and technical data from yfinance

ticker_data = yf.Ticker(ticker) time.sleep(2) # just added today, doesn't help ticker_info = ticker_data.info

I tried adding sleep (above), but it didn't help either.

The weird part, is that for a short time when I wrote the code line by line in the Python Console (pycharm) it worked, but now even in the console I get ticker_info as 'None'

[AnubisSg1]

Same failure, im def not ip blocked and didn't spam. It started yesterday and today I get the same failure the whole time

[RichardAtCT]

Just to +1 that I am having the same issue. Just checking 1 ticker, once a minute.

[chris-hmmr]

I believe the cause of this relates to the underlying data structure this library scrapes from Yahoo Finance has changed. They introduced encryption on the ticker data before the turn of the year, which was fixed here: https://github.com/ranaroussi/yfinance/pull/1253/commits/8e5f0984af347afda6be74b27a989422e49a975b. This time around they changed the parameters on the encryption.

[ValueRaider]

In case it helps, these were the decryption params Yahoo was providing:

_cs = '3b1885bff81b'
_cr = {"words":[1975313454,-1113096907,-1675002031,-87263202], "sigBytes":16}

[ValueRaider]

We know the decryption is failing. Please keep future comments related to debugging or fixing.

[ValueRaider]

Where Yahoo used to provide the decryption parameters, there is this new key-value. It appears to be constant.

'93634354fa08860f003e': '4f566bf243f93d68718581c573aee0ff86456eef74bf9c33244c765b6ae2470059db233bf70a4fb3c16c24047befc4e993e717df7cbd4f1bebddb9649abdf5e3'

Any ideas?

[alexa-infra]

I think 93634354fa08860f003e key/value will change with time..

         encrypted_stores = data['context']['dispatcher']['stores']
-        _cs = data["_cs"]
-        _cr = data["_cr"]
-
-        _cr = b"".join(int.to_bytes(i, length=4, byteorder="big", signed=True) for i in json.loads(_cr)["words"])
-        password = hashlib.pbkdf2_hmac("sha1", _cs.encode("utf8"), _cr, 1, dklen=32).hex()
+        password_key = next(key for key in data.keys() if key not in ["context", "plugins"])
+        password = data[password_key]

         encrypted_stores = b64decode(encrypted_stores)

and it seems this

-    plaintext = unpad(plaintext, 16, style="pkcs7")
+    plaintext = unpad(plaintext, 128, style="pkcs7")

[dschaefer]

That is working for me though I had to leave the unpad block_size arg at 16 when using usePycryptodome. But also works without usePycryptodome.

[ValueRaider]

Can you submit a pull request? If a Github newbie see #1084

[galashour]

Hi, I'm also impacted by this issue.

Do I need to declare the encryption parameters within my script or do I need to download latest version of yfinance module?

Everyone is impacted. Per @ValueRaider request, no need to continue reporting the issue.

just continue tracking this thread, and when a new release with a fix is available/merged to main, a comment will be put in this thread and then we should be able to pip-install the new version.

The plan, similar to past cases, is that once a solution is available in the new release no change to the existing scripts will be required (or at least this is the motivation).

[ValueRaider]

PR #1297 seems to work. But before I merge & release, I want to make other changes to reduce Yahoo server load - see #1300 for discussion.

[ValueRaider]

@james-stevens Please discuss the patch code in it's pull request.

[cmjordan42]

PR #1297 seems to work. But before I merge & release, I want to make other changes to reduce Yahoo server load - see #1300 for discussion.

I'm not sure it makes sense to conflate the release of a critical fix with a potential efficiency improvement. 100% of the yfinance community will benefit from #1297 urgently being released.

[ValueRaider]

I think it does make sense to conflate. Why is Yahoo encrypting this data? Is it to stop scraping because of the high server load? Then it's in our benefit to reduce load of yfinance. What will Yahoo do next?

Plus this will not delay much, most groundwork was done weeks-months ago.

EDIT: Actually, makes more sense to release the info change afterwards, because will require a new major release 0.3. So fix will be released first asap.

[bruce1095]

I would appreciate an expedited roll out. I use Yfinance pretty extensively and all my code is now defunct. Sent from my iPhoneOn Jan 14, 2023, at 11:52 AM, ValueRaider @.***> wrote: I think it does make sense to conflate. Why is Yahoo encrypting this data? Is it to stop scraping because of the high server load? Then it's in our benefit to reduce load of yfinance. What will Yahoo do next? Plus this will not delay much, most groundwork was done weeks-months ago.

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: @.***>

[ValueRaider]

PIP release out

[dannydoritoeth]

The google collab notebook I'm using has this in it and its still failing

!pip3 install yfinance

Would you expect that to be working now?

[interstormtech]

Thank you very much - your time and expertise is appreciated by the community. I uninstalled and reinstalled latest...everything working perfectly! Again - many thanks!

[dannydoritoeth]

Got it. It was just defaulting to the old version. This works:

!pip3 install yfinance==0.2.4

Thank you very much for the quick turn around.

[GPN87]

Is there any reason why I would be getting a 'ModuleNotFoundError: No module named 'multitasking'" error? Multitasking, as well as yfinance 0.2.4 are both installed in my environment.

[CarlosEspinoTimon]

Hi all!

Thank you very much for the fix, I'll try it right away :smile:

I have one question related to this that is bothering me and maybe you could help me understand... isn't it pointless that Yahoo encrypts this AND shares the key to decrypt it?

I mean, If I encrypt something it's because I want to secure the data and then I would share the key to whomever should be allowed to decrypt it, so, why encrypt and give the key.

To me sound wrong, but probably I am missing something here :sweat_smile:

[bachan007]

Also started facing the same issue, any solution here or do need to change the code structure? TypeError: string indices must be integers

[galashour]

Also started facing the same issue, any solution here or do need to change the code structure? TypeError: string indices must be integers

Double check you upgraded to the most recent version 0.2.4.

[bachan007]

Naah, using yfinance==0.1.93

[ValueRaider]

Download latest PIP release to get the fix. That's how bug fixes work.

[bachan007]

Download latest PIP release to get the fix. That's how bug fixes work.

still not working

[bachan007]

Also started facing the same issue, any solution here or do need to change the code structure? TypeError: string indices must be integers

I just upgraded the pip and yfinance, everything is working fine now.

[no0be]

Upgraded to yfinance==0.2.4 and it works fine now 👍 !

[JMKwins]

Thx for the effort guys and girls. very much appreciated from here !