0.2.29 bug: OperationalError('unable to open database file')

[githubbla]

Describe bug

fail when downloading array, work ok in 0.2.28

Simple code that reproduces your problem

from pandas_datareader import data as pdr
import yfinance as yf
from numpy import isnan

##yf.pdr_override()

def loadtickers(tickers,start="2005-1-1",end=None,renamefield={},prev=None):
    for i in range(0,2):
        try:
            df = yf.download(tickers, start, end, auto_adjust=True,ignore_tz = True)
            df.columns = ['_'.join(((str.lower(col[1]) if col[1] not in renamefield.keys() else str.lower(renamefield[col[1]]), str.lower(col[0])))) for col in df.columns]
            for t in tickers:
                if t in renamefield.keys():
                    t=renamefield[t]
                print("{}_return".format(t).lower(),"{}_close".format(t).lower() )
                df["{}_return".format(t).lower()] = df["{}_close".format(t).lower()].pct_change().fillna(0)
            if prev is not None:
                df=df.join(prev,how='left')
            return df
        except Exception as e:
            print(e)
            continue
        break
    return

p1=loadtickers(["SPY","TLT"],start="2004-12-31",renamefield={"^VXTLT":"VXTLT","^TNX":"TNX",})

Debug log

DEBUG Entering download() DEBUG:yfinance:Entering download() DEBUG Disabling multithreading because DEBUG logging enabled DEBUG:yfinance: Disabling multithreading because DEBUG logging enabled DEBUG Entering history() DEBUG:yfinance: Entering history() DEBUG Entering history() DEBUG:yfinance: Entering history() DEBUG Entering history() DEBUG:yfinance: Entering history() DEBUG Entering history() DEBUG:yfinance: Entering history() DEBUG Entering history() DEBUG:yfinance: Entering history() DEBUG Entering history() DEBUG:yfinance: Entering history() DEBUG Entering history() DEBUG:yfinance: Entering history() DEBUG Entering history() DEBUG:yfinance: Entering history() DEBUG Entering history() DEBUG:yfinance: Entering history() DEBUG Entering history() DEBUG:yfinance: Entering history() DEBUG Entering history() DEBUG:yfinance: Entering history() DEBUG Entering history() DEBUG:yfinance: Entering history() DEBUG Entering history() DEBUG:yfinance: Entering history() DEBUG Entering history() DEBUG:yfinance: Entering history() DEBUG Entering history() DEBUG:yfinance: Entering history() DEBUG Entering history() DEBUG:yfinance: Entering history() DEBUG Entering history() DEBUG:yfinance: Entering history() DEBUG Entering history() DEBUG:yfinance: Entering history() DEBUG Entering history() DEBUG:yfinance: Entering history() DEBUG Entering history() DEBUG:yfinance: Entering history() DEBUG Entering history() DEBUG:yfinance: Entering history() DEBUG Entering history() DEBUG:yfinance: Entering history() DEBUG Entering history() DEBUG:yfinance: Entering history() DEBUG Entering history() DEBUG:yfinance: Entering history() ERROR
24 Failed downloads: ERROR:yfinance: 24 Failed downloads: ERROR ['TMF', 'TMV', 'VBINX', 'SHV', '^GVZ', 'VXZ', '^VIX', 'SGOV', 'TLT', 'QQQ', '^VXN', 'GLL', 'SPHD', 'SPY', 'TQQQ', 'QLD', 'SPLV', 'TBF', 'IAU', '^VXTLT', 'GLD', 'XLU', 'VXX', 'UGL']: OperationalError('unable to open database file') ERROR:yfinance: ['TMF', 'TMV', 'VBINX', 'SHV', '^GVZ', 'VXZ', '^VIX', 'SGOV', 'TLT', 'QQQ', '^VXN', 'GLL', 'SPHD', 'SPY', 'TQQQ', 'QLD', 'SPLV', 'TBF', 'IAU', '^VXTLT', 'GLD', 'XLU', 'VXX', 'UGL']: OperationalError('unable to open database file') DEBUG ['TMF', 'TMV', 'VBINX', 'SHV', '^GVZ', 'VXZ', '^VIX', 'SGOV', 'TLT', 'QQQ', '^VXN', 'GLL', 'SPHD', 'SPY', 'TQQQ', 'QLD', 'SPLV', 'TBF', 'IAU', '^VXTLT', 'GLD', 'XLU', 'VXX', 'UGL']: Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/peewee.py", line 3192, in connect self._state.set_connection(self._connect()) File "/usr/local/lib/python3.10/dist-packages/peewee.py", line 3536, in _connect conn = sqlite3.connect(self.database, timeout=self._timeout, sqlite3.OperationalError: unable to open database file

      During handling of the above exception, another exception occurred:

      Traceback (most recent call last):
        File "/usr/local/lib/python3.10/dist-packages/yfinance/multi.py", line 275, in _download_one
          data = Ticker(ticker, session=session).history(
        File "/usr/local/lib/python3.10/dist-packages/yfinance/utils.py", line 114, in wrapper
          result = func(*args, **kwargs)
        File "/usr/local/lib/python3.10/dist-packages/yfinance/base.py", line 148, in history
          tz = self._get_ticker_tz(proxy, timeout)
        File "/usr/local/lib/python3.10/dist-packages/yfinance/base.py", line 1644, in _get_ticker_tz
          cache = utils.get_tz_cache()
        File "/usr/local/lib/python3.10/dist-packages/yfinance/utils.py", line 985, in get_tz_cache
          _tz_cache = _TzCache()
        File "/usr/local/lib/python3.10/dist-packages/yfinance/utils.py", line 926, in __init__
          db.connect()
        File "/usr/local/lib/python3.10/dist-packages/peewee.py", line 3191, in connect
          with __exception_wrapper__:
        File "/usr/local/lib/python3.10/dist-packages/peewee.py", line 3019, in __exit__
          reraise(new_type, new_type(exc_value, *exc_args), traceback)
        File "/usr/local/lib/python3.10/dist-packages/peewee.py", line 192, in reraise
          raise value.with_traceback(tb)
        File "/usr/local/lib/python3.10/dist-packages/peewee.py", line 3192, in connect
          self._state.set_connection(self._connect())
        File "/usr/local/lib/python3.10/dist-packages/peewee.py", line 3536, in _connect
          conn = sqlite3.connect(self.database, timeout=self._timeout,
      peewee.OperationalError: unable to open database file

DEBUG:yfinance: ['TMF', 'TMV', 'VBINX', 'SHV', '^GVZ', 'VXZ', '^VIX', 'SGOV', 'TLT', 'QQQ', '^VXN', 'GLL', 'SPHD', 'SPY', 'TQQQ', 'QLD', 'SPLV', 'TBF', 'IAU', '^VXTLT', 'GLD', 'XLU', 'VXX', 'UGL']: Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/peewee.py", line 3192, in connect self._state.set_connection(self._connect()) File "/usr/local/lib/python3.10/dist-packages/peewee.py", line 3536, in _connect conn = sqlite3.connect(self.database, timeout=self._timeout, sqlite3.OperationalError: unable to open database file

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/yfinance/multi.py", line 275, in _download_one data = Ticker(ticker, session=session).history( File "/usr/local/lib/python3.10/dist-packages/yfinance/utils.py", line 114, in wrapper result = func(*args, *kwargs) File "/usr/local/lib/python3.10/dist-packages/yfinance/base.py", line 148, in history tz = self._get_ticker_tz(proxy, timeout) File "/usr/local/lib/python3.10/dist-packages/yfinance/base.py", line 1644, in _get_ticker_tz cache = utils.get_tz_cache() File "/usr/local/lib/python3.10/dist-packages/yfinance/utils.py", line 985, in get_tz_cache _tz_cache = _TzCache() File "/usr/local/lib/python3.10/dist-packages/yfinance/utils.py", line 926, in init db.connect() File "/usr/local/lib/python3.10/dist-packages/peewee.py", line 3191, in connect with __exception_wrapper: File "/usr/local/lib/python3.10/dist-packages/peewee.py", line 3019, in exit__ reraise(new_type, new_type(exc_value, exc_args), traceback) File "/usr/local/lib/python3.10/dist-packages/peewee.py", line 192, in reraise raise value.with_traceback(tb) File "/usr/local/lib/python3.10/dist-packages/peewee.py", line 3192, in connect self._state.set_connection(self._connect()) File "/usr/local/lib/python3.10/dist-packages/peewee.py", line 3536, in _connect conn = sqlite3.connect(self.database, timeout=self._timeout, peewee.OperationalError: unable to open database file

DEBUG Exiting download() DEBUG:yfinance:Exiting download() 2023-09-22-15-59 tickers.pkl loading fresh 2023-09-22-15-59 tickers.pkl spy_return spy_close tlt_return tlt_close iau_return iau_close gld_return gld_close qqq_return qqq_close vix_return vix_close vxtlt_return vxtlt_close gvz_return gvz_close qld_return qld_close vxz_return vxz_close tbf_return tbf_close vxn_return vxn_close vxx_return vxx_close tmf_return tmf_close tmv_return tmv_close splv_return splv_close sphd_return sphd_close vbinx_return vbinx_close xlu_return xlu_close tqqq_return tqqq_close ugl_return ugl_close gll_return gll_close sgov_return sgov_close shv_return shv_close

Bad data proof

No response

yfinance version

0.2.29

Python version

No response

Operating system

No response

[ValueRaider]

You plan on editing with error message and working code?

[githubbla]

i added, tell me of this is enough, it's fail with 0.2.29, but ok with 0.2.28

[ValueRaider]

Run again debug logging enabled. Like the bug report template instructed.

[githubbla]

done!

[arieller]

same

[ValueRaider]

I can't debug until tomorrow, but here's summary in case someone else can.

Previous version was 99% thread-safe. In rare cases the thread-safety failed.

Latest version switches to peewee module. Supposed to be 100% thread-safe, and I tested on few threads. Maybe it's configured wrong?

[enzbus]

Confirmed. Fails on my CI pipeline in python <=3.10, works on my mac with 3.11.

[ValueRaider]

There's clearly a bug in 0.2.29. Keep replies constructive and concise.

[ValueRaider]

@enzbus how many CPUs on your mac, vs how many on system running pipeline?

[githubbla]

My sample fail on Google colab

[enzbus]

@enzbus how many CPUs on your mac, vs how many on system running pipeline?

So far I've only observed failure on github CI, which is single processor (I believe). My machine has multiple processors. The only way I call yfinance is via yf.download, from a single thread. The errors that I get on github testing are like this

ERROR:yfinance: 1 Failed download: ERROR:yfinance:['AAPL']: OperationalError('unable to open database file')

(And then yf.download returns an empty dataframe, I believe.)

[MrDvs]

I am encountering the same problem, both on python 3.9 and 3.11. FWIW, the machine I'm running uses Ryzen 5 5600X CPU.

[Himel55]

Not sure if this is related, but I am having a similar issue when using yf.download, I am getting a sqlite3.OperationalError: unable to open database file error. I am running on Windows and I believe it may have something to do with the cache path folder not existing.

If I manually create the py-yfinance folder in C:\Users\Himel\AppData\Local\, I no longer get the exception.

[j-nx]

In my case (windows using Git Bash), I found that this is due to the fact that peewee.py is trying to create a db at %LocalAppData%/py-yfinance/tkr-tz.db and since sqlite3.connect cannot force create the py-yfinance folder this fails.

I manually created the %LocalAppData%/py-yfinance/ folder and was able to bypass this issue, in my instance.

Edit: I see it's been identified already here

[ValueRaider]

@Himel55 That's the problem. Can people try fix in #1698 (instructions #1080)

[federico999999]

but I don't know how to solve it

[The-Milad-A]

The problem is beyond just this. I tried a few other yahoo finance packages & they also couldn't return data. There might be an api issue...

[ValueRaider]

@The-Milad-A That's a different problem. But working for me.

[joshuacuellar1]

I can confirm the error on my Linux environment. The peewee library does not create the py-yfinance folder. Tried your fix @ValueRaider and it worked.

[ValueRaider]

Can I get more confirmation that my refactor hasn't introduced bugs? Then can get a release out before markets open.

[joshuacuellar1]

@ValueRaider Sorry for the late reply. Tested the latest version of yfinance (with your fixes) and it works perfectly in my simple use case: data = ticker.history(period="max", interval="1d", actions=False, auto_adjust=True).

[federico999999]

@ValueRaider please can you explain to me better what code should I use to solve the problem? Because I'm an inexperienced user... and I don't understand what I have to do.

[janfrederik]

@federico999999 The fix is included in the new release 0.2.30. You can just fetch the new version from pypi: pip install --upgrade yfinance if you installed with pip. Or git pull if you cloned this git repo.

[janfrederik]

@ValueRaider thanks for the fast fix!

[federico999999]

@federico999999 The fix is included in the new release 0.2.30. You can just fetch the new version from pypi: pip install --upgrade yfinance if you installed with pip. Or git pull if you cloned this git repo.

Thank you