search

ValueRaider / yfinance-cache

Caching wrapper for yfinance module. Intelligent caching, not dumb caching of web requests.
MIT License
22 stars 9 forks source link

Failed to map these Yahoo intervals to xcal #51

Closed IvanBayan closed 2 weeks ago

IvanBayan commented 3 months ago

I have a simple application with fastapi with single endpoint it fetches historical data and return it in webpack format. I run it few days ago and didn't used it. Now I have tried to fetch data and got error below. Restart has cleared the issue. Have used the latest library installed from git.

ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/var/mail/.local/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 411, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/mail/.local/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 69, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/mail/.local/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/var/mail/.local/lib/python3.11/site-packages/starlette/applications.py", line 123, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/var/mail/.local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in __call__
    raise exc
  File "/var/mail/.local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "/var/mail/.local/lib/python3.11/site-packages/msgpack_asgi/_middleware.py", line 29, in __call__
    await responder(scope, receive, send)
  File "/var/mail/.local/lib/python3.11/site-packages/msgpack_asgi/_middleware.py", line 72, in __call__
    await self.app(scope, self.receive_with_msgpack, self.send_with_msgpack)
  File "/var/mail/.local/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/var/mail/.local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/var/mail/.local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/var/mail/.local/lib/python3.11/site-packages/starlette/routing.py", line 758, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/var/mail/.local/lib/python3.11/site-packages/starlette/routing.py", line 778, in app
    await route.handle(scope, receive, send)
  File "/var/mail/.local/lib/python3.11/site-packages/starlette/routing.py", line 299, in handle
    await self.app(scope, receive, send)
  File "/var/mail/.local/lib/python3.11/site-packages/starlette/routing.py", line 79, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/var/mail/.local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/var/mail/.local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/var/mail/.local/lib/python3.11/site-packages/starlette/routing.py", line 74, in app
    response = await func(request)
               ^^^^^^^^^^^^^^^^^^^
  File "/var/mail/.local/lib/python3.11/site-packages/fastapi/routing.py", line 278, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/mail/.local/lib/python3.11/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/ESP-TickerGraph.py", line 21, in get_history
    hist = getHist(ticker, points)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/ESP-TickerGraph.py", line 28, in getHist
    hist = t.history(period=period, interval=interval)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/mail/.local/lib/python3.11/site-packages/yfinance_cache/yfc_ticker.py", line 233, in history
    h = hist.get(start=None, end=None, period=period, max_age=max_age, trigger_at_market_close=trigger_at_market_close, quiet=quiet)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/mail/.local/lib/python3.11/site-packages/yfinance_cache/yfc_prices_manager.py", line 897, in get
    self._fetchAndAddRanges_sparse(pstr, ranges_to_fetch, prepost, debug_yf, quiet=quiet)
  File "/var/mail/.local/lib/python3.11/site-packages/yfinance_cache/yfc_prices_manager.py", line 1233, in _fetchAndAddRanges_sparse
    h2 = self._fetchYfHistory(pstr, fetch_start, fetch_end, prepost, debug)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/mail/.local/lib/python3.11/site-packages/yfinance_cache/yfc_prices_manager.py", line 2311, in _fetchYfHistory
    accept = click.confirm(msg, default=False)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/mail/.local/lib/python3.11/site-packages/click/termui.py", line 235, in confirm
    raise Abort() from None
click.exceptions.Abort
esp-tickergraph_server-esp-tickergraph-1  |
Failed to map these Yahoo intervals to xcal: (tkr=FIVN, exchange=NGM, xcal=NASDAQ). Normally happens when 'exchange_calendars' is wrong so inform developers.
                               Close  Volume  Dividends  Stock Splits
Datetime
2024-04-03 15:30:00-04:00  60.505001   21274        0.0           0.0
Accept into cache anyway? [y/N]: INFO:     192.168.1.2:51284 - "GET /hist?ticker=FIVN&points=200 HTTP/1.0" 500 Internal Server Error
ValueRaider commented 3 months ago

What was interval?

Did you respond with 'Y' to accept into cache? That would explain error not occurring again.

Run this for me, I get 2024-04-03 16:00:00-04:00:

from yfinance_cache import yfc_cache_manager as yfcm
cal = yfcm.ReadCacheDatum('exchange-NCM', 'cal')
print(cal.schedule['close'].loc['2024-04-03'])
IvanBayan commented 3 months ago

It had happen again. No I didn't respond with anything. The app is running in docker container and not supposed to be run in interactive mode. I can't (don't know how) run it in the same context, but I was able to run it in the same container:

mail@a6ce0ef43f49:/app$ python3
Python 3.11.4 (main, Jul 28 2023, 05:02:22) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from yfinance_cache import yfc_cache_manager as yfcm
>>> cal = yfcm.ReadCacheDatum('exchange-NCM', 'cal')
>>> print(cal.schedule['close'].loc['2024-04-03'])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'NoneType' object has no attribute 'schedule'
>>>
ValueRaider commented 3 months ago

Happening again is good, makes sense. But I can't reproduce so want your help to understand. I'll prepare a debug branch, (and a branch with patch avoiding user input).

IvanBayan commented 3 months ago

Sure, no prob. If you wish I can PM you Dockerfile and the code or can run it by myself.

ValueRaider commented 3 months ago

Yeah email me that.

ValueRaider commented 3 months ago

I can't reproduce error with your codes

localhost:8000/hist?ticker=FIVN&interval=1h&period=10d&points=100

I didn't use Docker.

Can you simplify the code to a bare minimum that reproduces error?

IvanBayan commented 3 months ago

I'm going to run this for a while, probably it be possible to reproduce even without fastapi.

from fastapi import FastAPI, Query
import yfinance_cache as yfc

tickers = {}

app = FastAPI()
# Apply the MessagePack middleware to encode responses in MessagePack format
app.add_middleware(MessagePackMiddleware)

@app.get("/hist")
async def get_history(ticker: str = Query(..., description="Stock ticker"), 
                      period: str = Query(default="5d"),
                      interval: str = Query(default="5m"),
                      ):
    hist = getHist(ticker, period=period, interval = interval)
    return hist.to_dict()

def getHist(ticker, period="5d",interval="5m", tz=None):
    if not tickers.get(ticker):
        tickers[ticker] = yfc.Ticker(ticker)
    t = tickers[ticker]
    hist = t.history(period=period, interval=interval)
    return hist

if __name__ == "__main__":
    import uvicorn
    from uvicorn.config import LOGGING_CONFIG
    LOGGING_CONFIG["formatters"]["default"]["fmt"] = "%(asctime)s [%(name)s] %(levelprefix)s %(message)s"
    LOGGING_CONFIG["formatters"]["access"]["fmt"] = '%(asctime)s [%(name)s] %(levelprefix)s %(client_addr)s - "%(request_line)s" %(status_code)s'
    uvicorn.run("simplify-fastapi:app", host="0.0.0.0", port=8001, reload=True)
IvanBayan commented 2 months ago

I was able to reproduce it with code above, I have started it on 2024-04-06 20:03:02 (utc+1) and it has failed at 2024-04-08 13:35:47,735 (utc+1)

2024-04-08 13:35:47,735 [uvicorn.access] INFO:     172.17.0.1:33442 - "GET /
hist?ticker=NAT HTTP/1.1" 500 Internal Server Error

Failed to map these Yahoo intervals to xcal: (tkr=NAT, exchange=NYQ, xcal=XN
YS). Normally happens when 'exchange_calendars' is wrong so inform developer
s.
                           Close  Volume  Dividends  Stock Splits
Datetime
2024-04-08 09:35:00-04:00    4.0   22557        0.0           0.0
Accept into cache anyway? [y/N]:

It looks like a close correlation with a new trading day of exchange (09:30 EDT or 13:30 UTC+1).

ValueRaider commented 2 months ago

I've created a new debugging branch debug/schedule-mapping, now it will print out its belief of the trading schedule when the mapping fails. Also, you can modify accept to always be True, just check your logs for these messages and post here.

It looks like a close correlation with a new trading day of exchange (09:30 EDT or 13:30 UTC+1).

Are you suggesting that after a few minutes/hours into trading, the error stops occurring?

ValueRaider commented 2 months ago

I've encountered this error myself but on daily intervals not intraday. The cause was using an old version of exchange_calendars - is it up-to-date?

IvanBayan commented 2 months ago

Hello, looks like I have the latest release:

mail@98fd17034aaa:/app$ pip list|grep  exchange_calendars
exchange_calendars    4.5.3

Sorry, didn't saw you previous message. I'm going to try build and run container with your custom debugging branch.

Are you suggesting that after a few minutes/hours into trading, the error stops occurring?

The opposite. Looks like an error happens when the new trading day begins.

IvanBayan commented 2 months ago

An error with debug branch:

2024-04-30 15:47:17,621 [uvicorn.access] INFO:     172.17.0.1:48222 - "GET /hist?ticker=NAT HTTP/1.1" 200 OK
2024-04-30 15:47:47,790 [uvicorn.access] INFO:     172.17.0.1:40840 - "GET /hist?ticker=NAT HTTP/1.1" 200 OK
2024-04-30 15:48:17,905 [uvicorn.access] INFO:     172.17.0.1:48518 - "GET /hist?ticker=NAT HTTP/1.1" 200 OK
2024-04-30 15:48:48,014 [uvicorn.access] INFO:     172.17.0.1:48144 - "GET /hist?ticker=NAT HTTP/1.1" 200 OK
2024-04-30 15:49:18,183 [uvicorn.access] INFO:     172.17.0.1:56134 - "GET /hist?ticker=NAT HTTP/1.1" 200 OK
2024-04-30 15:49:49,107 [uvicorn.access] INFO:     172.17.0.1:52878 - "GET /hist?ticker=NAT HTTP/1.1" 200 OK
WARNING: NAT: No 5m-price data fetched for 2024-04-30 11:50:00-04:00 -> 2024-04-30 11:55:00-04:00
2024-04-30 15:50:19,809 [uvicorn.access] INFO:     172.17.0.1:47254 - "GET /hist?ticker=NAT HTTP/1.1" 200 OK
2024-04-30 15:50:50,343 [uvicorn.access] INFO:     172.17.0.1:33294 - "GET /hist?ticker=NAT HTTP/1.1" 500 Internal Server Error

Failed to map these Yahoo intervals to xcal: (tkr=NAT, exchange=NYQ, xcal=XNYS). Normally happens when 'exchange_calendars' is wrong so inform developers.
                           Close  Volume  Dividends  Stock Splits
Datetime
2024-04-30 11:50:00-04:00  3.885     612        0.0           0.0
- schedule intervals:
[2024-04-30 09:30:00, 2024-04-30 09:35:00)
[2024-04-30 09:35:00, 2024-04-30 09:40:00)
[2024-04-30 09:40:00, 2024-04-30 09:45:00)
[2024-04-30 09:45:00, 2024-04-30 09:50:00)
[2024-04-30 09:50:00, 2024-04-30 09:55:00)
[2024-04-30 09:55:00, 2024-04-30 10:00:00)
[2024-04-30 10:00:00, 2024-04-30 10:05:00)
[2024-04-30 10:05:00, 2024-04-30 10:10:00)
[2024-04-30 10:10:00, 2024-04-30 10:15:00)
[2024-04-30 10:15:00, 2024-04-30 10:20:00)
[2024-04-30 10:20:00, 2024-04-30 10:25:00)
[2024-04-30 10:25:00, 2024-04-30 10:30:00)
[2024-04-30 10:30:00, 2024-04-30 10:35:00)
[2024-04-30 10:35:00, 2024-04-30 10:40:00)
[2024-04-30 10:40:00, 2024-04-30 10:45:00)
[2024-04-30 10:45:00, 2024-04-30 10:50:00)
[2024-04-30 10:50:00, 2024-04-30 10:55:00)
[2024-04-30 10:55:00, 2024-04-30 11:00:00)
[2024-04-30 11:00:00, 2024-04-30 11:05:00)
[2024-04-30 11:05:00, 2024-04-30 11:10:00)
[2024-04-30 11:10:00, 2024-04-30 11:15:00)
[2024-04-30 11:15:00, 2024-04-30 11:20:00)
[2024-04-30 11:20:00, 2024-04-30 11:25:00)
[2024-04-30 11:25:00, 2024-04-30 11:30:00)
[2024-04-30 11:30:00, 2024-04-30 11:35:00)
[2024-04-30 11:35:00, 2024-04-30 11:40:00)
[2024-04-30 11:40:00, 2024-04-30 11:45:00)
[2024-04-30 11:45:00, 2024-04-30 11:50:00)
Accept into cache anyway? [y/N]: 2024-04-30 15:51:20,862 [uvicorn.access] INFO:     172.17.0.1:45758 - "GET /hist?ticker=NAT HTTP/1.1" 500 Internal Server Error

Failed to map these Yahoo intervals to xcal: (tkr=NAT, exchange=NYQ, xcal=XNYS). Normally happens when 'exchange_calendars' is wrong so inform developers.

And exception:

Traceback (most recent call last):
  File "/var/mail/.local/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 411, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/mail/.local/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 69, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/mail/.local/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/var/mail/.local/lib/python3.11/site-packages/starlette/applications.py", line 123, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/var/mail/.local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in __call__
    raise exc
  File "/var/mail/.local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "/var/mail/.local/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 65, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/var/mail/.local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc

    await route.handle(scope, receive, send)
  File "/var/mail/.local/lib/python3.11/site-packages/starlette/routing.py", line 297, in handle
    await self.app(scope, receive, send)
  File "/var/mail/.local/lib/python3.11/site-packages/starlette/routing.py", line 77, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/var/mail/.local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/var/mail/.local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/var/mail/.local/lib/python3.11/site-packages/starlette/routing.py", line 72, in app
    response = await func(request)
               ^^^^^^^^^^^^^^^^^^^
  File "/var/mail/.local/lib/python3.11/site-packages/fastapi/routing.py", line 278, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/mail/.local/lib/python3.11/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/simplify-fastapi.py", line 15, in get_history
    hist = getHist(ticker, period=period, interval = interval)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/simplify-fastapi.py", line 22, in getHist
    hist = t.history(period=period, interval=interval)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/mail/.local/lib/python3.11/site-packages/yfinance_cache/yfc_ticker.py", line 233, in history
    h = hist.get(start=None, end=None, period=period, max_age=max_age, trigger_at_market_close=trigger_at_market_close, quiet=quiet)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/mail/.local/lib/python3.11/site-packages/yfinance_cache/yfc_prices_manager.py", line 897, in get
    self._fetchAndAddRanges_sparse(pstr, ranges_to_fetch, prepost, debug_yf, quiet=quiet)
  File "/var/mail/.local/lib/python3.11/site-packages/yfinance_cache/yfc_prices_manager.py", line 1233, in _fetchAndAddRanges_sparse
    h2 = self._fetchYfHistory(pstr, fetch_start, fetch_end, prepost, debug)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/mail/.local/lib/python3.11/site-packages/yfinance_cache/yfc_prices_manager.py", line 2317, in _fetchYfHistory
    accept = click.confirm(msg, default=False)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/mail/.local/lib/python3.11/site-packages/click/termui.py", line 235, in confirm
    raise Abort() from None
click.exceptions.Abort
ValueRaider commented 2 months ago

Brilliant. I may have seen problem, leave this with me.

ValueRaider commented 2 months ago

My suspicion was this LoC https://github.com/ValueRaider/yfinance-cache/blob/67ed0a51af2eec6f5316fbb923774ecc15a92cf6/yfinance_cache/yfc_time.py#L610 was accidentally discarding the last interval from schedule, but I'm having trouble reproducing your error.

1) Try latest branch, it tests my idea. has more debug logging

2) Is your system time correct? Not shifted by a few minutes?

IvanBayan commented 2 months ago

Sorry, I have missed your comment again, thanks for update. I've re-built image from the latest commit in debug branch and left it running for a test. Will let you know if it failed again or if it be working without issues for a few days. System time should be correct:

 % chronyc tracking
Reference ID    : C1889848 (ntp2.tecnico.ulisboa.pt)
Stratum         : 3
Ref time (UTC)  : Tue May 07 12:07:41 2024
System time     : 0.000727860 seconds slow of NTP time
Last offset     : -0.000454845 seconds
RMS offset      : 0.000245534 seconds
Frequency       : 5.152 ppm slow
Residual freq   : -0.001 ppm
Skew            : 0.025 ppm
Root delay      : 0.022814699 seconds
Root dispersion : 0.010683046 seconds
Update interval : 1036.6 seconds
Leap status     : Normal

Thou container have default timezone (UTC) which is differ from system timezone.

IvanBayan commented 2 months ago

Has failed again:

Failed to map these Yahoo intervals to xcal: (tkr=NAT, exchange=NYQ, xcal=XNYS). Normally happens when 'exchange_calendars' is wrong so inform developers.
                            Close  Volume  Dividends  Stock Splits
Datetime
2024-05-07 09:35:00-04:00  3.9927    5265        0.0           0.0
- schedule intervals:
[2024-05-07 09:30:00, 2024-05-07 09:35:00)
Accept into cache anyway? [y/N]: 2024-05-08 11:29:32,974 [uvicorn.access] INFO:     172.17.0.1:43048 - "GET /hist?ticker=NAT HTTP/1.1" 500 Internal Server Error
2024-05-08 11:29:32,974 [uvicorn.error] ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/var/mail/.local/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 411, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/mail/.local/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 69, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/mail/.local/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/var/mail/.local/lib/python3.11/site-packages/starlette/applications.py", line 123, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/var/mail/.local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in __call__
    raise exc
  File "/var/mail/.local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "/var/mail/.local/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 65, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/var/mail/.local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/var/mail/.local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/var/mail/.local/lib/python3.11/site-packages/starlette/routing.py", line 756, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/var/mail/.local/lib/python3.11/site-packages/starlette/routing.py", line 776, in app
    await route.handle(scope, receive, send)
  File "/var/mail/.local/lib/python3.11/site-packages/starlette/routing.py", line 297, in handle
    await self.app(scope, receive, send)
  File "/var/mail/.local/lib/python3.11/site-packages/starlette/routing.py", line 77, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/var/mail/.local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/var/mail/.local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/var/mail/.local/lib/python3.11/site-packages/starlette/routing.py", line 72, in app
    response = await func(request)
               ^^^^^^^^^^^^^^^^^^^
  File "/var/mail/.local/lib/python3.11/site-packages/fastapi/routing.py", line 278, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/mail/.local/lib/python3.11/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/simplify-fastapi.py", line 15, in get_history
    hist = getHist(ticker, period=period, interval = interval)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/simplify-fastapi.py", line 22, in getHist
    hist = t.history(period=period, interval=interval)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/mail/.local/lib/python3.11/site-packages/yfinance_cache/yfc_ticker.py", line 233, in history
    h = hist.get(start=None, end=None, period=period, max_age=max_age, trigger_at_market_close=trigger_at_market_close, quiet=quiet)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/mail/.local/lib/python3.11/site-packages/yfinance_cache/yfc_prices_manager.py", line 897, in get
    self._fetchAndAddRanges_sparse(pstr, ranges_to_fetch, prepost, debug_yf, quiet=quiet)
  File "/var/mail/.local/lib/python3.11/site-packages/yfinance_cache/yfc_prices_manager.py", line 1233, in _fetchAndAddRanges_sparse
    h2 = self._fetchYfHistory(pstr, fetch_start, fetch_end, prepost, debug)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/mail/.local/lib/python3.11/site-packages/yfinance_cache/yfc_prices_manager.py", line 2317, in _fetchYfHistory
    accept = click.confirm(msg, default=False)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/mail/.local/lib/python3.11/site-packages/click/termui.py", line 235, in confirm
    raise Abort() from None
click.exceptions.Abort
ValueRaider commented 2 months ago

I wrongly thought you had logging enabled. Add this:

import logging
logging.getLogger('yfinance_cache').setLevel(logging.DEBUG)
logging.basicConfig(level=logging.DEBUG, format='%(asctime)s - %(name)s - %(levelname)s - %(message)s')

You should see a lot more messages from YFC.

ValueRaider commented 1 month ago

I'll start frequently running this fetch myself to try and reproduce, avoid this back-and-forth.

yfc.Ticker('NAT').history(interval='5m', period='1wk')

IvanBayan commented 1 month ago

Ok, but I've captured an exception with enabled debug logging. Tell me if you need it.

ValueRaider commented 1 month ago

Go ahead.

IvanBayan commented 1 month ago

yfc.log

ValueRaider commented 1 month ago

Confirms my suspicion in right area - GetExchangeScheduleIntervals isn't returning intervals for 13th May despite being called after market open. I've added even more logging, and tweaked it so enable this way (after pulling latest):

from yfinance_cache import yfc_logging as yfcl
import logging
yfcl.EnableLogging(logging.DEBUG)

If you get error TypeError: EnableLogging() takes 0 ... then you haven't pulled latest.

Run as normally, then when error occurs send me this file:

~/.cache/py-yfinance-cache/exchange-NYQ/events.log

IvanBayan commented 1 month ago

events.log stdout_stderr.log

ValueRaider commented 1 month ago

Thanks! I spotted a bug in schedule caching that would explain error, I've pushed a hopeful fix. Probably can't confirm fixed until markets open again.

IvanBayan commented 3 weeks ago

Hello, I have been running test for 2 weeks and didn't observe any exception. I think the issue is fixed. Thank you.