history start arg does not behave as expected for 24hr instruments

[maread99]

@maread99 @dpguthrie I'm not sure this fixes the problem (or indeed manifests a new one). Implementing the workaround has caused history to look 1 day (I assume whatever period you use) further back. e.g. t.history(start='2023-02-22', end=None, interval='1d')

symbol date ... BTC-USD 2023-02-21 ...

EDIT: It seems this error only occurs with tickers that daily close/open at midnight. For me, I've confirmed this with both crypto (BTC-USD) and FX (AUDUSD=X).

All other tickers I'm tracking are on the ASX which opens at 10:00 and closes at 16:00 AEDT, and for these, the history function returns the correct data. From a purely speculative observation, it looks as if close times on 24hr tickers are marked as 00:00:00 for time of closing (which is actually 24hrs prior).

Originally posted by @benhowell in https://github.com/dpguthrie/yahooquery/issues/142#issuecomment-1445600786

[maread99]

Hi @benhowell

Thanks for the comment. I've raised it as a separate issue here.

I'm getting the data back as expected and am unable to recreate the behavior you describe (even after accounting for the possibility that it might be related to different user' timezones).

Could you execute the following and post a reply with everything that's printed. Thanks.

import yahooquery as yq
import pandas as pd
print(yq.__version__)
print(pd.Timestamp.now())

ticker = yq.Ticker("BTC-USD")
df = ticker.history(start='2023-02-22', end=None, interval='1d')
print(df)

[benhowell]

Hi @maread99,

This is the change I made from advice in #142 (utils/init.py): last_trade = pd.Timestamp.fromtimestamp(timestamp, tz="UTC") #last_trade = pd.Timestamp.fromtimestamp(timestamp) #last_trade = last_trade.tz_localize("UTC")

Here's the output, in full (apologies for the formatting):

Python 3.10.6 (main, Nov 14 2022, 16:10:14) [GCC 11.3.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import yahooquery as yq >>> import pandas as pd >>> print(yq.version) 2.3.0 >>> print(pd.Timestamp.now()) 2023-02-28 09:50:20.398458 >>> >>> ticker = yq.Ticker("BTC-USD") >>> df = ticker.history(start='2023-02-22', end=None, interval='1d') >>> print(df) open high low close volume adjclose symbol date BTC-USD 2023-02-21 24833.048828 25126.851562 24200.363281 24436.353516 31252098714 24436.353516 2023-02-22 24437.417969 24472.339844 23644.318359 24188.843750 30199996781 24188.843750 2023-02-23 24190.718750 24572.089844 23693.919922 23947.492188 30476264066 23947.492188 2023-02-24 23946.007812 24103.705078 23007.072266 23198.126953 26811744928 23198.126953 2023-02-25 23200.125000 23210.210938 22861.558594 23175.375000 16100721565 23175.375000 2023-02-26 23174.150391 23654.367188 23084.220703 23561.212891 16644534842 23561.212891 2023-02-27 22:48:00+00:00 23551.751953 23848.814453 23208.105469 23414.634766 22782625792 23414.634766

[maread99]

I suspect your issue is the same as #49. This should be fixed within the #141 PR by this change to the _convert_to_timestamp function. The PR was merged subsequent to the most recent PyPI release (2.3.0).

To implement the fix now either change / overwrite the _convert_to_timestamp function locally or, preferably to get all the fixes, just install from the master branch. Assuming you've got git installed then the following should work:

$ pip install git+https://github.com/dpguthrie/yahooquery.git#egg=yahooquery

Let me know if that fixes it.

[benhowell]

Thanks @maread99, The pip install above didn't update the changes made with #141.

I made the change myself and replaced the _convert_to_timestamp fn and it looks to have fixed the problem. So there's another positive test at least.

Thanks for taking the time to look into it, much appreciated.

output: Python 3.10.6 (main, Nov 14 2022, 16:10:14) [GCC 11.3.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import yahooquery as yq >>> import pandas as pd >>> print(yq.version) 2.3.0 >>> print(pd.Timestamp.now()) 2023-02-28 11:12:31.602429 >>> ticker = yq.Ticker("BTC-USD") >>> df = ticker.history(start='2023-02-22', end=None, interval='1d') >>> print(df) open high low close volume adjclose symbol date BTC-USD 2023-02-22 24437.417969 24472.339844 23644.318359 24188.843750 3.020000e+10 24188.843750 2023-02-23 24190.718750 24572.089844 23693.919922 23947.492188 3.047626e+10 23947.492188 2023-02-24 23946.007812 24103.705078 23007.072266 23198.126953 2.681174e+10 23198.126953 2023-02-25 23200.125000 23210.210938 22861.558594 23175.375000 1.610072e+10 23175.375000 2023-02-26 23174.150391 23654.367188 23084.220703 23561.212891 1.664453e+10 23561.212891 2023-02-28 00:11:00+00:00 23512.177734 23512.177734 23496.955078 23496.955078 2.256071e+10 23496.955078

[maread99]

No worries, glad it worked @benhowell.

If you're using yahooquery for price data then you might be interested in market_prices. It sits on top of yahooquery and provides for the likes of enhanced querying, post processing and caching.

[benhowell]

Thanks @maread99, I'm using a lot of the functionality in yq besides ohlcv, e.g. divs, splits, insider_transactions, insider_holders, news, screeners for top gainers, losers, etc.