problem with today's data

[darekkot]

When you call today (10/10/2022): data2 = historical_data(investing_id=curr_id, from_date=st_date, to_date=end_date)

where: investing_id=1166004,from_date=10/01/2022,to_date=10/10/2022

you'll get json: {'date': ['10/03/2022', '10/04/2022', '10/05/2022', '10/06/2022', '10/07/2022'], 'open': [7.0425000190735, 7.1424999237061, 7.2750000953674, 7.352499961853, 7.2849998474121], 'high': [7.0875000953674, 7.272500038147, 7.3274998664856, 7.3825001716614, 7.3550000190735], 'low': [7.0349998474121, 7.1199998855591, 7.2224998474121, 7.3049998283386, 7.2849998474121], 'close': [7.0599999427795, 7.2674999237061, 7.3274998664856, 7.335000038147, 7.3474998474121], 'volume': [1557937, 169883, 401434, 215784, 275430]}

so there are data for period: 10/03/2022 ... 10/07/2022. There is no data for 10/10/2022!

when you call it without to_date parameter

data = historical_data(investing_id=curr_id, 
                       from_date=st_date
                       ) 

, you'll get:

{'date': ['09/12/2022', '09/13/2022', '09/14/2022', '09/15/2022', '09/16/2022', '09/20/2022', '09/21/2022', '09/22/2022', '09/23/2022', '09/26/2022', '09/27/2022', '09/28/2022', '09/29/2022', '09/30/2022', '10/03/2022', '10/04/2022', '10/05/2022', '10/06/2022', '10/07/2022', '10/10/2022'], 'open': [7.4499998092651, 7.6100001335144, 7.5300002098083, 7.5900001525879, 7.3850002288818, 7.3499999046326, 7.3000001907349, 7.3425002098083, 7.2649998664856, 6.9800000190735, 7.0174999237061, 6.897500038147, 7.0799999237061, 7.0900001525879, 7.0425000190735, 7.1424999237061, 7.2750000953674, 7.352499961853, 7.2849998474121, 7.3600001335144], 'high': [7.5725002288818, 7.6599998474121, 7.6275000572205, 7.5999999046326, 7.3949999809265, 7.3899998664856, 7.4800000190735, 7.4099998474121, 7.2649998664856, 7.0549998283386, 7.039999961853, 7.0199999809265, 7.1574997901917, 7.1399998664856, 7.0875000953674, 7.272500038147, 7.3274998664856, 7.3825001716614, 7.3550000190735, 7.3800001144409], 'low': [7.4499998092651, 7.5275001525879, 7.4899997711182, 7.3699998855591, 7.3074998855591, 7.3150000572205, 7.3000001907349, 7.3249998092651, 7.0475001335144, 6.9749999046326, 6.9875001907349, 6.8775000572205, 7.0174999237061, 7.039999961853, 7.0349998474121, 7.1199998855591, 7.2224998474121, 7.3049998283386, 7.2849998474121, 7.3400001525879], 'close': [7.5650000572205, 7.5500001907349, 7.5749998092651, 7.4050002098083, 7.3625001907349, 7.3850002288818, 7.3375000953674, 7.3474998474121, 7.0650000572205, 6.9825000762939, 7.0050001144409, 6.9899997711182, 7.0700001716614, 7.0900001525879, 7.0599999427795, 7.2674999237061, 7.3274998664856, 7.335000038147, 7.3474998474121, 7.3800001144409], 'volume': [127182, 172399, 2245501, 107892, 143964, 265991, 547998, 227319, 201673, 541426, 511374, 251056, 272732, 7209297, 1557937, 169883, 401434, 215784, 275430, 8015]}

so now, probably both parameters are ignored and in such case there is the data for the whole month with todays data.

When I request data till today, I should get todays data as well. Could you please take a look on this?

[alvarobartt]

Hi @darekkot, I'll check this! Even though this is part of the branch that I'm working on named data-limit which I'll release soon hopefully and already fixes that as all the datetimes are converted to UTC to avoid timezone issues, which I assume it's related to the bug that you're reporting...

Install it as pip install git+https://github.com/alvarobartt/investiny.git@data-limit 🤗

[alvarobartt]

Labeling this as a bug as this is potentially related to #24 and, so on, should be solved in #27

[darekkot]

Hi @alvarobartt, I've installed the version you advised however, the problem remain

Here is an output from my test for the newest version and parameters as above.

Test with both parameters:

{'date': ['10/03/2022', '10/04/2022', '10/05/2022', '10/06/2022', '10/07/2022'], 'open': [7.0425000190735, 7.1424999237061, 7.2750000953674, 7.352499961853, 7.2849998474121], 'high': [7.0875000953674, 7.272500038147, 7.3274998664856, 7.3825001716614, 7.3550000190735], 'low': [7.0349998474121, 7.1199998855591, 7.2224998474121, 7.3049998283386, 7.2849998474121], 'close': [7.0599999427795, 7.2674999237061, 7.3274998664856, 7.335000038147, 7.3474998474121], 'volume': [1557937, 169883, 401434, 215784, 275430]}

Test with only 1 parameter: {'date': ['09/12/2022', '09/13/2022', '09/14/2022', '09/15/2022', '09/16/2022', '09/20/2022', '09/21/2022', '09/22/2022', '09/23/2022', '09/26/2022', '09/27/2022', '09/28/2022', '09/29/2022', '09/30/2022', '10/03/2022', '10/04/2022', '10/05/2022', '10/06/2022', '10/07/2022', '10/10/2022'], 'open': [7.4499998092651, 7.6100001335144, 7.5300002098083, 7.5900001525879, 7.3850002288818, 7.3499999046326, 7.3000001907349, 7.3425002098083, 7.2649998664856, 6.9800000190735, 7.0174999237061, 6.897500038147, 7.0799999237061, 7.0900001525879, 7.0425000190735, 7.1424999237061, 7.2750000953674, 7.352499961853, 7.2849998474121, 7.3600001335144], 'high': [7.5725002288818, 7.6599998474121, 7.6275000572205, 7.5999999046326, 7.3949999809265, 7.3899998664856, 7.4800000190735, 7.4099998474121, 7.2649998664856, 7.0549998283386, 7.039999961853, 7.0199999809265, 7.1574997901917, 7.1399998664856, 7.0875000953674, 7.272500038147, 7.3274998664856, 7.3825001716614, 7.3550000190735, 7.3800001144409], 'low': [7.4499998092651, 7.5275001525879, 7.4899997711182, 7.3699998855591, 7.3074998855591, 7.3150000572205, 7.3000001907349, 7.3249998092651, 7.0475001335144, 6.9749999046326, 6.9875001907349, 6.8775000572205, 7.0174999237061, 7.039999961853, 7.0349998474121, 7.1199998855591, 7.2224998474121, 7.3049998283386, 7.2849998474121, 7.3400001525879], 'close': [7.5650000572205, 7.5500001907349, 7.5749998092651, 7.4050002098083, 7.3625001907349, 7.3850002288818, 7.3375000953674, 7.3474998474121, 7.0650000572205, 6.9825000762939, 7.0050001144409, 6.9899997711182, 7.0700001716614, 7.0900001525879, 7.0599999427795, 7.2674999237061, 7.3274998664856, 7.335000038147, 7.3474998474121, 7.3699998855591], 'volume': [127182, 172399, 2245501, 107892, 143964, 265991, 547998, 227319, 201673, 541426, 511374, 251056, 272732, 7209297, 1557937, 169883, 401434, 215784, 275430, 22599]}

Let me know if you want me to perform some other test.

[alvarobartt]

Ok thanks for the detailed explanation @darekkot, I'll explore this further today and try to provide a solution! 👍🏻

[ramakrishnamekala129]

i know this problem we just need to add extra date suppose we today is 10 october if we want from today to past month then historical_data(investing_id=curr_id, from_date=st_date, to_date=end_date) just add timedelta(days=1)

[ramakrishnamekala129]

it is frequent problem in multiple broker in india historical data

[darekkot]

It's rather workaround. If I want data till today then I should specify today as to_date because it's logical :).

[ramakrishnamekala129]

[{'symbol': 'NSEBANK', 'full_name': 'NSE:NSEBANK', 'description': 'Nifty Bank', 'type': 'Index', 'ticker': '17950', 'exchange': 'NSE'}] {'date': ['10/04/2022', '10/06/2022', '10/07/2022', '10/10/2022'], 'open': [38700.3984375, 39343.69921875, 39093.5, 38641.55078125], 'high': [39177.5, 39608.3515625, 39235.05078125, 39315.69921875], 'low': [38596.8984375, 39232.8984375, 38807.5, 38518.30078125], 'close': [39110.05078125, 39282.8515625, 39178.05078125, 39093.1015625]} open high low close date 2022-10-04 38700.398438 39177.500000 38596.898438 39110.050781 2022-10-06 39343.699219 39608.351562 39232.898438 39282.851562 2022-10-07 39093.500000 39235.050781 38807.500000 39178.050781 2022-10-10 38641.550781 39315.699219 38518.300781 39093.101562

[ramakrishnamekala129]

data = historical_data(investing_id=investing_id, from_date="10/04/2022", to_date="10/11/2022") u can try

[alvarobartt]

I've already tried and it's working fine for me... see the output below:

Maybe it's because the Indian Market was not open yet and there was no data available on Investing.com.

[alvarobartt]

BTW I'll fix the check on the dates, as currently it's just taken into consideration if both are provided, but what I want to do is to make to_date not mandatory when from_date has a value, so default to_date is current date :+1:

[darekkot]

@alvarobartt , Firstly the to_date in your test is set on 10/11/2022 when it should be 10/10/2022. Secondly, I tested it for investing_id=1166004 when the market was opened

If you want to check it now for opened market, please check for example 532 which is vti ticker.

[darekkot]

@alvarobartt, I've just repeated it for the parameters as below: investing_id=1166004,from_date=10/01/2022,to_date=10/10/2022

and the result is still the same:

{'date': ['10/03/2022', '10/04/2022', '10/05/2022', '10/06/2022', '10/07/2022'], 'open': [7.0425000190735, 7.1424999237061, 7.2750000953674, 7.352499961853, 7.2849998474121], 'high': [7.0875000953674, 7.272500038147, 7.3274998664856, 7.3825001716614, 7.3550000190735], 'low': [7.0349998474121, 7.1199998855591, 7.2224998474121, 7.3049998283386, 7.2849998474121], 'close': [7.0599999427795, 7.2674999237061, 7.3274998664856, 7.335000038147, 7.3474998474121], 'volume': [1557937, 169883, 401434, 215784, 275430]}

One more thing. It's a good idea to set by default to_date on today however it'll not solve the problem as when you set the parameters as: investing_id=1166004,from_date=10/01/2022,to_date=10/07/2022

then, you're getting data as follows:

{'date': ['10/03/2022', '10/04/2022', '10/05/2022', '10/06/2022'], 'open': [7.0425000190735, 7.1424999237061, 7.2750000953674, 7.352499961853], 'high': [7.0875000953674, 7.272500038147, 7.3274998664856, 7.3825001716614], 'low': [7.0349998474121, 7.1199998855591, 7.2224998474121, 7.3049998283386], 'close': [7.0599999427795, 7.2674999237061, 7.3274998664856, 7.335000038147], 'volume': [1557937, 169883, 401434, 215784]}

so in this case there is no data for 10/07/2022.

For me it looks like you select date where transaction_date < to_date when as I see it, it should be: transactiion_date <= to_date

[alvarobartt]

@darekkot so to_date will default to the current date, and investiny is converting the date to timestamp and sending the request with that information, but it may be due to the UTC conversion on the dates... Let me carefully check during this week 👍🏻

[alvarobartt]

Hi both @darekkot and @ramakrishnamekala129, I didn't get to set the default to_date's value as the current date, but I've already fixed the issue that you are describing since the dates formatted as %m/%d/%Y were also transformed to UTC timezone, which means that if you're in India your timezone is UTC+5:30, so that when converting e.g. 01/01/2022 to UTC investiny was converting that to UTC so that the calculated date was 01/01/2022 00:00 - 05:30 which was 12/31/2021 18:30

[darekkot]

I tested this bug again with the latest build and the problem still exists.

Steps to replicate the problem:

1. set the date to 10/14/2022 19:00:00 (I'm currently in Spain, so the timezone is the same)
2. call the code below:

from investiny import historical_data
curr_id=14178
st_date="04/30/2022"
end_date="10/14/2022"
print(f'investing_id={curr_id},from_date={st_date},to_date={end_date}')
data = historical_data(investing_id=curr_id, 
                        from_date=st_date,
                        to_date=end_date) 

last = len(data['date']) - 1
print(f"last date, when to_date is specified:{data['date'][last]}")
data2 = historical_data(investing_id=curr_id, 
                        from_date=st_date
                        ) 

last = len(data2['date']) - 1
print(f"last date, when no to_date is specified:{data2['date'][last]}") 

I expected, that for both: data and data2 I should get the same set of data from 04/30/2022 ... 10/14/2022(included) but here is an output from execution of the code above:

investing_id=14178,from_date=04/30/2022,to_date=10/14/2022 last date, when to_date is specified:10/13/2022 last date, when no to_date is specified:10/14/2022

[alvarobartt]

Let me test that again and add the required unit tests so as to avoid this for happening... Also FYI the timezone in Spain is UTC+2, while the timezone used in investiny is UTC.

Also, in order to get the last date from the retrieved data you can just data["date"][-1] 😄

[alvarobartt]

Bug fix version already released at https://github.com/alvarobartt/investiny/releases/tag/v0.6.1, please @darekkot install it as it follows pip install investiny --upgrade or pip install investiny==0.6.1 and let me know whether that worked for you or not! 😄

[darekkot]

I updated investiny to the latest version and I can confirm, that the bug is fixed. I'll check out one more time on Monday how it behaves during the trading day but so far so good. Thanks for you effort during the weekend.

[alvarobartt]

Hi again @darekkot it seems that while trying to solve this I introduced a bug, it seems that just some financial product dates are shifted by one day, and the current fix will fail for some specific scenarios when the from_date value is either a Saturday, Sunday, bank holiday, or market closed day. So on I'll push a patch so as to solve that and just shift by one day the bonds, which seems to be the only asset that provides wrong dates.