reqHistoricalTicks doesn't properly accommodate startDateTime

[leongold]

Hey!

First things first -- great project.

Ever since Jan 1st 2020 it seems like startDateTime misbehaves (which of course means that if there's indeed a problem, it's on IB's side, however I wanted your thoughts first).

I am using the idiom of fetching ticks using startDateTime (and numberOfTicks set to 1000). The last datetime in any given batch becomes the new startDateTime (unless they're equal, in that case I manually increment it by 10 minutes). This cycle is repeated until the last datetime equals exceeds my overall required datetime.

I have made no modifications to this code in a long time.

Now, however, although startDateTime is increased, the fetched results remain the same:

metis_1        | 2020-01-04 08:30:19,156 [INFO] apis.ib: _req_historical_ticks ('Bid_Ask', 'SMART', 'AAPL', 'USD'): fetching 2019-12-17 15:11:13+00:00
metis_1        | 2020-01-04 08:30:21,102 [INFO] apis.ib: _req_historical_ticks ('Bid_Ask', 'SMART', 'AAPL', 'USD'): fetched 2019-12-17 15:11:13+00:00 -- 2019-12-17 14:29:59+00:00 => 2019-12-17 14:31:13+00:00 (1004)
metis_1        | 2020-01-04 08:30:21,103 [INFO] apis.ib: _req_historical_ticks ('Bid_Ask', 'SMART', 'AAPL', 'USD'): last_dt: 2019-12-17 15:11:13+00:00, prev_last_dt: 2019-12-17 15:11:13+00:00, eq: True
metis_1        | 2020-01-04 08:30:21,103 [INFO] apis.ib: _req_historical_ticks ('Bid_Ask', 'SMART', 'AAPL', 'USD'): incrementing last_dt by 10 mins: 2019-12-17 15:11:13+00:00 => 2019-12-17 15:21:13+00:00
metis_1        | 2020-01-04 08:30:21,103 [INFO] apis.ib: _req_historical_ticks ('Bid_Ask', 'SMART', 'AAPL', 'USD'): new last_dt: 2019-12-17 15:21:13+00:00
metis_1        | 2020-01-04 08:30:21,103 [INFO] apis.ib: _req_historical_ticks ('Bid_Ask', 'SMART', 'AAPL', 'USD'): fetching 2019-12-17 15:21:13+00:00
metis_1        | 2020-01-04 08:30:23,251 [INFO] apis.ib: _req_historical_ticks ('Bid_Ask', 'SMART', 'AAPL', 'USD'): fetched 2019-12-17 15:21:13+00:00 -- 2019-12-17 14:29:59+00:00 => 2019-12-17 14:31:13+00:00 (1004)
metis_1        | 2020-01-04 08:30:23,252 [INFO] apis.ib: _req_historical_ticks ('Bid_Ask', 'SMART', 'AAPL', 'USD'): last_dt: 2019-12-17 15:21:13+00:00, prev_last_dt: 2019-12-17 15:11:13+00:00, eq: False
metis_1        | 2020-01-04 08:30:23,252 [INFO] apis.ib: _req_historical_ticks ('Bid_Ask', 'SMART', 'AAPL', 'USD'): fetching 2019-12-17 15:21:13+00:00
metis_1        | 2020-01-04 08:30:25,298 [INFO] apis.ib: _req_historical_ticks ('Bid_Ask', 'SMART', 'AAPL', 'USD'): fetched 2019-12-17 15:21:13+00:00 -- 2019-12-17 14:29:59+00:00 => 2019-12-17 14:31:13+00:00 (1004)
metis_1        | 2020-01-04 08:30:25,299 [INFO] apis.ib: _req_historical_ticks ('Bid_Ask', 'SMART', 'AAPL', 'USD'): last_dt: 2019-12-17 15:21:13+00:00, prev_last_dt: 2019-12-17 15:21:13+00:00, eq: True
metis_1        | 2020-01-04 08:30:25,299 [INFO] apis.ib: _req_historical_ticks ('Bid_Ask', 'SMART', 'AAPL', 'USD'): incrementing last_dt by 10 mins: 2019-12-17 15:21:13+00:00 => 2019-12-17 15:31:13+00:00
metis_1        | 2020-01-04 08:30:25,299 [INFO] apis.ib: _req_historical_ticks ('Bid_Ask', 'SMART', 'AAPL', 'USD'): new last_dt: 2019-12-17 15:31:13+00:00
metis_1        | 2020-01-04 08:30:25,299 [INFO] apis.ib: _req_historical_ticks ('Bid_Ask', 'SMART', 'AAPL', 'USD'): fetching 2019-12-17 15:31:13+00:00

Despite startDateTime (last_dt), the fetched results range from 2019-12-17 14:29:59+00:00 through 2019-12-17 14:31:13+00:00

I'm using IB GW 972 & ib_insync 0.9.53

Any help is greatly appreciated.

[leongold]

As a summary, what I'd like to know is whether or not anyone else experiences consecutive calls to reqHistoricalTicks using startDateTime to be not working as expected while incrementing startDateTime after each batch -- to me it seems like any consecutive calls ignore any new startDateTimes and continues to serve requests using the initial startDateTime.

I know this sounds like a silly bug in my code, and I don't rule it out, however I'm 90% sure I made no modifications to any relevant code, and that this broken behavior started happening in 2020.

Here's a code snippet:

        while last_dt < end_dt:
            ticks = self._ib.reqHistoricalTicks(
                Stock(s0, e, s1),
                last_dt,
                "",
                1000,
                what_to_show,
                True,
                True
            )

            if not ticks:
                break

            for tick in ticks:
                dt = tick.time

                if dt <= start_dt or dt < last_dt:
                    continue

                last_dt = dt

                if last_dt >= end_dt:
                    break

                entity = convert_func(tick, e, s0, s1)
                entities.append(entity)

            if prev_last_dt >= last_dt:
                last_dt = prev_last_dt + datetime.timedelta(minutes=10)
            else:
                prev_last_dt = last_dt

[erdewit]

reqHistoricalTicks runs fine for me. Note that there's no data for Jan 1 and perhaps that messes up your looping logic. Try it out with something simpler, like

import datetime
from ib_insync import *

ib = IB().connect()
eurusd = Forex('EURUSD')
start = datetime.datetime(2020, 1, 3, tzinfo=datetime.timezone.utc)
end = ''
ticks = ib.reqHistoricalTicks(eurusd, start, end, 1000, 'BID_ASK', useRth=False)

[leongold]

Hey @erdewit, I figured out the issue and I'd like your input if you don't mind.

I noticed that after 2 hours I began receiving proper results -- since I use useRth, I immediately suspected there's a timezone issue where I'm unknowingly querying for a time period that's outside rth.

I pass UTC aware datetime.datetime objects as startDateTime, causing it to be formatted to a local machine time. My system is distributed across several data centers, and since the service fetching ticks is in a different timezone than the service hosting IB GW, there's a gap (coincidently it was moved to a different timezone very recently, causing me to suspect a bug2k20 :))

def formatIBDatetime(dt: Union[date, datetime, str, None]) -> str:
    """Format date or datetime to string that IB uses."""
    if not dt:
        s = ''
    elif isinstance(dt, datetime):
        if dt.tzinfo:
            # convert to local system timezone  <---- creates a gap.
            dt = dt.astimezone()
        s = dt.strftime('%Y%m%d %H:%M:%S')
    elif isinstance(dt, date):
        s = dt.strftime('%Y%m%d 23:59:59')
    else:
        s = dt
    return s

My plan is to stop passing aware datetime objects unless you have a better idea how to tackle this.

[erdewit]

If the gateway has a different timezone from the macine were the API client is running then there is indeed a problem, since the gateway's timezone can't be obtained via the API.

My prefered approach is to have everything in UTC. Using a tz-naive datetime will work in this specific case too.