Tick data from bdtick starts 1 minute late

[DS-London]

I am in London (unsurprisingly) so GMT+1 at the time of writing. So NYC is GMT - 4.

Running this command (xbbg 7.5 on Python 3.7):

df = blp.bdtick('ESZ1 Index',datetime(2021,8,17),session='day',types=['BID','ASK'],ref='CME')
print(df.head())

And this is the output:

                          ESZ1 Index
                                 typ    value volume exch cond
2021-08-17 08:01:00-04:00        BID  4443.50      1    M  NaN
2021-08-17 08:01:00-04:00        BID  4443.75      1    M  NaN
2021-08-17 08:01:00-04:00        BID  4443.50      1    M  NaN
2021-08-17 08:01:00-04:00        BID  4443.75      1    M  NaN
2021-08-17 08:01:00-04:00        BID  4443.50      1    M  NaN

My exch.yml has:

CME:
  tz: America/New_York
  allday: [1800, 1700]
  day: [800, 1700]

So why does the tick data start a minute late?

[alpha-xone]

Should be a Bloomberg issue - there is no manipulation on time stamps.

[DS-London]

Compare this code then (update to xbbg 0.7.6). Two methods: one uses the session='day' parameter, the second uses an explicit time_range:

df1 = blp.bdtick('ESZ1 Index',datetime(2021,8,17),session='day',types=['BID','ASK'],ref='CME')
print(df1)

df2 = blp.bdtick('ESZ1 Index',datetime(2021,8,17),types=['BID','ASK'],time_range=('08:00','17:00'),ref='CME')
print(df2)

Output:

                          ESZ1 Index                          
                                 typ    value volume exch cond
2021-08-17 08:01:00-04:00        BID  4443.50      1    M  NaN
2021-08-17 08:01:00-04:00        BID  4443.75      1    M  NaN
2021-08-17 08:01:00-04:00        BID  4443.50      1    M  NaN
2021-08-17 08:01:00-04:00        BID  4443.75      1    M  NaN
2021-08-17 08:01:00-04:00        BID  4443.50      1    M  NaN
...                              ...      ...    ...  ...  ...
2021-08-17 16:59:59-04:00        ASK  4428.00      3    M  NaN
2021-08-17 16:59:59-04:00        ASK  4428.00      2    M  NaN
2021-08-17 17:00:00-04:00        BID  4415.00      4    M  NaN
2021-08-17 17:00:00-04:00        ASK  4430.00      3    M  NaN
2021-08-17 17:00:00-04:00        ASK  4436.00      2    M  NaN

[327366 rows x 5 columns]
                          ESZ1 Index                          
                                 typ    value volume exch cond
2021-08-17 08:00:00-04:00        BID  4443.75      1    M  NaN
2021-08-17 08:00:00-04:00        ASK  4445.00      2    M  NaN
2021-08-17 08:00:00-04:00        ASK  4445.00      1    M  NaN
2021-08-17 08:00:00-04:00        ASK  4445.00      2    M  NaN
2021-08-17 08:00:00-04:00        ASK  4445.00      1    M  NaN
...                              ...      ...    ...  ...  ...
2021-08-17 16:59:59-04:00        ASK  4428.00      3    M  NaN
2021-08-17 16:59:59-04:00        ASK  4428.00      2    M  NaN
2021-08-17 17:00:00-04:00        BID  4415.00      4    M  NaN
2021-08-17 17:00:00-04:00        ASK  4430.00      3    M  NaN
2021-08-17 17:00:00-04:00        ASK  4436.00      2    M  NaN

[327481 rows x 5 columns]

If I explicitly give I let xbbg determine the time range, it starts a minute after 08:00. If I explicitly define the range, it starts as I expect. So it does not seem to be Bloomberg which is adding the minute. Both calls end at the same time.

EDIT: I dug a little into the dbtick code, and this is the underlying call that defines the UTC datetimes that are sent to Bloomberg.

from xbbg import blp
from datetime import datetime
from xbbg.core import process

time_rng = process.time_range(datetime(2021,8,17), ticker='ESZ1 Index', session='day', ref='CME')
print(time_rng)

Gives the output: Session(start_time='2021-08-17T12:01:00', end_time='2021-08-17T21:00:00')

So you can see that an extra minute has crept in somewhere, and that is before any communication with Bloomberg.

[DS-London]

Having looked in the xbbg code, from intervals.py:

    def market_normal(self, session, after_open, before_close) -> Session:
        """
        Time intervals between market

        Args:
            session: [allday, day, am, pm, night]
            after_open: mins after open
            before_close: mins before close

        Returns:
            Session of start_time and end_time
        """
        logger = logs.get_logger(self.market_normal)

        if session not in self.exch: return SessNA
        ss = self.exch[session]

        s_time = shift_time(ss[0], int(after_open) + 1)
        e_time = shift_time(ss[-1], -int(before_close))

        request_cross = pd.Timestamp(s_time) >= pd.Timestamp(e_time)
        session_cross = pd.Timestamp(ss[0]) >= pd.Timestamp(ss[1])
        if request_cross and (not session_cross):
            logger.warning(f'end time {e_time} is earlier than {s_time} ...')
            return SessNA

        return Session(s_time, e_time)

Highlight this line: s_time = shift_time(ss[0], int(after_open) + 1)

For some reason, a minute is added to the start time. Any idea of the reasoning behind this?

[alpha-xone]

hmm.. was for my personal preference. makes more sense not adding 1 minute to this.

[alpha-xone]

you can remove this and I will merge it into main branch

[Psyfun233]

Hi, fyi the behavior still persists.

[tbacher]

Hi @alpha-xone , as pointed out by @Psyfun233 the issue is still there. I don't think it is an issue becuase one can just set the config argument. But for that you would have to set session=session instead of session='allday' when you determine time_rng in blp.py. Not sure why you determine time_rng if you already have ss_rng.

` ss_rng = process.time_range(dt=dt, ticker=ticker, session=session, tz=ex_info.tz, **kwargs) data_file = storage.bar_file(ticker=ticker, dt=dt, typ=typ) if files.exists(data_file) and kwargs.get('cache', True) and (not kwargs.get('reload', False)): res = ( pd.read_parquet(data_file) .pipe(pipeline.add_ticker, ticker=ticker) .loc[ss_rng[0]:ss_rng[1]] ) if not res.empty: logger.debug(f'Loading Bloomberg intraday data from: {data_file}') return res

if not process.check_current(dt=dt, logger=logger, **kwargs): return pd.DataFrame()

cur_dt = pd.Timestamp(dt).strftime('%Y-%m-%d')
q_tckr = ticker
if ex_info.get('is_fut', False):
    is_sprd = ex_info.get('has_sprd', False) and (len(ticker[:-1]) != ex_info['tickers'][0])
    if not is_sprd:
        q_tckr = fut_ticker(gen_ticker=ticker, dt=dt, freq=ex_info['freq'])
        if q_tckr == '':
            logger.error(f'cannot find futures ticker for {ticker} ...')
            return pd.DataFrame()

info_log = f'{q_tckr} / {cur_dt} / {typ}'
trial_kw = dict(ticker=ticker, dt=dt, typ=typ, func='bdib')
num_trials = trials.num_trials(**trial_kw)
if num_trials >= 2:
    if kwargs.get('batch', False): return pd.DataFrame()
    logger.info(f'{num_trials} trials with no data {info_log}')
    return pd.DataFrame()

while conn.bbg_session(**kwargs).tryNextEvent(): pass
**time_rng = process.time_range(dt=dt, ticker=ticker, session='allday', **kwargs)**
request = process.create_request(
    service='//blp/refdata',
    request='IntradayBarRequest',
    settings=[
        ('security', ticker),
        ('eventType', typ),
        ('interval', kwargs.get('interval', 1)),
        ('startDateTime', time_rng[0]),
        ('endDateTime', time_rng[1]),
    ],
    **kwargs,
)

`

Thanks for writing the package in the first place.