Support `handle_revised_bars` for Internal Aggregator

[rsmb7z]

Feature Request

Raising in relation to latest conversation going on in #811, it would actually be great if we can support handle_revised_bars action in Internal Aggregator, which will allow for both backtesting/live scenarios. I see some demand for this, someone asked similar in Discord couple of weeks ago as well, I couldn't locate it now. Following are the few use-cases:

1. User trading on one timeframe for example M5 timeframe and assume its mid-day or EOD and want to check the Daily Bars stats (without loosing last Bar even if it is incomplete which is better than a being a day behind), price is not the problem here. So in many trading scenarios one may need this approach.
2. In the GUI feed as shown below - not possible without receiving revisions feed. This even on Binance website when you see the chart, you will see Bar is being updated continously.

Let me know if you need further information or clarification and I would be happy to respond.

[cjdsellers]

@rsmb7z I just need a little more information and context to help here.

How are you currently handling revised bars in the strategy, to prevent them updating the indicators?

Would it help to add some logic for revised bars here?

    cpdef void handle_bar(self, Bar bar) except *:
        """
        Handle the given bar data.

        If state is ``RUNNING`` then passes to `on_bar`.

        Parameters
        ----------
        bar : Bar
            The bar received.

        Warnings
        --------
        System method (not intended to be called by user code).

        """
        Condition.not_none(bar, "bar")

        # Update indicators
        cdef list indicators = self._indicators_for_bars.get(bar.bar_type)
        if indicators:
            if bar.is_revision:  # <-- something here?
                return
            self._handle_indicators_for_bar(indicators, bar)

How are you storing intermediate values for indicators? Most of the indicators just store the last value, not a list of values (previous state), as this is potentially implementation specific for a startegy/actor.

Are you using lower time frame bars to update higher time frame bars? Are you subscribing to ticks at all for your strategies, or they're purely fed with closed and partial bars?

The example you give is indeed more applicable to a GUI, and is what the BarBuilder might look like inside the aggregator.

cdef class BarBuilder:
    cdef BarType _bar_type

    cdef readonly uint8_t price_precision
    """The price precision for the builders instrument.\n\n:returns: `uint8`"""
    cdef readonly uint8_t size_precision
    """The size precision for the builders instrument.\n\n:returns: `uint8`"""
    cdef readonly bint initialized
    """If the builder is initialized.\n\n:returns: `bool`"""
    cdef readonly uint64_t ts_last
    """The UNIX timestamp (nanoseconds) when the builder last updated.\n\n:returns: `uint64_t`"""
    cdef readonly int count
    """The builders current update count.\n\n:returns: `int`"""

    cdef bint _partial_set
    cdef Price _last_close
    cdef Price _open
    cdef Price _high
    cdef Price _low
    cdef Price _close
    cdef Quantity volume

    cpdef void set_partial(self, Bar partial_bar) except *
    cpdef void update(self, Price price, Quantity size, uint64_t ts_event) except *
    cpdef void reset(self) except *
    cpdef Bar build_now(self)
    cpdef Bar build(self, uint64_t ts_event, uint64_t ts_init)

I feel like for internally aggregated bars, you'd like some option to publish partial bars as revisions (maybe partial is better naming than revision)?

Previously in another thread, you mentioned that a strategy has no access to the "currently being built" bar?

[rsmb7z]

How are you currently handling revised bars in the strategy, to prevent them updating the indicators?

Within Strategy I am handling on_bar, however for Indicators I am using TA-Lib within Indicator class which makes is_revision check at handle_bar.

How are you storing intermediate values for indicators? Most of the indicators just store the last value, not a list of values (previous state), as this is potentially implementation specific for a startegy/actor.

We don't need to store the previous values in this case as well, if one required can store in strategy/actor as current case. It should just use self._inputs.append(value) vz self._inputs[-1]=value similar handling of Cache in DataEngine. Example case:

1. AVERAGE[10, 15, 20, 25, 30] = 20.0 (5 Period Average)
2. is_revision=True: AVERAGE[10, 15, 20, 25, 35] = 21.0 (5 Period Average - Bar value was just replaced instead of append then continue to calculate the Indicator value with no difference)
3. is_revision=False: AVERAGE[15, 20, 25, 35, 40] = 27.0

Are you using lower time frame bars to update higher time frame bars?

This is not related scenario of handle_revised_bars and not being used.

Are you subscribing to ticks at all for your strategies, or they're purely fed with closed and partial bars?

Strategy is subscribing to Ticks as well for price updates. Now this question includes bit of answer why is_revision is required for Indicators. A Trader do enter or exit of trade based on Price and/or Indicator values. Let's say we are within a trade and Exit is planned based on two scenarios:

1. Stop Loss Price based: We don't have any problem with this scenario because we always have latest price using Ticks. Imagine if we don't have the Ticks (or latest price) then Stop Loss will trigger when the Bar is closed. However by this time, depending on Bar Size, the market could move away far and Stop Loss will not be that effective as it is in normal cases.
2. Stop Loss Indicator value based: We don't have the Ticks here. However with bar_revision handling we always have upto date value of Indicator without actually waiting for the Bar to close

Just as note, if the trading strategy is 1H then any indicators values of lower timeframes cannot help in this case.

I feel like for internally aggregated bars, you'd like some option to publish partial bars as revisions (maybe partial is better naming than revision)?

Yes correct, internally aggregation will allow the support for Backtesting as well as if any Adapter doesn't support updates. Which otherwise is not available using external data. The reason for choosing name revision is because for external updates we know when the Bar is new_bar based on it's timestamp (when receiving from DataProvider) however if any Bar is final_bar, we do not know this until new Bar arrives with new timestamp. So to set is_partial=False we need to know that the Bar is final Bar, and holding the Bar from publishing for by waiting for next Bar wouldn't be good idea because of lagging then. Although this will not be the case for Internal aggregated Bars however then we will have to introduce another flag for Bar.

Previously in another thread, you mentioned that a strategy has no access to the "currently being built" bar?

I think you are referring to this thread https://github.com/nautechsystems/nautilus_trader/discussions/811, raised when I was new to system. So basically what I meant is Bar is not published by BarBuilder unless Bar is complete - so in that sense strategy have no information about "currently being built" bar.

Current Flow of Bar and how is_revision being handled

Potentially more appropriate scneario

[cjdsellers]

With the flow charts you provided, if you wanted to take different actions in the strategy depending on whether the received bar is a revision, could you not also do something like this?

def on_bar(self, bar: Bar) -> None:
    if bar.is_revision:
        # Do something different

Likewise for the indicator. I'm just trying to think of some solutions which doesn't involve extending the API by adding two additional methods.

How are you working around this currently?

[rsmb7z]

How are you working around this currently?

Currently I am doing it as in Current Flow of Bar (red boxes). This is okay, I was thinking in terms of performance, we will be doing same test 1+n times (where n is the number of indicators). If there will not be significant improvement in performance then we can leave it like that.

Though more important than this is the support for publishing of intermitant Bars by BarBuilder.

[OnlyC]

I suggest a new method: on_bar_update(self, bar: Bar) which is the current Bar forming. You can use this method in needed indicator to re-calculate value.

This method can be triggered by internal Bar aggregation (triggered on each tick - should be limit by time interval or ticks interval for better performance) or by External Data engine (triggered on each websocket Bar update endpoint - if they provide)

[cjdsellers]

Though more important than this is the support for publishing of intermitant Bars by BarBuilder

Isn't this just the same as creating an internal bar type for whatever interval or aggregation rule you need? If currently you have 1-HOUR bars and desire the intermediate minutes, then also subscribe to 1-MINUTE?

Just trying to figure out the part I'm missing here?

[rsmb7z]

Isn't this just the same as creating an internal bar type for whatever interval or aggregation rule you need? If currently you have 1-HOUR bars and desire the intermediate minutes, then also subscribe to 1-MINUTE?

Hmm, hence the confusion. No it is not same thing. Indicators produce different results at each aggregation level otherwise there would have no point of having or using different aggregation levels. For example 20-periods for 1-HOUR will be 20-hours whereas 20-periods for 1-MINUTE will be 20-minutes and both cases having different OHLCV values and different set of data. So if you have ATR for example, in both cases will produce different result and same applies to other indicators. So one wanting to trade for very short period could be using 1-MINUTE and for medium period could be using 1-HOUR and they cannot replace each other - just like you see complete different picture on graph for different time-frames, so does the indicators and machine.

[rsmb7z]

Here there are 3 different timeframes in chart and some indicators. As you see there is completely different perspective and indicator values.

[OnlyC]

@rsmb7z You might get that wrong, you use the 1 minute on_bar to REPLACE the last bar.close price in your indicators, which is holding a moving window of 20 value of high timeframe bars. So indicator still alway have 20 values (20 hours), but you replacing the last one every minute.

You might have to modify some indicators to be able to do that.

[rsmb7z]

@OnlyC Thanks for the idea. Yes you are right it can be done like that and we event don't need 1-MINUTE bar in this case and simply Tick prices can do this. I believe this will be more of level work-around though. Yes I am already using TA-Lib temporarily to support partial Bars as I don't want to keep the built-in indicator modications as workaround.

@cjdsellers So I can say to opt out of this request, if having a simple flag and publishing by BarBuilder is not worth or over complicates the system.

I suggest a new method: on_bar_update(self, bar: Bar) which is the current Bar forming. You can use this method in needed indicator to re-calculate value.

However this I will leave open as in any case we need this efficient handling.

[cjdsellers]

I suggest a new method: on_bar_update(self, bar: Bar) which is the current Bar forming. You can use this method in needed indicator to re-calculate value.

For this proposed on_bar_update handler, how would we specify when the partial bar should be published?

[rsmb7z]

For this proposed on_bar_update handler, how would we specify when the partial bar should be published?

• For external Bars, when the Bar has is_revision=True it shall go to on_bar_update
• For internal Bars, if support added, when the Bar has is_revision=True it shall go to on_bar_update. About when to publish the partial bar, ideally when Bar is updated within BarBuilder or can be user configured interval as @OnlyC suggested.
• In case support not added in BarBuilder, user will be calling on_bar_update directly (by making last_bar of that timeframe as HLC will keep changing) to update indicators.

[cjdsellers]

I think I understand the requirements now, the OHLCV values of partial bars would not be equal to all of the lower timeframe bars in my one minute bar example.

I need to think about it a little more, but we can probably add another parameter to subscribe_bars which is some sort of intermediate publish interval.

This would end up as a setting for the BarBuilder which would just run two timers, one for the partial bar publishing (which doesn't reset the bar after building), and one for the bar close time.

[rterbush]

FWIW, the behavior described here is what I am familiar with on other platforms, referred to as "Intra-Bar Order Generation".