search

mdeverdelhan / ta4j-origins

A Java library for technical analysis ***Not maintained anymore, kept for archival purposes, see #192***
https://github.com/mdeverdelhan/ta4j/issues/192
MIT License
375 stars 225 forks source link

Question on calculating indicators on incoming real-time data #72

Closed rterp closed 6 years ago

rterp commented 8 years ago

Is it possible to calculate indicators in real-time, ie, as new ticks arrive, calculating a new value for say the RSI indicator?

In the example below I am first reading in my own historical data, and building a time series from Ticks using the data. In the 2nd step, I subscribe to market data, and as new data arrives, I get the most recent tick, and add a trade to it. I was expecting at this point for the rsi value for the last tick to be different than before the new trade was added, but it looks like its not. Is there a way to get the latest value of the RSI given that a new trade has been placed in the most recent tick?

    for( BarData data : historicalData ) {
        LocalDateTime date = data.getDateTime();
        DateTime dateTime = new DateTime(date.getYear(), date.getMonthValue(), date.getDayOfMonth(), date.getHour(), date.getMinute(), date.getSecond());
        Tick tick = new Tick(dateTime, data.getOpen(), data.getHigh(), data.getLow(), data.getClose(), data.getVolume());
        series.addTick(tick);
    }

    ClosePriceIndicator close = new ClosePriceIndicator(series);
    RSIIndicator rsi2 = new RSIIndicator(close, 2);

    client.subscribeLevel1(ticker, (ILevel1Quote q) -> {
        if( q.getType() == QuoteType.LAST ) {
            series.getLastTick().addTrade(100, q.getValue().doubleValue());
            Decimal value = rsi2.getValue(series.getEnd());
            System.out.println("Last Price: " + q.getValue().doubleValue() + " RSI value: " + value);
        }
    });`
gcauchis commented 8 years ago

Yes it's possible. I recommend reading the wiki https://github.com/mdeverdelhan/ta4j/wiki/Live trading.

rterp commented 8 years ago

Yes, I did read this, but as far as I understand, a "Tick" is really an OHLC bar. Let's say I'm tracking 1 hour bars, I load them via historical data, as in my example above and on the wiki. I then subscribe to real-time data from the broker which comes streaming in with multiple quotes per second. All I am getting from this stream is the last trade (price and volume), however I'm not going to get a new OHLC bar from the broker for another 60 minutes. Between now and when the new bar arrives (in 60 minutes). I would like the RSI calculation to reflect whatever the "Last" price was that just came in from the streaming data. I would do this until a new 60 minute bar arrived from the broker at which point I could create a new "Tick" object from the bar received from the broker, and add it to the TimeSeries. I was hoping I could simply add the incoming quote from the real-time data stream as a trade to the most recent Tick, and then recalculate the RSI, but it does not appear to work that way.

mdeverdelhan commented 8 years ago

Thank you @gcauchis ! ;)

@rterp You are doing it wrong. To feed your time series with real-time data you have to call addTrade(...). That's right. But not on the same Tick. If you always update the same last tick it will grow indefinitely and will end being much much (much) bigger than the previous ones. Moreover, since most of the indicators are cached (see the CachedIndicator class), it is normal to get the same result every time.

It's up-to-you to create your last tick when you think the previous is full. Without loop it would looks like:

Tick myNewLastTick = new Tick();
series.addTick(myNewLastTick);
myNewLastTick.addTrade(...);
myNewLastTick.addTrade(...);
myNewLastTick.addTrade(...);
myNewLastTick.addTrade(...);
// Enough: my new timestamp needs a new Tick (for instance: I reached 60 minutes)
myNewLastTick = new Tick();
series.addTick(myNewLastTick);
myNewLastTick.addTrade(...);
myNewLastTick.addTrade(...);
// ...

Then you can call rsi2.getValue(series.getEnd()); always using series.getEnd() since the end index of the series is incremented after every addTick(...).

Maybe we should add an automated Tick creation when adding a trade to the time series.

PS: Don't forget to call the setMaximumTickCount() method if you are doing live trading.

rterp commented 8 years ago

Ah ok. Thanks for the tip. I will give it a try and hopefully it will fixed the issue.

rterp commented 8 years ago

@mdeverdelhan ok, here is my revised code below. I've updated it to create a new Tick with an end time 1 minute after the last tick. I then add the tick to the CloseSeries, and begin adding the incoming data stream to the new Tick as Trades, however I am now getting NaN for the RSI value.

`

for( BarData data : historicalData ) {
        LocalDateTime date = data.getDateTime();
        DateTime dateTime = new DateTime(date.getYear(), date.getMonthValue(), date.getDayOfMonth(), date.getHour(), date.getMinute(), date.getSecond());
        Tick tick = new Tick(dateTime, data.getOpen(), data.getHigh(), data.getLow(), data.getClose(), data.getVolume());
        series.addTick(tick);
    }

    Tick newTick = new Tick(Period.minutes(1), series.getLastTick().getEndTime().plusMinutes(1));
    series.addTick(newTick);
    ClosePriceIndicator close = new ClosePriceIndicator(series);
    RSIIndicator rsi2 = new RSIIndicator(close, 2);

    client.subscribeLevel1(ticker, (ILevel1Quote q) -> {
        if( q.getType() == QuoteType.LAST ) {
            newTick.addTrade(100, q.getValue().doubleValue());
            Decimal value = rsi2.getValue(series.getEnd());
            System.out.println("Last Price: " + q.getValue().doubleValue()  + " Last tick: " + series.getLastTick().getClosePrice() + " RSI value: " + value);
        }
    });`
mdeverdelhan commented 8 years ago

Check the AverageLossIndicator. If it returns Decimal.ZERO it is normal to get Decimal.NaN for your RSI (since it is used as a divisor in the formula, see the code of RSIIndicator).

rterp commented 8 years ago

Ok, that seems to be the issue, plus the fact that the indicator value is cached, as new ticks come in the value returned will remain NaN. Short of creating a brand new RSI object for each incoming tick is there a way to force a cached indicator to calculate a new value based on the the latest value in the time series?

mdeverdelhan commented 8 years ago

No, there isn't. Such a mechanism could be implemented. It would need to update the CachedIndicator class by checking if the Tick has been updated and, if so, by recalculating the value. Even if it seems feasible I don't know if it's relevant. You ask for the RSI value after each trade. Conventionally the calculation of any indicator should be asked only when the tick ends. The process should be: 1) You receive a trade. 2) The received trade is in the time frame of the tick you are currently feeding. 3) You add the received trade to the tick you are currently feeding. 4) You don't execute any calculation. 5) You receive a trade. 6) The received trade is out of the time frame of the tick you are currently feeding. 7) You put the received trade aside and call it "A". 8) You add the tick you are currently feeding to the time series. 9) You do any calculation you want, you execute your strategy, you decide/enter/exit/buy/sell/etc. 10) You create a new instance of Tick: it's your new tick you are currently feeding. 11) You add the trade "A" to the tick you are currently feeding. 12) You repeat the previous steps as needed.

If you work directly on trades (instead of ticks) then:

rterp commented 8 years ago

The requirements for my project are that the indicators need to be calculated in real-time on the current bar. Backtest tools such as MetaTrader and Multicharts will allow you to calculate indicators as the current bar (tick) is updated in real-time, as do the indicators on the charts provided by Interactive Brokers Trader Workstation. ie, you don't need to wait for the completion of a bar in order to find out the value of the indicators, if your system is configured to place a trade before the next bar/tick arrives. If you are working with one hour bars and say a 14 period RSI you are unlikely to get extremes as the price updates on the current tick, but you will see the change in RSI as the latest price changes.

I'm attempting to add indicators to my trading API https://github.com/rterp/SumZeroTrading which will interface with Interactive Brokers in order to automate some trading strategies that I have created, but in order to do this I would need to be able to update the indicators in real-time.

I wouldn't mind contributing to ta4j library to make something like this a feature which could be enabled as an option, but you as the project owner would need to decide if that's something you want included in the library.

thanks, -Rob

mdeverdelhan commented 8 years ago

OK. I understood your use case, so why not.

Feel free to add this to the CachedIndicator class. Possibly you may add an addTrade(...) method to the time series, that would automatically create/add a new Tick if needed.

rterp commented 8 years ago

Cool, I'll probably start working on this later this week.

thanks, -Rob

cyrus13 commented 8 years ago

I came across the ta4j library a few days ago. I want to start using it, but I also require this feature. Do you have an update, Rob? I could give a hand if you want or start implementing it if you haven't done so already.

rterp commented 8 years ago

Hi Cyrus, I did a little playing around with this at the end of the week, trying to find a good way to do it. I was able to get the ClosePriceIndicator to update with the latest real-time price, but was not able to get Indicators based on the ClosePriceIndicator, such as the RSIIndicator to recalculate itself based on the modified ClosePrice. Will be continuing to look at it this week, but if you want to take a look and see if there may be a better way to go about implementing it, I'm open to suggestions.

cyrus13 commented 8 years ago

Thank you Rob. Ok, I can give it a shot. As I said I am new to the library, so let me explain what I have in mind to make sure we are on the same page / I am not doing something awfully wrong.

We need the indicators to update with real time price data. So:

1.

We initialize the TimeSeries with the OHLC of the period we are interested (e.g. based on 1 hour charts). Then as each new real time bid/ask comes it will be added to the TimeSeries using a new addTrade() method. This will initially create a new close price and for each new price that comes this same price will be updated.

2.

We also need some logic in the timeseries that will be able to create a new tick (OHLC) when the new trades (prices from the broker) that come are part of a new period. For example assume you have a TimeSeries with period = 1H. Then you receive only two real-time prices, one at: 09:59:59 and the other one at: 10:00:01. When the price of 10:00:01 arrives we should create a new tick, add it to the timeseries using as close price the price we received at: 09:59:59 and then add the price we received at: 10:00:01 as a new trade.

3.

The cachedIndicator needs to always use as the last close price in the series the latest real-time value. This will require the cachedindicator to know that the timeseries has changed and the position of the change, so that it can recalculate all values from the changed position of the timeseries onwards the next time the getValue() method is called on the cachedindicator.

@rterp is this what you also intended to do? @mdeverdelhan what do you think?

rterp commented 8 years ago

@cyrus13 Yes, this is exactly my use case as well. I'm brand new to this library as well, so haven't had a chance to get too deep into the internals of the CachedIndicator and other indicator classes to figure out the best way to go about this.

What provider are you using for your real-time data, and what library are you using to connect to them?

I'm currently using Interactive Brokers, and my open source framework https://github.com/rterp/SumZeroTrading

mdeverdelhan commented 8 years ago

Would another solution not be possible? Would the job be done if the last result of all the indicators (e.g. close price but also RSI, etc.) was always calculated (never cached)? If so, it would be pretty simple and would avoid weird design. Though, it would also need some performance benchmarks and an explanation about the cache system in the wiki.

rterp commented 8 years ago

Yes, this is exactly what I would be looking for, just the last result to be calculated. If the results of the previous ticks/bars remained cached it would be fine.

cyrus13 commented 8 years ago

@mdeverdelhan You can always avoid caching the last result of the indicators, but this could need to invalidate also the previous from the last result if you end up creating a new tick based on the real-time prices.

I found some time today to work on a solution. Basically I created a new class that is responsible to create new ticks if the real-time prices that come are after the last tick period of the series. It is also responsible for invalidating the relevant cachedindicators cached values.

In this way you cache all values, even the latest, which doesn't affect performance when using the library for backtesting. However, when you use the library to take decisions based on the real-time data, you can do so using this approach.

Sample code:

double[] ticks = {1,2,3};
DateTime[] times = {new DateTime(2016,01,01,6,0,0), new DateTime(2016,01,01,6,15,0), new DateTime(2016,01,01,6,30,0)};
TimeSeries series = new MockTimeSeries( ticks, times);
RealTimeTradeController realTimeTradeController = new RealTimeTradeController(series);
ClosePriceIndicator cpIndicator = new ClosePriceIndicator(series);
SMAIndicator sma = new SMAIndicator(cpIndicator, 3);
realTimeTradeController.addRelevantCachedIndicators(cpIndicator,sma);
assertEquals(2,sma.getValue(2).toDouble(),TestUtil.ESPILON);
realTimeTradeController.addTrade(4, 5, new DateTime(2016,01,01,6,29,0));
assertEquals(2.66666,sma.getValue(2).toDouble(),TestUtil.ESPILON);

I have committed the change to a fork: https://github.com/cyrus13/ta4j. I plan to write some javadoc when I have time. Also even though I wrote unit tests, I haven't had time to actually try it. Plan to do so next week. Will create a pull request if you think the solution is ok.

mdeverdelhan commented 8 years ago

Your solution comes with a controller that is used only because the results are cached. I think that the caching mechanism should be transparent for the user and, if it's possible, I don't want to add concepts (like "relevancy") for cached indicators. Moreover I have in mind to refactor the caching system to let the advanced user choose the underlying cache mechanism (i.e. no cache, basic arraylist cache, DB cache, etc.).

Anyway, why, if you end up creating a new tick, would you need to invalidate the previous one? If you create a new tick N, the tick N-1 is not yet cached, then:

Or maybe I missed something.

cyrus13 commented 8 years ago

I totally agree with you that introducing new concepts like the controller/relatedIndicators brings new complexity that should be avoided if possible. I followed this solution based on the requirements and the fact that I didn't want to change the existing code, but add something on top.

What I had in mind when I wrote that you will need to invalidate N-1 cached result is the following scenario:

Assume you have a timeseries with 5 ticks. You do:

In this scenario you need to invalidate the 5th (N-1) cached value as price1 was added.

From a theoretical perspective since you can addTrade() on a Tick you can always change its min/max/close price. In such cases you would need to have a way to invalidate the respective elements in the cached indicators that use the Ticks.

However, the normal scenario in real time processing is:

so probably not caching the very last value, as you suggested would work for realtime data processing.

Also note that the controller has some logic to create new Ticks if the trade is in the next period compared to the previous Tick. This logic is required for real time price processing. This logic could probably go into an addTradeToLastTick() or something in the timeseries?

mdeverdelhan commented 8 years ago

Even in your first scenario I don't see when you have to invalidate the 5th result.

  1. getValue(5). [not cached because it's the last value, no need for invalidation]
  2. lastTick.addTrade(price1) [no need for invalidation of getValue(5) because it's still not cached]
  3. price2 comes from broker that is in new period. Crete 6th Tick in series and addTrade(price2) on new last tick [no need for invalidation of getValue(5) because it's still not cached]
  4. getValue(6) [not cached because it's the last value, no need for invalidation]

Even if you add two new steps like:

  1. getValue(5) [no need for invalidation because it's not cached till now]
  2. getValue(5) [no need for invalidation because it's not the last result]

That way, the only problem you could get is if you execute the following section:

On the last getValue(5) you will get an unconsistent result. But it's only in the case of adding a trade to an old (i.e. not the last one) tick, and I don't think that taking this case into account be relevant/useful. What do you think?

PS: Of course I retain the logic of the addTrade*() method to the time series. I just think that the period problems should be addressed before (see #80 and #81).

cyrus13 commented 8 years ago

Yes I agree the first scenario you describe is the one that we need for realtime price processing. As I mentioned in my previous comment this would work with your original suggestion (i.e. not caching the last result). I believe currently the last result is cached. For example the following test fails with version 0.8:

 timeSeries = new MockTimeSeries(1,2,3);
closePrice = new ClosePriceIndicator(timeSeries);
assertEquals(3, closePrice.getValue(2).toDouble(),0.001);
timeSeries.getLastTick().addTrade(10, 5);
assertEquals(5, closePrice.getValue(2).toDouble(),0.001);```

Not caching the very last value appears to be an easy fix that would work for real time prices as you also suggested.

Ok, it is understood that you want to address (#80 and #81) before adding the addTrade() to timeseries. In the meantime I would like to use the library for realtime data processing. To do so, I will keep the logic of adding the new tick if it's outside of the period in the forked branch.
berlinguyinca commented 7 years ago

has been there any updates to this?

mdeverdelhan commented 7 years ago

@berlinguyinca No. Not enough time to address this.

nimo23 commented 6 years ago

Maybe a good solution for real-time-indicator could be the following:

Add a new method tick#addSubTick(s).

"addSubTick" extends TimeSeries where we can put a "timeSeries" to each "tick". The "subtick"-timeseries can then be further analyzed with the already given tradingrecord and the analyzers. Maybe the subtick has one special "limitation", it has no open, high, low. Only one "price" (close price).

moodstubos commented 6 years ago

Hi @all

I also need real time indicators. Here I have a simple hack for you that works for me.

In CachedIndicator.java change this:

comment out this line // results.set(results.size()-1, result);

few lines below add this condition before setting the result value to cache

if( index < highestResultIndex  )
   results.set(resultInnerIndex, result);

this prevents the CachedIndicator to cache the very last value. You can simply use tick.addTrade(...) to update the last candle "close" price and all indicators are up to date.

I hope this helps.

mdeverdelhan commented 6 years ago

Please continue this thread on the new repo: https://github.com/ta4j/ta4j