Allow historical data importation

[straumat]

Is your feature request related to a problem? Please describe. Some strategies require historical data to be able to start taking positions. For example, some strategies requires to analyse 21 days of historical data... So, if your bot starts now, nothing will happened for 21 days which is quite a problem...

[straumat]

As I do in the spring boot starter test, I can easily load existing tickers from files. The problem is that I don't know what to do with those tickers:

• I cannot send them to the strategy throw updates as it would certainly trigger strategies on old data.
• I could distrctly import them in ta4j based strategy and not call enter() or exit() but it doesn't solve the problem of users not using my util ta4J class.
• I could make them available to the user and let him integrate those data as he wants?

[010011x]

I could distrctly import them in ta4j based strategy and not call enter() or exit() but it doesn't solve the problem of users not using my util ta4J class.

If strategy is using ta4j's UnstableIndicator, it won't trigger. So I don't see this as a big problem, if it's properly documented. But duration of the unstable period should be equal to the length of the historical pre-load.

The other option - adding additional flag to the strategy, if possible. for example startupPeriod = true | false.

I could make them available to the user and let him integrate those data as he wants?

This is viable option, at least for me, too. We can have example of BasicTa4jCassandreStrategy that is taking advantage of this feature. This can be a good start. For example BasicTa4jCassandreStrategyUnstable.

[straumat]

Ok. So what do you think if I implement this:

• You can put tickers to import in a CSV file.
• A method is added to the strategy class that allows you to get Historical data from the CSV, something like List<TickerDTO> getHistoricalData(CurrencyPair, startDate, endDate)
• You are in charge of retrieving the ticker and update your strategy, your ta4j or your bars or anything ?

[010011x]

Ok. So what do you think if I implement this:

• You can put tickers to import in a CSV file.
• A method is added to the strategy class that allows you to get Historical data from the CSV, something like List<TickerDTO> getHistoricalData(CurrencyPair, startDate, endDate)
• You are in charge of retrieving the ticker and update your strategy, your ta4j or your bars or anything ?

So for backtesting, I can read this data from CSV, right ? But for live testing, it will use xchange to pull this data, right ?

UPDATE: I'm assuming xchange can pull historical data.

[straumat]

@Potato334343 XChange can't do that. What i'm doing for my test is that I add the historical data with curl in the continuous integration.

[010011x]

@Potato334343 XChange can't do that. What i'm doing for my test is that I add the historical data with curl in the continuous integration.

In that case this sounds good.

But I've seen some xchange methods that can do that https://github.com/knowm/XChange/blob/develop/xchange-binance/src/main/java/org/knowm/xchange/binance/service/BinanceMarketDataServiceRaw.java#L64

But, as I've said, this should be enough. Maybe in the future this can be an option.

[straumat]

@Potato334343 yes, i know but what bothers me is that it's Binance specific objects like : BinanceTicker24h. I will dig a bit more into XChange to see if they have something we could use

[straumat]

Seems MarketDataService doesn't support getting historical data :(

[straumat]

Ok, here is what seems the "best" solution...

• Creates a table to store imported tickers (IMPORTED_TICKERS table).
• IMPORTED_TICKERS has one primary key on two fields: CURRENCY_PAIR & TIMESTAMP
• When starting, Cassandre will search all files starting with "tickers" and will import all data in IMPORTED_TICKERS.
• In user strategy, a new method will be available: Set<CurrencyPairDTO> getImportedTickers(CurrencyPairDTO currencyPair); to retrieve the tickers imported in IMPORTED_TICKERS.
• In user Strategy, users will be able to override an init() method. This method will be called before any data is pushed to the strategy.

With this, a user can, in the init() method, use imported tickers to initiate its strategy data.

@Potato334343 what do you think ?

[010011x]

@straumat

sounds great. importing will be done via getHistoricalData, which is called inside init, right ?

[straumat]

@Potato334343 importing will be done automatically by Cassandre; no need to call it in init

[010011x]

cool, now it all makes sense. sounds great. thank you!

[straumat]

Ok! i will implement it quickly!

[straumat]

@Potato334343 where do you take your data from ? I was thinking about the import format. I was thinking of CSV with header like : CURRENCY_PAIR, OPEN, LAST... but can you easily generate something like that?

[010011x]

@Potato334343 where do you take your data from ? I was thinking about the import format. I was thinking of CSV with header like : CURRENCY_PAIR, OPEN, LAST... but can you easily generate something like that?

I don't have strong preference for the format, should be easy to implement regardless of the format. You can choose whatever format to be consistent with the cassandre ecosystem.

[straumat]

@Potato334343 i use something like this:

curl -s "https://api.kucoin.com/api/v1/market/candles?type=1day&symbol=BTC-USDT&startAt=${startDate}&endAt=${endDate}" \
| jq -r -c ".data[] | @tsv" \
| tac $1

It works quite well but with CSV and column names, the generation will be more complicated on command line. I'm interested to know how you do that? Thx

[huahouye]

@straumat I am not sure if I should ask my question here or just open a new issue. I meet a problem that is similar to preparing for data before a strategy can be started (take a long time). Is there any possible way to make CassandreStrategyInterface to provide a method for asking me wether those work or data has been done or not before the specific strategy start? I currently implement BeanPostProcessor but it takes a long time when project startup. Thanks in advance.

[straumat]

@huahouye I added a method void initialize(); that you can override in your strategy. This method is called by Cassandre before any flux is started on your strategy. I think you can implement your data preparation there. https://github.com/cassandre-tech/cassandre-trading-bot/commit/80fa749389329b50ed26db1b223247903f50a41f

[huahouye]

@straumat Thanks, I will give it a try.

[010011x]

@Potato334343 i use something like this:

curl -s "https://api.kucoin.com/api/v1/market/candles?type=1day&symbol=BTC-USDT&startAt=${startDate}&endAt=${endDate}" \
| jq -r -c ".data[] | @tsv" \
| tac $1

It works quite well but with CSV and column names, the generation will be more complicated on command line. I'm interested to know how you do that? Thx

I'm planning to use this data downloader https://www.freqtrade.io/en/latest/data-download/ It's exchange-independent (it uses ccxt inside) and output's data in universal format. It has json, csv and hdf5 data formats. It's very easy to keep it up to date with new candles. I can give you example of output format a bit later.

[010011x]

@straumat

This is example of data format. freqtrade doesn't support csv.

[
   [
      1632009600000,
      2934.92,
      2939.72,
      2907.61,
      2908.9,
      977.6293
   ],
   [
      1632013200000,
      2908.99,
      2923.17,
      2884.0,
      2919.58,
      1051.1069
   ],
   [
      1632016800000,
      2918.5,
      2937.0,
      2910.27,
      2923.23,
      600.3657
   ],
   [
      1632020400000,
      2922.2,
      2927.24,
      2908.51,
      2917.76,
      534.3656
   ],
   [
      1632024000000,
      2917.99,
      2935.07,
      2904.34,
      2916.72,
      826.1541
   ],
   [
      1632027600000,
      2916.88,
      2931.42,
      2910.09,
      2927.72,
      520.9959
   ],
   [
      1632031200000,
      2927.49,
      2936.33,
      2919.65,
      2936.33,
      892.4469
   ],
   [
      1632034800000,
      2936.74,
      2955.37,
      2929.21,
      2940.99,
      1138.1236
   ],
   [
      1632038400000,
      2941.74,
      2951.97,
      2938.53,
      2943.21,
      625.7372
   ],
   [
      1632042000000,
      2943.01,
      2949.02,
      2922.77,
      2927.32,
      757.4832
   ],
   [
      1632045600000,
      2927.79,
      2932.49,
      2913.7,
      2914.83,
      609.6637
   ],
   [
      1632049200000,
      2914.43,
      2914.68,
      2826.33,
      2844.33,
      2967.5013
   ],
   [
      1632052800000,
      2844.0,
      2860.95,
      2831.0,
      2856.31,
      1726.5687
   ],
   [
      1632056400000,
      2855.01,
      2871.0,
      2852.01,
      2867.82,
      795.6449
   ],
   [
      1632060000000,
      2867.02,
      2880.58,
      2859.05,
      2860.43,
      1161.2575
   ],
   [
      1632063600000,
      2859.8,
      2891.95,
      2857.94,
      2878.64,
      863.1984
   ],
   [
      1632067200000,
      2878.2,
      2889.0,
      2868.8,
      2872.59,
      782.2001
   ],
   [
      1632070800000,
      2872.17,
      2882.82,
      2846.97,
      2861.57,
      1060.4983
   ],
   [
      1632074400000,
      2861.68,
      2890.89,
      2853.66,
      2877.77,
      1008.9791
   ],
   [
      1632078000000,
      2878.41,
      2882.58,
      2853.03,
      2856.91,
      935.0413
   ],
   [
      1632081600000,
      2857.03,
      2879.94,
      2850.3,
      2875.36,
      691.8665
   ],
   [
      1632085200000,
      2874.63,
      2879.64,
      2854.62,
      2861.63,
      559.1446
   ],
   [
      1632088800000,
      2860.9,
      2874.0,
      2808.32,
      2822.12,
      2090.6085
   ],
   [
      1632092400000,
      2821.57,
      2852.94,
      2813.39,
      2848.03,
      1062.708
   ],
   [
      1632096000000,
      2848.48,
      2863.44,
      2797.41,
      2813.01,
      2179.4494
   ],
   [
      1632099600000,
      2813.35,
      2841.66,
      2805.75,
      2837.31,
      939.8859
   ],
   [
      1632103200000,
      2836.74,
      2839.99,
      2748.67,
      2749.79,
      2919.7223
   ],
   [
      1632106800000,
      2749.07,
      2759.55,
      2707.44,
      2736.15,
      2896.308
   ],
   [
      1632110400000,
      2735.45,
      2760.54,
      2713.5,
      2747.01,
      1314.7225
   ],
   [
      1632114000000,
      2747.0,
      2762.48,
      2743.22,
      2752.96,
      1124.8236
   ],
   [
      1632117600000,
      2752.97,
      2760.49,
      2721.54,
      2735.69,
      1709.9683
   ],
   [
      1632121200000,
      2735.18,
      2751.0,
      2665.63,
      2687.3,
      4244.8984
   ],
   [
      1632124800000,
      2686.62,
      2725.51,
      2674.5,
      2716.09,
      2665.9985
   ],
   [
      1632128400000,
      2716.21,
      2728.38,
      2692.91,
      2695.5,
      1545.0449
   ],
   [
      1632132000000,
      2695.5,
      2732.72,
      2673.24,
      2684.87,
      4461.3839
   ],
   [
      1632135600000,
      2685.44,
      2692.47,
      2591.06,
      2614.54,
      6603.10424
   ],
   [
      1632139200000,
      2614.41,
      2639.03,
      2502.07,
      2595.1,
      7701.07905
   ]
]

Obviously, I'm planning to adapt this data to whatever format you are going to have in cassandre.

[straumat]

Thx. From what i see, there is no column description...

[010011x]

Thx. From what i see, there is no column description...

exactly! But I think it's

date, open, high, low, close, volume in the example above, based on this https://github.com/freqtrade/freqtrade/blob/develop/freqtrade/data/history/jsondatahandler.py

[straumat]

@Potato334343 Hi! I created a branch to make the development. Can you take a look ? Documentation is here: https://github.com/cassandre-tech/cassandre-trading-bot/blob/feature/imported_tickers/docs/src/learn/import-historical-data.md

[straumat]

@Potato334343 I decided to merge into development! Documentation here: https://trading-bot.cassandre.tech/learn/graphql-api.html. Tell me if i works for you. If so, you can close the issue.

[straumat]

Re open in case of problem :)