Running multiple behaviors

[ccharp]

Quick question about architectural vision: how are you planning on supporting multiple behaviors at once? For instance, running the reporter and server simultaneously. Will app.py just loop over each behavior that is configured and call run asynchronously? Or will we need to run multiple instances of the app, one behavior for each (won't work with docker + sqlite)? Or something else I missed? I've locally I've just taken the app.py modification approach; happy to share what I've got, just wanted to know the direction you're going to take

[jrbartola]

The idea is that all behaviors (currently) are mutually exclusive. Backtesting with the UI at the moment has no integration with the default behavior, or reporter behavior, and vice versa. The UI is only responsible for testing strategies against historical data and does not interface with the database in any way. In the future we may use worker threads or some other concurrent utility to implement simultaneous backtesting and live trading.

@shads2

[ccharp]

I see. Thanks for the quick response. How will historical data be aggregated for the purpose of backtesting (and maybe some ml algo)? This is particularly important for candle data, which exchanges do not keep as long as we'd like. We had originally thought to implement a Behavior for collecting data, but that necessitates simultaneous Behaviours unless the db is shared between instances of Crypto-Signal. Alternatively, exchange/analyzer.get_historical_data() could persist that stuff as well.

[ghost]

So right now the intent is that the different behaviours do integrate on the database even if that is architecturally a little evil. This interaction can already be seen: https://github.com/AbenezerMamo/crypto-signal/blob/master/docker-compose.reporter.yml

Realistically though solving this problem in a more reasonable way involves doing something a little more architecturally involved which is outside of our immediate scope. Our current scope being make the core functionality work and make it stable.

[keeler]

@shads2 Can you elaborate a bit on what you mean by solving it in a more reasonable way? It seems reasonable (or at least not completely insane) to me to have behaviors simultaneously interacting with a DB. For example, a behavior to write candles as @ccharp mentioned and a behavior to read the candles and do something with them (say predictions using ML).

[ghost]

There is an uncomfortable mixing of concerns by integrating it all on a single DB that would make it quite an ugly duckling in the long term, particularly given the diverse amount of things it is trying to do. For now I don't think it is an unreasonable thing, I just worry about the future mess :)

Something that ingests OHLCV data into the database for long term storage is something I did want to do at some point, however due to the rate at which you are able to access the data from the API of most exchanges it's hard to do anything more granular than 30min sticks at best.

[ccharp]

Yeah, I suppose that makes sense for the present state of things.

Regarding the point about sticks and exchange rate limiting, what about using a 3rd party data aggregator like cryptocompare (https://www.cryptocompare.com/api/#introduction)? It looks like they're a bit more liberal with they're throttling and granularity, granted it'd probably limit this project to the exchanges they support instead of ccxt's. The main issue we've encountered, though, is most exchanges simply do not keep stick data around very longs. CryptoCompare could at least be used for filling historical data for backtestings.

[ghost]

I think regardless of if we do it, or they do it, the API rate limits still apply... those are set by the exchanges not ccxt. The data is also more likely to be accurate if we pull it directly from the exchanges themselves than if we use an intermediary.

[ccharp]

That's true. Thanks for the info