Sharing trading data with multiple bots

[darioneto]

• Operating system: Docker
• Python Version: ___Python 3.11.4
• CCXT version: ____ccxt==4.0.34
• Freqtrade Version: __docker-2023.7.dev-52ec2324

In the context of operating multiple trading bots on different exchanges, what would be the recommended method to communicate with each other about their trades? Specifically, I'd like to implement a feature where if a bot has a trade that is x % in loss/profit, then another bot, when generating a signal in the same direction and for the same trading pair, will only execute the trade if it can validate the loss/profit condition from the first bot. The bots use the same strategy and trade the same pair, but the second bot's decision to trade is influenced by the performance of the first bot's trades. Not sure if this would be easier/recommended to maintain external deployment in the form of database/filesystem/api/shared memory or can be integrated with the producer/consumer concept mentioned here https://www.freqtrade.io/en/stable/producer-consumer/

[xmatthias]

well when running multiple strategies (different exchange counts as different strategy, really - as pairs do not move identically across exchanges - so what works on binance can terribly fail on bybit or viceversa), then the base assumption should be that they are all independent.

if it's really the same strategy, they'll most likely enter the same pair at the same time (in which case, your logic won't work). If they don't (diff. pairlists, ...), then i don't think your above logic makes sense - as a trade you entered 1 hour ago can be down 8% - but that doesn't mean it's no good time to enter - if it'll recover, the other bot will take the full 8% profits, while the first bot get's barely even. If it doesn't recover - work on your strategy, then your signal wasn't good (neither of the 2 that caused the 2 different entries).

what you're asking above will also cause very subtle problems - where you'll no longer can rely on "why was this signal ignored" - nor can you backtest this - as for every trade, you can't know if you've REALLY been in this trade, or if it's been "blocked" by some other logic.

---

producer/consumer will work if you can assign "whatever" you have to the dataframe. It'll however be one-way communication - as i don't think creating a loop (where the 2 bots bounce dataframes from each other) will work (though i never attempted this, either, and don't know a technical reason why it shouldn't work - so it may depend on the implementation in the strategy if the two block or not).

---

The most likely simplest way is to have the bot's use each other's "blacklist" feature through the API - where you block a pair if bot A doesn't want bot B to enter (don't forget to unblock it).

[darioneto]

Thanks for your reply, my strategy focused on a single trading pair and I'm pulling the signals from an external database. This pretty much evens out the playing field between the exchanges to about near-simultaneous entries and exits across the board, with only some negligible rounding errors and exchange quirks popping up. But that's I can totally live with it.

your idea of a "blacklist". Sounds interesting, but here's the concern - I'm planning to roll out 6+ bots and the tangle of relation-conditions could turn into a real spaghetti. Any chance you've got some docs about this approach on the site to read about it?

Then there's this whole producer/consumer angle I've been meaning to look into at some point. The more I think about it, the more I see that the database might throw a wrench in the scalability works if I start piling on conditions.

[xmatthias]

This pretty much evens out the playing field between the exchanges to about near-simultaneous entries and exits across the board

Then see above - you'll have all bots in the same position - so if you assume they all open at 12:00 exactly, they ALL will open - or one random one will not as it's slightly later - turning this into a complicated pointless endeavor.

Any chance you've got some docs about this approach on the site to read about it?

No - and we'll also don't intend to include such (or provide such) information. From a concept perspective, all bots should be treated individually.

We have some basic API docs and the rest client script - assumption if you intend to use the API is that you'll also be able to look at the code.

[darioneto]

Ok thanks for the insights, I will explore my options for API and the consumer/producer then.

[xmatthias]

Please rethink your overall aproach before investing time (and then losing it) :)

your statements were

• i have one central place generating signals for all bots (this should be the producer, unless all bots get the same source without resource problem - which you shouldn't have with 6 bots)
• all bots use the same pair and strategy - (so they are expected to all have the same trades at the same time)

Combining this with your question, your question does no longer make sense - as

• if one bot is down, all bots are down (remember, same strategy, same pair)

therefore all logic preventing an entry when "another bot is down" is unnecessary (you'll find it to be quite some effort to make this work reliably, and operationally, VERY prone to breaking) - unless your assumptions (the way i've listed them above) are wrong - in which case your whole "system" seems to not live up to your expectations.

[darioneto]

ok so to follow up on the consumer/producer how reliable could we expect this setup to be in a real-world scenario? Due to time constraints, I wasn't able to test it out yet. Yes, all bots currently utilize a common data source without encountering resource issues. However, there's a concern about scalability moving forward though. The bots operate on a K8s platform, with layers of physical and logical separation, with fast recovery capabilities due to monitors etc. So far, this setup has been robust with minimal issues. The main aim is to optimize trade entries while minimizing risk. Regardless of how good is the strategy, none of them will pick up perfect tops or bottoms. The concept I described would help with optimizing for those conditions

[xmatthias]

producer/consumer is as stable as your network (and the producer) gets.

obviously if the producer "leaves" - then the others won't get new information until they're able to reconnect.

regarding scalability would concern me more that you're running

• the same strategy
• on one pair

At one point (not far away if you're on 6 exchanges already), you'll run out of reliable exchanges - so the real scalability will end there.

[darioneto]

Do you know if HA mechanism can be implemented with the producer-consumer concept? For example, if the primary producer leaves then standby can take over. Since this is the same strategy it would be limited by the availability of the producer.

The "one strategy" has a very complex chain of conditions which is not exactly a flat single strategy (this is where the concern about the scalability comes from, each of the conditions is putting the burden on DBs and constant recalculation of the conditions taking a significant hit ). "One pair" is plenty to work with :-)

[xmatthias]

You can't do HA - but you can connect to 2 upstreams. If that'll be additional, or "either/or" or "fallback" will then solely depend on your implementation in the strategy.

Please refer to the documentation for supported configs for producer/consumer mode.

[darioneto]

ok thanks for the insights, will check it out to see if I can make it work

[darioneto]

If I opt for a low-hanging fruit via the API would this logic make sense in its overall integration with freqtade?

To summarise this operation "just high level", Bot2 API calls Bot 1 to fetch its trade status. Then it looks for trades with a tag that contains "-XXX-" and an unrealized profit percent less than or equal to -X%. If it finds such a trade, it sets enter_long_condition to True. This condition is then included in the conditions for setting the 'buy' column in Bot 2's dataframe to 1.

def populate_buy_trend(self, dataframe: DataFrame, metadata: dict) -> DataFrame:
    BOT_1_API_URL = "http://<bot_1_ip>:<bot_1_port>/api/v1/"

    status = requests.get(BOT_1_API_URL + "status").json()
    enter_long_condition = False
    for trade in status:
        tag = trade['tag']
        unrealized_profit_percent = trade['unrealized_roi']
        if tag.startswith("XXX-"):
            if unrealized_profit_percent <= -X: 
                enter_long_condition = True
                break

    dataframe.loc[
        (
            (dataframe['volume'] > 0) &
            enter_long_condition 
        ),
        'buy'] = 1
    return dataframe

I'm not planning on doing any backtesting or hyperopt in such a state for obvious reasons. My main concern is the potential performance impact on the bot's operations due to the added API calls. Additionally, if possible I'd like Bot2-6 to only perform calculations and execute trades once Bot 1's first trade has been triggered and certain conditions are met, not sure if there is another more suitable option to put the Bot2-6 on idle or pause the recalculation, invoking stop entry if that makes sense.

[xmatthias]

the way you have it rn - you'll lock your bot up if the "main" one disappears - as your call has no timeout.

there's also no error handling - so eventually, this code will probably die. aside from that - 'buy' is really deprecated and should not be used on new strategies.

[darioneto]

Yes, you are correct, This was just a rough draft to understand if my assumptions were correct. I managed to put this code below to a test. It's pretty simplistic in its approach but works.

        BOT_1_API_URL = "http://192.168.3.207:8080/api/v1/"
        username = "admin"
        password = "admin"

        max_attempts = 3  
        status = [] 
        for attempt in range(max_attempts):
            try:
                response = requests.get(BOT_1_API_URL + "status", timeout=10, auth=requests.auth.HTTPBasicAuth(username, password))  
                if response.status_code == 200:
                    status = response.json()
                    break  
                else:
                    logging.error(f"Error fetching status: HTTP {response.status_code}")
            except requests.exceptions.Timeout:
                logging.error("Timeout occurred while fetching status")
            except requests.exceptions.ConnectionError:
                logging.error("Network problem occurred while fetching status")
            except requests.exceptions.HTTPError:
                logging.error("Invalid HTTP response while fetching status")
            except requests.exceptions.TooManyRedirects:
                logging.error("Too many redirects while fetching status")
            except Exception as e:
                logging.error(f"Error fetching status: {e}")

        enter_long_condition = False
        enter_short_condition = False

        for trade in status:
            tag = trade['enter_tag']
            unrealized_profit_percent = trade['profit_ratio']
            if tag.startswith("ETH-LONG-"):
                if unrealized_profit_percent <= -0.0055: 
                    enter_long_condition = True
                    logging.info(f"Enter long condition met with tag {tag} and unrealized profit percent {unrealized_profit_percent}")
                    break
            elif tag.startswith("ETH-SHORT-"):
                if unrealized_profit_percent <= -0.0055: 
                    enter_short_condition = True
                    logging.info(f"Enter short condition met with tag {tag} and unrealized profit percent {unrealized_profit_percent}")
                    break

Is this something half decent to put onto prod you reckon?