Bitfinex - Causing 100% CPU Utilization and eventually crashes.

[0xferit]

Describe the bug After a while, the bot starts using 100% CPU and eventually crashes.

Steps To Reproduce I don't know how to reproduce exactly, but after I run pure market-making for a few days, the problem appears.

Screenshots 25% is the maximum utilization you can get with a single core.

Release version 1.8.0

[rapcmia]

Hi @0xferit good day and thanks for the submitting this ticket. We assume this data was from your AWS or VM instances, we'd like to know if you can possibly share the details of your setup? Also sample strategy configs and number of bots when you generate this data. Thank you 🙇

[tomerqueves]

could you try deactivate page file. it's swap virtual memory and would be overflow. if you deactivates posiby prompt not enough resources but isolate the issue

[rapcmia]

Hi @0xferit , good day We would like to request for follow up with the info we need to reproduce the issue? Else we can close this ticket for now. Thanks

Hi @0xferit good day and thanks for the submitting this ticket. We assume this data was from your AWS or VM instances, we'd like to know if you can possibly share the details of your setup? Also sample strategy configs and number of bots when you generate this data. Thank you 🙇

[rapcmia]

Hi @0xferit we shall close this ticket for now, else please reopen the ticket with details of setup so we can investigate. Thank you

[0xferit]

Sorry, I was busy.

It's a Digital Ocean droplet. It has 8 GB memory. I don't think it's about a memory leak; see the graph.

It's only one bot.

########################################################
###       Pure market making strategy config         ###
########################################################

template_version: 24
strategy: pure_market_making

# Exchange and token parameters.
exchange: bitfinex

# Token trading pair for the exchange, e.g. BTC-USDT
market: REDACTED

# How far away from mid price to place the bid order.
# Spread of 1 = 1% away from mid price at that time.
# Example if mid price is 100 and bid_spread is 1.
# Your bid is placed at 99.
bid_spread: 0.2

# How far away from mid price to place the ask order.
# Spread of 1 = 1% away from mid price at that time.
# Example if mid price is 100 and ask_spread is 1.
# Your bid is placed at 101.
ask_spread: 0.2

# Minimum Spread
# How far away from the mid price to cancel active orders
minimum_spread: 0.1

# Time in seconds before cancelling and placing new orders.
# If the value is 60, the bot cancels active orders and placing new ones after a minute.
order_refresh_time: 120.0

# Time in seconds before replacing existing order with new orders at the same price.
max_order_age: 200.0

# The spread (from mid price) to defer order refresh process to the next cycle.
# (Enter 1 to indicate 1%), value below 0, e.g. -1, is to disable this feature - not recommended.
order_refresh_tolerance_pct: 0.0

# Size of your bid and ask order.
order_amount: 150000.0

# Price band ceiling.
price_ceiling: 1.0

# Price band floor.
price_floor: 0.001

# enable moving price floor and ceiling.
moving_price_band_enabled: false

# Price band ceiling pct.
price_ceiling_pct: 500.0

# Price band floor pct.
price_floor_pct: -500.0

# price_band_refresh_time.
price_band_refresh_time: 300.0

# Whether to alternate between buys and sells (true/false).
ping_pong_enabled: false

# Whether to enable Inventory skew feature (true/false).
inventory_skew_enabled: false

# Target base asset inventory percentage target to be maintained (for Inventory skew feature).
inventory_target_base_pct: 50.0

# The range around the inventory target base percent to maintain, expressed in multiples of total order size (for
# inventory skew feature).
inventory_range_multiplier: 1.0

# Initial price of the base asset. Note: this setting is not affects anything, the price is kept in the database.
inventory_price: 1.0

# Number of levels of orders to place on each side of the order book.
order_levels: 10

# Increase or decrease size of consecutive orders after the first order (if order_levels > 1).
order_level_amount: 0.0

# Order price space between orders (if order_levels > 1).
order_level_spread: 0.2

# How long to wait before placing the next order in case your order gets filled.
filled_order_delay: 1.0

# Whether to stop cancellations of orders on the other side (of the order book),
# when one side is filled (hanging orders feature) (true/false).
hanging_orders_enabled: false

# Spread (from mid price, in percentage) hanging orders will be canceled (Enter 1 to indicate 1%)
hanging_orders_cancel_pct: 50.0

# Whether to enable order optimization mode (true/false).
order_optimization_enabled: false

# The depth in base asset amount to be used for finding top ask (for order optimization mode).
ask_order_optimization_depth: 0.0

# The depth in base asset amount to be used for finding top bid (for order optimization mode).
bid_order_optimization_depth: 0.0

# Whether to enable adding transaction costs to order price calculation (true/false).
add_transaction_costs: false

# The price source (current_market/external_market/custom_api).
price_source: current_market

# The price type (mid_price/last_price/last_own_trade_price/best_bid/best_ask/inventory_cost).
price_type: last_own_trade_price

# An external exchange name (for external exchange pricing source).
price_source_exchange:

# A trading pair for the external exchange, e.g. BTC-USDT (for external exchange pricing source).
price_source_market:

# An external api that returns price (for custom_api pricing source).
price_source_custom_api:

# An interval time in second to update the price from custom api (for custom_api pricing source).
custom_api_update_interval: 5.0

#Take order if they cross order book when external price source is enabled
take_if_crossed:

# Use user provided orders to directly override the orders placed by order_amount and order_level_parameter
# This is an advanced feature and user is expected to directly edit this field in config file
# Below is an sample input, the format is a dictionary, the key is user-defined order name, the value is a list which includes buy/sell, order spread, and order amount
# order_override:
#   order_1: [buy, 0.5, 100]
#   order_2: [buy, 0.75, 200]
#   order_3: [sell, 0.1, 500]
# Please make sure there is a space between : and [
order_override:

# Simpler override config for seperate bid and order level spreads
split_order_levels_enabled: false
bid_order_level_spreads: 0.25,0.5,0.75,1,1.25,1.5,1.75,2,2.25,2.5
ask_order_level_spreads: 0.25,0.5,0.75,1,1.25,1.5,1.75,2,2.25,2.5
bid_order_level_amounts: '99'
ask_order_level_amounts: '99'
# If the strategy should wait to receive cancellations confirmation before creating new orders during refresh time
should_wait_order_cancel_confirmation: true

# For more detailed information, see:
# https://docs.hummingbot.io/strategies/pure-market-making/#configuration-parameters

[0xferit]

Also, when entering 100% utilization, the bot fails to remove orders, and orders quickly pile up until reaching the maximum the exchange API permits.

If you stop the bot at this stage, most of the orders don't get removed. They seem orphaned like the bot no longer knows about them.

@rapcmia

[rapcmia]

Hi @0xferit, good day to you I reopened this ticket so add on our queue for investigation and thanks for the added info.

[rapcmia]

Hi @0xferit, apologies on the delay

• We setup a server with T2.Large, 8 GB memory on AWS and ran a PMM strategy along with another testbots.
• Setup a PMM strategy but instead of bitfinex we used binance with same config
• Setup other testbots on the server as well running on different strategies

Monitoring the long term bot for couple of days we have not occurrence of stuck nor crashed. The CPU ulitization however ramps up to around 90% but that seems to be expected as we are running/install other testbots on the server. My assumption would be its possible that its the connector that causes the issue since there were no updates added to Bitfinex connector.

[0xferit]

The problem is likely due to the Bitfinex connector. I think so because the console is always printing out errors about connection issues on Bitfinex, and often bot fails to clear up some orders, thus orphan orders accumulate until the max number of orders permitted by exchange API is reached. Also, when you stop the bot almost always it fails the clear the order book.

[0xferit]

Hello, any update on this? The problem still persists in version 1.12.0.

[0xferit]

This still persists in the latest version. I kill and reinit the bot every day to cope with it.