KomodoPlatform / komodo-defi-framework

This is the official Komodo DeFi Framework repository
https://komodoplatform.com/en/docs/komodo-defi-framework/
106 stars 94 forks source link

reduce number of `eth_gasPrice` requests to EVM RPC nodes #1882

Open cipig opened 1 year ago

cipig commented 1 year ago

The call is done multiple times and sent to the EVM RPC nodes. mm2 does it on every setprice request and i do it too in my bot for every token (with get_trade_fee) before posting the order, so i can calculate the min_volume i want to set. There are several requests for eth_gasPrice from the same mm2 node to the same RPC node in a short period of time when you are a market maker. Just imagine using USDT-PLG20 as base coin and posting orders to other PLG20 tokens... USDT-PLG20/ETH-PLG20, then USDT-PLG20/DAI-PLG20 and so on... for 50 tokens, as fast as possible. That would be several eth_gasPrice requests in a short period of time sent to the same PLG20 RPC node.

Would it be possible/make sense to save on those requests by caching the response from the RPC node about fee for eg avg_blocktime x? If x=2 then it would be 1.8s 2 = 3.6s for PLG20.

Another way of saving on these requests would be to use the configured gas_station_url exclusively, so if that is set, no getfee requests get sent to the RPC nodes, everything is sent to the gas_station_url.

shamardy commented 1 year ago

These are the calls to RPC nodes that gets done before a maker order is placed if the order is between 2 ERC20 coins

check_balance_for_maker_swap
    get_sender_trade_fee
        1 - make_gas_station_request if there is a gas station URL
        2 - eth_gasPrice
        3 - eth_feeHistory
        4 - Call to allowance

    get_receiver_trade_fee
        5 - make_gas_station_request if there is a gas station URL
        6 - eth_gasPrice
        7 - eth_feeHistory
        8 - Call to allowance

check_my_coin_balance_for_swap
    9 - Call to balanceOf for ERC20 Balance
    10 - eth_getBalance for base coin balance

check_other_coin_balance_for_swap
    11 - Call to balanceOf for ERC20 Balance
    12 - eth_getBalance for base coin balance

Caching fees globally for a small time (per base coin) and allowance per ERC20 coin (update allowance cache when call to approve is done) can reduce the calls a lot. As for the balance calls, the below calls can be done concurrently to reduce overhead.

check_my_coin_balance_for_swap
    9 - Call to balanceOf for ERC20 Balance
    10 - eth_getBalance for base coin balance

check_other_coin_balance_for_swap
    11 - Call to balanceOf for ERC20 Balance
    12 - eth_getBalance for base coin balance
cipig commented 1 year ago

seeing errors like this from time to time from https://rpc.api.moonbeam.network

21 05:39:29, coins::eth:4523] ERROR Error request MethodCall(MethodCall { jsonrpc: Some(V2), method: "eth_gasPrice", params: Array([]), id: Num(26865) }) failed: Transport("Server 'HttpTransportNode { uri: https://rpc.api.moonbeam.network/, gui_auth: false }' response !200: 429 Too Many Requests, Too many connections. Please try again later.");  on eth_gasPrice request

21 06:01:18, coins::eth:4523] ERROR Error request MethodCall(MethodCall { jsonrpc: Some(V2), method: "eth_gasPrice", params: Array([]), id: Num(27035) }) failed: Transport("Server 'HttpTransportNode { uri: https://rpc.api.moonbeam.network/, gui_auth: false }' response !200: 429 Too Many Requests, Too many connections. Please try again later.");  on eth_gasPrice request

429 Too Many Requests, Too many connections on eth_gasPrice

DEX metrics shows this for GLMR · 2023-09-21 05:01:18 +0000 [coin=GLMR client=ethereum] rpc_client.request.count=28633 rpc_client.response.count=28633 rpc_client.traffic.in=2327280 rpc_client.traffic.out=2744422

but i set \"metrics_interval\":43200, not sure if that influences the values

cipig commented 10 months ago

can we get this caching implemented? the IPs of 2 of my market maker nodes seems to have been blacklisted because of too many requests and i had to shut them down till there is a solution for this the fee at the time of order posting is irrelevant anyway, since it has nothing to do with the fee at the time the swap starts, which can be 1 minute or 1 day after order posting