cirocosta / monero-exporter

Prometheus exporter for monero metrics.
https://github.com/monero-project/monero
Apache License 2.0
22 stars 8 forks source link

cache `/get_transactions` call on `lastblocks` collector #4

Open cirocosta opened 2 years ago

cirocosta commented 2 years ago

depending on the scraping interval, it can be somewhat expensive on monerod to serve the requests it takes from monero-exporter.

it we look at the lastblocks collector's fetchData (when it reaches out to monerod), we can note the call to /get_transactions passing the hash of every transaction included in that block:

https://github.com/cirocosta/monero-exporter/blob/652cf4022780a8d4f101e729a2a097bb398ddff9/pkg/collector/collector_lastblock.go#L79-L82

as we expect blocks to arrive every 2min, a prometheus instance configured with a polling interval of 15s would be making 8 times more calls to that endpoint than it really should (as they'll all return the same answer).

in that case, because the call /get_block is made using a hash (rather than a height), we could cache the response from get_transactions for such block, thus reducing the load on monerod.

BEFORE

        on every scrape:

                lastBlockHeader := get_lastblock_header()
                lastBlock := get_block(lastBlockHeader.hash)
                txns := get_transactions(lastBlock.hashes)

AFTER

        on every scrape:

                lastBlockHeader := get_lastblock_header()

                cacheEntry, found := blocksCache.get(lastBlockHeader.hash)
                if found {
                        // we're good
                        return
                }

                lastBlock := get_block(lastBlockHeader.hash)
                txns := get_transactions(lastBlock.hashes)
cirocosta commented 2 years ago

note that get_transactions is indeed solely used by the lastblock collector - transactionpool does not need it as the call to get_transaction_pool already returns the detailed information we need.

looking at rpc_access_tracking, we can tell get_transaction is indeed quite expensive atm

 ~ $ monero daemon rpc-access-tracking
RPC                             COUNT   TIME SPENT SERVING      CREDITS
get_version                     90      499.904µs               0
hard_fork_info                  46      611.606µs               46
get_alternate_chains            1       952.902µs               0
rpc_access_info                 416     1.807211ms              0
get_public_nodes                1       3.617268ms              0
get_txpool_backlog              81      11.612045ms             81
get_block_header_by_height      154     15.306969ms             154
get_block_headers_range         58      17.226973ms             58
get_net_stats                   12434   23.167619ms             0
get_outs                        21      31.506649ms             161
getblockcount                   8578    37.103013ms             0
get_outs_bin                    13      85.465299ms             3149
sync_info                       10      91.58133ms              20
get_bans                        8577    118.654689ms            0
send_raw_tx                     12      175.204242ms            1200
get_height                      484     183.888138ms            0
rpc_access_tracking             14847   346.920509ms            0
get_hashes                      281     2.024486379s            5886
get_output_distribution_bin     13      2.359060579s            260
get_base_fee_estimate           8604    4.66403806s             8604
get_last_block_header           28780   46.608057363s           28780
get_peer_list                   12561   1m10.117860338s         0
get_block                       20159   1m31.193389823s         20159
get_blocks                      99032   2m30.302909893s         137586
get_transaction_pool_stats      20121   4m3.598594881s          20121
get_connections                 31671   4m27.280631191s         0
get_transaction_pool            11606   5m5.373659997s          513958      <<<
get_transaction_pool_hashes     98092   8m3.274738272s          131568
get_info                        65858   10m3.184014312s         65858
get_transactions                93120   37m27.277965493s        592453      <<<

even though when we divide count by time, get_transaction_pool reveals to be more expensive (which sure, makes sense)

get_transactions       24135840.310619477 
get_transaction_pool   26308944.62566707

I don't think we can do much about get_transaction_pool though

cirocosta commented 2 years ago

how much of a big deal this is? well, not a lot

Screen Shot 2021-08-03 at 8 47 38 AM

but, there's clearly an opportunity for improvements here