search

hirosystems / ordhook

Build indexers, standards and protocols on top of Ordinals and Inscriptions (BRC20, etc).
Apache License 2.0
182 stars 54 forks source link

Which configuration should I use to avoid exceeding the memory？ #199

Closed taoyouxian closed 3 months ago

taoyouxian commented 10 months ago

image

I run the Ordhook service in the machine (8C32G), but it exceeds the max.

My Ordhook.toml configuration is as follows:

[limits]
max_number_of_bitcoin_predicates = 100
max_number_of_concurrent_bitcoin_scans = 100
max_number_of_processing_threads = 16
bitcoin_concurrent_http_requests_max = 16
max_caching_memory_size_mb = **28000**
lgalabru commented 10 months ago

hey @taoyouxian, ordhook will use all the memory it can, so 100% of memory is not an unhealthy signal :)

taoyouxian commented 10 months ago

memory

It sounds not very good, so the max memory is 64G needed? what should I do to keep it stable.

lgalabru commented 10 months ago

Our ordinals stack (http://ordinals.hiro.so/) is backed by an instance of ordhook, running on a 32Gb container, and I run ordhook for development purposes on my laptop (16Gb) without issues. Is ordhook crashing on your end after some time?

MikaelBTC commented 10 months ago

Our ordinals stack (http://ordinals.hiro.so/) is backed by an instance of ordhook, running on a 32Gb container, and I run ordhook for development purposes on my laptop (16Gb) without issues. Is ordhook crashing on your end after some time?

I have the similar problem, not crash but very slow to process after some time(maybe 5 blocks behind the mainnet after 2 days). I also run it on a 32G server. Can you share your Ordhook.toml, thanks.

MikaelBTC commented 10 months ago

Also, when ordhook hangs or becomes slow, probably it will show the following logs, maybe re-org or something consumes the memory and CPU.

Nov 05 04:15:08.486 INFO Bitcoin block hash announced #0000000000000000000031d8b0ef873f1f4571aa5267d2ebf62e8e490408a6ee Nov 05 04:15:08.868 INFO Bitcoin block #815386 dispatched for processing Nov 05 04:15:08.868 INFO Start processing Block #815386 (0x0000...a6ee) Nov 05 04:15:08.868 INFO Active fork 0: Fork [Block #815385 (0x0000...86a7), Block #815384 (0x0000...e41b), Block #815383 (0x0000...4f18), Block #815382 (0x0000...bfee), Block #815381 (0x0000...1a27), Block #815380 (0x0000...43ca)], height = 6 Nov 05 04:15:08.868 INFO Trying to append Block #815386 (0x0000...a6ee) to Fork [Block #815385 (0x0000...86a7), Block #815384 (0x0000...e41b), Block #815383 (0x0000...4f18), Block #815382 (0x0000...bfee), Block #815381 (0x0000...1a27), Block #815380 (0x0000...43ca)], height = 6 Nov 05 04:15:08.868 INFO Comparing Block #815385 (0x0000...86a7) with Block #815386 (0x0000...a6ee) Nov 05 04:15:08.868 INFO Active fork: 0 - Fork [Block #815386 (0x0000...a6ee), Block #815385 (0x0000...86a7), Block #815384 (0x0000...e41b), Block #815383 (0x0000...4f18), Block #815382 (0x0000...bfee), Block #815381 (0x0000...1a27), Block #815380 (0x0000...43ca)], height = 7 Nov 05 04:15:08.868 INFO Active fork selected as canonical: 0 Nov 05 04:15:08.889 INFO Handling PropagateBitcoinChainEvent command Nov 05 04:15:08.889 INFO Sending blocks to pre-processor Nov 05 04:15:08.893 INFO Waiting for blocks from pre-processor Nov 05 04:17:11.100 INFO Bitcoin block hash announced #0000000000000000000290b23921ac29bb213a0d79a6af3a6c45abd438a3e71e Nov 05 04:17:11.469 INFO Bitcoin block #815387 dispatched for processing Nov 05 04:17:11.469 INFO Start processing Block #815387 (0x0000...e71e) Nov 05 04:17:11.469 INFO Active fork 0: Fork [Block #815386 (0x0000...a6ee), Block #815385 (0x0000...86a7), Block #815384 (0x0000...e41b), Block #815383 (0x0000...4f18), Block #815382 (0x0000...bfee), Block #815381 (0x0000...1a27)], height = 6 Nov 05 04:17:11.469 INFO Trying to append Block #815387 (0x0000...e71e) to Fork [Block #815386 (0x0000...a6ee), Block #815385 (0x0000...86a7), Block #815384 (0x0000...e41b), Block #815383 (0x0000...4f18), Block #815382 (0x0000...bfee), Block #815381 (0x0000...1a27)], height = 6 Nov 05 04:17:11.469 INFO Comparing Block #815386 (0x0000...a6ee) with Block #815387 (0x0000...e71e) Nov 05 04:17:11.469 INFO Active fork: 0 - Fork [Block #815387 (0x0000...e71e), Block #815386 (0x0000...a6ee), Block #815385 (0x0000...86a7), Block #815384 (0x0000...e41b), Block #815383 (0x0000...4f18), Block #815382 (0x0000...bfee), Block #815381 (0x0000...1a27)], height = 7 Nov 05 04:17:11.469 INFO Active fork selected as canonical: 0

lgalabru commented 10 months ago

Hi @MikaelBTC! thanks for sharing logs. Would you mind enabling the ordinals logs, in your toml config file? I'm working on solving some sqlite locks / slowness in this PR https://github.com/hirosystems/ordhook/pull/200, we're currently re-indexing the chain with it, it should be merged early next week.

lgalabru commented 10 months ago

Also, are you pushing to a hirosystems/ordinals-api observer, or are you using your own components?

MikaelBTC commented 10 months ago

Hi @MikaelBTC! thanks for sharing logs. Would you mind enabling the ordinals logs, in your toml config file? I'm working on solving some sqlite locks / slowness in this PR #200, we're currently re-indexing the chain with it, it should be merged early next week.

Thanks @lgalabru, I have enabled the ordinals and chainhook logs, but no logs printed when hang. Normally there are some logs about inscriptions revealed or moved. The following is my whole toml config.

[storage] working_dir = "/data/btc/indexer/ordhook/ordhook"

[http_api]

[network] mode = "mainnet" bitcoind_rpc_url = "http://127.0.0.1:8332" bitcoind_rpc_username = "" bitcoind_rpc_password = ""

bitcoind_zmq_url = ""

[limits] max_number_of_bitcoin_predicates = 20 max_number_of_concurrent_bitcoin_scans = 20 max_number_of_processing_threads = 16 bitcoin_concurrent_http_requests_max = 16 max_caching_memory_size_mb = 16000

[logs]
ordinals_internals = true chainhook_internals = true

MikaelBTC commented 10 months ago

Also, are you pushing to a hirosystems/ordinals-api observer, or are you using your own components?

I am using my own components for processing the data ordhook pushed.

MikaelBTC commented 10 months ago

Also, are you pushing to a hirosystems/ordinals-api observer, or are you using your own components?

Also I have another ordhook server without pushing, same problem.

Thanks.

lgalabru commented 10 months ago

Hey @MikaelBTC! still actively working on this, I have a new version that I'm stabilizing / benchmarking. We're getting there!

taoyouxian commented 10 months ago

Also, when ordhook hangs or becomes slow, probably it will show the following logs, maybe re-org or something consumes the memory and CPU.

Nov 05 04:15:08.486 INFO Bitcoin block hash announced #0000000000000000000031d8b0ef873f1f4571aa5267d2ebf62e8e490408a6ee Nov 05 04:15:08.868 INFO Bitcoin block #815386 dispatched for processing Nov 05 04:15:08.868 INFO Start processing Block #815386 (0x0000...a6ee) Nov 05 04:15:08.868 INFO Active fork 0: Fork [Block #815385 (0x0000...86a7), Block #815384 (0x0000...e41b), Block #815383 (0x0000...4f18), Block #815382 (0x0000...bfee), Block #815381 (0x0000...1a27), Block #815380 (0x0000...43ca)], height = 6 Nov 05 04:15:08.868 INFO Trying to append Block #815386 (0x0000...a6ee) to Fork [Block #815385 (0x0000...86a7), Block #815384 (0x0000...e41b), Block #815383 (0x0000...4f18), Block #815382 (0x0000...bfee), Block #815381 (0x0000...1a27), Block #815380 (0x0000...43ca)], height = 6 Nov 05 04:15:08.868 INFO Comparing Block #815385 (0x0000...86a7) with Block #815386 (0x0000...a6ee) Nov 05 04:15:08.868 INFO Active fork: 0 - Fork [Block #815386 (0x0000...a6ee), Block #815385 (0x0000...86a7), Block #815384 (0x0000...e41b), Block #815383 (0x0000...4f18), Block #815382 (0x0000...bfee), Block #815381 (0x0000...1a27), Block #815380 (0x0000...43ca)], height = 7 Nov 05 04:15:08.868 INFO Active fork selected as canonical: 0 Nov 05 04:15:08.889 INFO Handling PropagateBitcoinChainEvent command Nov 05 04:15:08.889 INFO Sending blocks to pre-processor Nov 05 04:15:08.893 INFO Waiting for blocks from pre-processor Nov 05 04:17:11.100 INFO Bitcoin block hash announced #0000000000000000000290b23921ac29bb213a0d79a6af3a6c45abd438a3e71e Nov 05 04:17:11.469 INFO Bitcoin block #815387 dispatched for processing Nov 05 04:17:11.469 INFO Start processing Block #815387 (0x0000...e71e) Nov 05 04:17:11.469 INFO Active fork 0: Fork [Block #815386 (0x0000...a6ee), Block #815385 (0x0000...86a7), Block #815384 (0x0000...e41b), Block #815383 (0x0000...4f18), Block #815382 (0x0000...bfee), Block #815381 (0x0000...1a27)], height = 6 Nov 05 04:17:11.469 INFO Trying to append Block #815387 (0x0000...e71e) to Fork [Block #815386 (0x0000...a6ee), Block #815385 (0x0000...86a7), Block #815384 (0x0000...e41b), Block #815383 (0x0000...4f18), Block #815382 (0x0000...bfee), Block #815381 (0x0000...1a27)], height = 6 Nov 05 04:17:11.469 INFO Comparing Block #815386 (0x0000...a6ee) with Block #815387 (0x0000...e71e) Nov 05 04:17:11.469 INFO Active fork: 0 - Fork [Block #815387 (0x0000...e71e), Block #815386 (0x0000...a6ee), Block #815385 (0x0000...86a7), Block #815384 (0x0000...e41b), Block #815383 (0x0000...4f18), Block #815382 (0x0000...bfee), Block #815381 (0x0000...1a27)], height = 7 Nov 05 04:17:11.469 INFO Active fork selected as canonical: 0

Yes, I met the same problem, maybe a bug exists in it, and I restarted it for some time.

MikaelBTC commented 10 months ago

Hey @MikaelBTC! still actively working on this, I have a new version that I'm stabilizing / benchmarking. We're getting there!

Thank you very much! I have started a new ordhook server with 128G memory, it seems like this problem is gone. I need more time to double check it.

MikaelBTC commented 9 months ago

I have some clues, when I run the ordhook on centos 7, the thread [jbd2/sdc1-8] consume most of the disk, then the processing of the ordhook hangs with the previous logs.

lgalabru commented 9 months ago

@MikaelBTC are you building your own docker images? The latest develop should fix our issues, I see a 20x speed in our infrastructure.

MikaelBTC commented 9 months ago

@MikaelBTC are you building your own docker images? The latest develop should fix our issues, I see a 20x speed in our infrastructure.

No, I am just running ordhook on my own server. Thanks for your reply, I will try your latest version.

lgalabru commented 8 months ago

Closing in favor of https://github.com/hirosystems/ordhook/issues/245