Indexing inscriptions got stuck at height: 828102

[4ker-dep]

Hi again! I'm currently running ordhook as a container inside a k8s cluster on a node with the following characteristics:

OS: linux (amd64) OS Image: Amazon Linux 2 Kernel version: 5.10.199-190.747.amzn2.x86_64 Container runtime: containerd://1.7.2 Kubelet version: v1.24.17-eks-e71965b

running hirosystems/ordhook:2.1.0 image with the configuration in use described below:

[storage]
working_dir = "/opt/ordhook"

# The Http Api allows you to register / deregister
# dynamically predicates.
# Disable by default.
#
 [http_api]
 http_port = 20456
 database_uri = "xxx"

[network]
mode = "mainnet"
bitcoind_rpc_url = "xxx"
bitcoind_rpc_username = "xxx"
bitcoind_rpc_password = "xxx"
# Bitcoin block events can be received by Chainhook
# either through a Bitcoin node's ZeroMQ interface,
# or through the Stacks node. Zmq is being
# used by default:
bitcoind_zmq_url = "xxx"
# but stacks can also be used:
# stacks_node_rpc_url = "xxx"

[resources]
ulimit = 2048
cpu_core_available = 4
memory_available = 30
bitcoind_rpc_threads = 4
bitcoind_rpc_timeout = 15
expected_observers_count = 1

# Disable the following section if the state
# must be built locally
 [snapshot]
 download_url = "https://archive.hiro.so/mainnet/ordhook/mainnet-ordhook-sqlite-latest"

[logs]
ordinals_internals = true
chainhook_internals = true

From the looks of it seems it got stuck at height 828102 and restarts of the software are not helping. The available resources are not being used (see screenshots)

{"msg":"Starting service (git_commit = )...","level":"INFO","ts":"2024-03-06T12:10:56.015281488Z"}
{"msg":"Indexing inscriptions from block #828102 to block #833405","level":"INFO","ts":"2024-03-06T12:10:56.141870099Z"}

[lgalabru]

Hi @4ker-dep. The amount of informations is a bit limited I'm afraid. How are you approaching storage?

[4ker-dep]

Thank you @lgalabru! I didn't notice it was a disk issue, after extending the PVC we resumed with the data retrieval.

[4ker-dep]

We're continuing with the following logs:

{"msg":"Starting service (git_commit = )...","level":"INFO","ts":"2024-03-11T14:44:05.440892837Z"}
{"msg":"Indexing inscriptions from block #828102 to block #834222","level":"INFO","ts":"2024-03-11T14:44:05.613178891Z"}
{"msg":"Processing 3 blocks","level":"INFO","ts":"2024-03-11T14:44:14.573623005Z"}
{"msg":"Inscriptions data computation for block #828102 started","level":"INFO","ts":"2024-03-11T14:44:14.575583201Z"}
{"msg":"Number of inscriptions in block #828102 to process: 2297 (L1 cache hits: 0, queue: [828103, 828104], L1 cache len: 0, L2 cache len: 0)","level":"INFO","ts":"2024-03-11T14:44:14.5776314Z"}
{"msg":"Completed ordinal number retrieval for Satpoint 0x9bd0aff4b698ee544a4f12d0d39e8b5794c31194b477876c3873f2057268c9bb:0:0 (block: #459299:726982511, transfers: 111, progress: 1/2297, priority queue: true, thread: 0)","level":"INFO","ts":"2024-03-11T14:44:14.833434891Z"}
{"msg":"Completed ordinal number retrieval for Satpoint 0x833765c0ca5dee0682bd905ba8d8eb4b226591bbe622ade7238368d06bc4d762:0:0 (block: #421986:506951348, transfers: 84, progress: 2/2297, priority queue: true, thread: 0)","level":"INFO","ts":"2024-03-11T14:44:15.001393409Z"}
{"msg":"Completed ordinal number retrieval for Satpoint 0x08cb3db051b2f4a4e9fceb4a58bf2697383dca598e505834854c57b96ba2e33f:0:0 (block: #347332:1201887787, transfers: 734, progress: 3/2297, priority queue: true, thread: 0)","level":"INFO","ts":"2024-03-11T14:44:16.487250918Z"}
{"msg":"Completed ordinal number retrieval for Satpoint 0xfccade8c19d326686d04c3d5844a040fd626ac3c0c48d9723c05a4ccef729657:0:0 (block: #656730:525031860, transfers: 67, progress: 4/2297, priority queue: true, thread: 0)","level":"INFO","ts":"2024-03-11T14:44:16.584317692Z"}
{"msg":"Completed ordinal number retrieval for Satpoint 0x5ab31c1f999de1b90df131309997a814f1173347257ce75cf9bb2cf07401f804:0:0 (block: #111159:2286848870, transfers: 1377, progress: 5/2297, priority queue: true, thread: 1)","level":"INFO","ts":"2024-03-11T14:44:16.988852187Z"}
{"msg":"Completed ordinal number retrieval for Satpoint 0x743abf04036a35b9606270fa5a4d00e29e01c320209116ca19f95482d588101b:0:0 (block: #530059:36309381, transfers: 113, progress: 6/2297, priority queue: true, thread: 1)","level":"INFO","ts":"2024-03-11T14:44:17.179907768Z"}
{"msg":"Completed ordinal number retrieval for Satpoint 0x636dcec656308715ba084ae9dbc78dcf9bdc5cc85fb56d620fef89430f6e91c9:0:0 (block: #134032:4022903108, transfers: 585, progress: 7/2297, priority queue: true, thread: 0)","level":"INFO","ts":"2024-03-11T14:44:17.414852155Z"}
{"msg":"Completed ordinal number retrieval for Satpoint 0x0f90f7f3e4b4568187d0d5f952e997eb81c32a648aab1c966bfa86f31814e0ac:0:0 (block: #656730:525075046, transfers: 67, progress: 8/2297, priority queue: true, thread: 0)","level":"INFO","ts":"2024-03-11T14:44:17.423869592Z"}
{"msg":"Completed ordinal number retrieval for Satpoint 0x537ec29297e93aa3f50d6060e977aa5a696d96eaedcf18a20cf9f2440805ae4b:0:0 (block: #421986:506276076, transfers: 84, progress: 9/2297, priority queue: true, thread: 0)","level":"INFO","ts":"2024-03-11T14:44:17.431745496Z"}
{"msg":"Completed ordinal number retrieval for Satpoint 0xf4c84d0fa4f07b91a233b296902e9461e62dd676c1894d6dd43269301888b8e5:0:0 (block: #656730:524792374, transfers: 67, progress: 10/2297, priority queue: true, thread: 0)","level":"INFO","ts":"2024-03-11T14:44:17.438841461Z"}
{"msg":"Completed ordinal number retrieval for Satpoint 0xf2f6cc060dd1f8c23b8ce5e3979249eb3d8174d3e30f581e87317d20b929cbba:0:30360 (block: #807275:326680062, transfers: 189, progress: 11/2297, priority queue: true, thread: 1)","level":"INFO","ts":"2024-03-11T14:44:17.483283803Z"}
{"msg":"Completed ordinal number retrieval for Satpoint 0x3cc16acdc52d3f56f8289826ad668446453961cc2ccd3480d20a2a353d10b901:0:0 (block: #347332:1201844601, transfers: 734, progress: 12/2297, priority queue: true, thread: 1)","level":"INFO","ts":"2024-03-11T14:44:17.494601751Z"}
{"msg":"Completed ordinal number retrieval for Satpoint 0x2e408518cdc91693d6f77be542c4363517d366150729967f30e749daed7cac2d:0:0 (block: #139699:1881460100, transfers: 515, progress: 13/2297, priority queue: true, thread: 1)","level":"INFO","ts":"2024-03-11T14:44:17.939574473Z"}
{"msg":"Completed ordinal number retrieval for Satpoint 0xf2f6cc060dd1f8c23b8ce5e3979249eb3d8174d3e30f581e87317d20b929cbba:0:24750 (block: #807275:326674452, transfers: 189, progress: 14/2297, priority queue: true, thread: 1)","level":"INFO","ts":"2024-03-11T14:44:17.950522873Z"}
{"msg":"Completed ordinal number retrieval for Satpoint 0xd3aefe83dc27f59994305853fd2048ef49d06e6594a6d42fa3ea57c022f8ac32:0:0 (block: #634235:240096900, transfers: 718, progress: 15/2297, priority queue: true, thread: 1)","level":"INFO","ts":"2024-03-11T14:44:18.774034669Z"}
{"msg":"Completed ordinal number retrieval for Satpoint 0x56ed3cde4371d34f493e41e66f68173a736f8992cc4d76f2b6001513ef0636d9:0:0 (block: #634235:239983046, transfers: 718, progress: 16/2297, priority queue: true, thread: 0)","level":"INFO","ts":"2024-03-11T14:44:18.774235855Z"}
{"msg":"Completed ordinal number retrieval for Satpoint 0x80a6711f5ce4e321f2d591c388e5454960821bcc75bd25895cce4ff07dce8828:0:0 (block: #656730:525251716, transfers: 67, progress: 17/2297, priority queue: true, thread: 1)","level":"INFO","ts":"2024-03-11T14:44:18.785054786Z"}
{"msg":"Completed ordinal number retrieval for Satpoint 0xafe0e720ab4c11236f73f825939b11ef2fb4507564ddcd3b176ea98d19c7e69c:0:0 (block: #656730:525396978, transfers: 67, progress: 18/2297, priority queue: true, thread: 1)","level":"INFO","ts":"2024-03-11T14:44:18.793167106Z"}
{"msg":"Completed ordinal number retrieval for Satpoint 0xd42bb800e3318e7a356820f86433a694627c94a9d92903ad6d2b398a913b2713:0:0 (block: #51743:1931741463, transfers: 188, progress: 19/2297, priority queue: true, thread: 0)","level":"INFO","ts":"2024-03-11T14:44:18.905191995Z"}
{"msg":"Completed ordinal number retrieval for Satpoint 0x8b476dac68488cd0733bda27945448466354178630331279054433f13ce1c71a:0:0 (block: #823035:375466217, transfers: 34, progress: 20/2297, priority queue: true, thread: 0)","level":"INFO","ts":"2024-03-11T14:44:18.936106809Z"}
{"msg":"Completed ordinal number retrieval for Satpoint 0x1866f5544292a111e9eb00a013e8b73a38da8df79cc2da32471cf5165ab7b121:0:0 (block: #96403:3952489060, transfers: 175, progress: 21/2297, priority queue: true, thread: 0)","level":"INFO","ts":"2024-03-11T14:44:19.100746717Z"}
{"msg":"Completed ordinal number retrieval for Satpoint 0x40796b21e3f1c6dd0258385f398645a6b17e1a518b79c83dda1612501894d510:0:0 (block: #297127:74482097, transfers: 331, progress: 22/2297, priority queue: true, thread: 1)","level":"INFO","ts":"2024-03-11T14:44:19.123599695Z"}

Everything seems to work in order after adding more disk space.

[lgalabru]

Sounds good. Note that high quality storage have massive impacts in terms performance.