search

hirosystems / ordhook

Build indexers, standards and protocols on top of Ordinals and Inscriptions (BRC20, etc).
Apache License 2.0
190 stars 56 forks source link

Unable to evaluate predicate on Bitcoin chainstate #283

Closed 4ker-dep closed 5 months ago

4ker-dep commented 7 months ago

Hello. I encountered a new problem while running ordhook as a container inside a k8s cluster on a node with the following characteristics:

OS: linux (amd64) OS Image: Amazon Linux 2 Kernel version: 5.10.199-190.747.amzn2.x86_64 Container runtime: containerd://1.7.2 Kubelet version: v1.24.17-eks-e71965b

running hirosystems/ordhook:2.1.0 image with the configuration in use described below:

[storage]
working_dir = "/opt/ordhook"

# The Http Api allows you to register / deregister
# dynamically predicates.
# Disable by default.
#
 [http_api]
 http_port = 20456
 database_uri = "xxx"

[network]
mode = "mainnet"
bitcoind_rpc_url = "xxx"
bitcoind_rpc_username = "xxx"
bitcoind_rpc_password = "xxx"
# Bitcoin block events can be received by Chainhook
# either through a Bitcoin node's ZeroMQ interface,
# or through the Stacks node. Zmq is being
# used by default:
bitcoind_zmq_url = "xxx"
# but stacks can also be used:
# stacks_node_rpc_url = "xxx"

[resources]
ulimit = 2048
cpu_core_available = 4
memory_available = 30
bitcoind_rpc_threads = 4
bitcoind_rpc_timeout = 15
expected_observers_count = 1

# Disable the following section if the state
# must be built locally
 [snapshot]
 download_url = "https://archive.hiro.so/mainnet/ordhook/mainnet-ordhook-sqlite-latest"

[logs]
ordinals_internals = true
chainhook_internals = true

The received errors logs are:

{"msg":"Processing block #771717 through 5da4bf95-0472-4c3c-8e02-55af98856bb2 predicate revealed 1 new inscriptions [7] and 0 transfers","level":"INFO","ts":"2024-03-19T19:12:18.173548191Z"}
{"msg":"unable to send request error sending request for url (http://127.0.0.1/payload): error trying to connect: tcp connect error: Connection refused (os error 111)","level":"WARN","ts":"2024-03-19T19:12:18.18250965Z"}
{"msg":"unable to send request error sending request for url (http://127.0.0.1/payload): error trying to connect: tcp connect error: Connection refused (os error 111)","level":"WARN","ts":"2024-03-19T19:12:21.182762497Z"}
{"msg":"unable to send request error sending request for url (http://127.0.0.1/payload): error trying to connect: tcp connect error: Connection refused (os error 111)","level":"WARN","ts":"2024-03-19T19:12:24.183030687Z"}
{"msg":"unable to send request error sending request for url (http://127.0.0.1/payload): error trying to connect: tcp connect error: Connection refused (os error 111)","level":"WARN","ts":"2024-03-19T19:12:27.183251339Z"}
{"msg":"unable to send request error sending request for url (http://127.0.0.1/payload): error trying to connect: tcp connect error: Connection refused (os error 111)","level":"WARN","ts":"2024-03-19T19:12:30.183514121Z"}
{"msg":"unable to send request error sending request for url (http://127.0.0.1/payload): error trying to connect: tcp connect error: Connection refused (os error 111)","level":"WARN","ts":"2024-03-19T19:12:33.183756893Z"}
{"msg":"unable to send request error sending request for url (http://127.0.0.1/payload): error trying to connect: tcp connect error: Connection refused (os error 111)","level":"WARN","ts":"2024-03-19T19:12:36.183993801Z"}
{"msg":"unable to send request error sending request for url (http://127.0.0.1/payload): error trying to connect: tcp connect error: Connection refused (os error 111)","level":"WARN","ts":"2024-03-19T19:12:39.18424525Z"}
...
{"msg":"unable to send request error sending request for url (http://127.0.0.1/payload): error trying to connect: tcp connect error: Connection refused (os error 111)","level":"WARN","ts":"2024-03-19T19:14:03.190995579Z"}
{"msg":"unable to send request error sending request for url (http://127.0.0.1/payload): error trying to connect: tcp connect error: Connection refused (os error 111)","level":"WARN","ts":"2024-03-19T19:14:06.191237287Z"}
{"msg":"unable to send request error sending request for url (http://127.0.0.1/payload): error trying to connect: tcp connect error: Connection refused (os error 111)","level":"WARN","ts":"2024-03-19T19:14:09.191506627Z"}
{"msg":"unable to send request error sending request for url (http://127.0.0.1/payload): error trying to connect: tcp connect error: Connection refused (os error 111)","level":"WARN","ts":"2024-03-19T19:14:12.191749303Z"}
{"msg":"unable to send request error sending request for url (http://127.0.0.1/payload): error trying to connect: tcp connect error: Connection refused (os error 111)","level":"WARN","ts":"2024-03-19T19:14:15.191998965Z"}
{"msg":"unable to send request error sending request for url (http://127.0.0.1/payload): error trying to connect: tcp connect error: Connection refused (os error 111)","level":"WARN","ts":"2024-03-19T19:14:18.19222868Z"}
{"msg":"unable to send request error sending request for url (http://127.0.0.1/payload): error trying to connect: tcp connect error: Connection refused (os error 111)","level":"WARN","ts":"2024-03-19T19:14:21.192489542Z"}
{"msg":"unable to send request error sending request for url (http://127.0.0.1/payload): error trying to connect: tcp connect error: Connection refused (os error 111)","level":"WARN","ts":"2024-03-19T19:14:24.192760862Z"}
{"msg":"unable to send request error sending request for url (http://127.0.0.1/payload): error trying to connect: tcp connect error: Connection refused (os error 111)","level":"WARN","ts":"2024-03-19T19:14:27.193010865Z"}
{"msg":"unable to send request error sending request for url (http://127.0.0.1/payload): error trying to connect: tcp connect error: Connection refused (os error 111)","level":"WARN","ts":"2024-03-19T19:14:30.193236388Z"}
{"msg":"unable to send request error sending request for url (http://127.0.0.1/payload): error trying to connect: tcp connect error: Connection refused (os error 111)","level":"WARN","ts":"2024-03-19T19:14:33.19348379Z"}
{"msg":"unable to send request error sending request for url (http://127.0.0.1/payload): error trying to connect: tcp connect error: Connection refused (os error 111)","level":"WARN","ts":"2024-03-19T19:14:36.193734248Z"}
{"msg":"unable to send request error sending request for url (http://127.0.0.1/payload): error trying to connect: tcp connect error: Connection refused (os error 111)","level":"WARN","ts":"2024-03-19T19:14:39.193997755Z"}
{"msg":"unable to send request error sending request for url (http://127.0.0.1/payload): error trying to connect: tcp connect error: Connection refused (os error 111)","level":"WARN","ts":"2024-03-19T19:14:42.194232317Z"}
{"msg":"unable to send request error sending request for url (http://127.0.0.1/payload): error trying to connect: tcp connect error: Connection refused (os error 111)","level":"WARN","ts":"2024-03-19T19:14:45.194479221Z"}
{"msg":"unable to send request error sending request for url (http://127.0.0.1/payload): error trying to connect: tcp connect error: Connection refused (os error 111)","level":"WARN","ts":"2024-03-19T19:14:48.194727475Z"}
{"msg":"unable to send request error sending request for url (http://127.0.0.1/payload): error trying to connect: tcp connect error: Connection refused (os error 111)","level":"WARN","ts":"2024-03-19T19:14:51.194987084Z"}
{"msg":"unable to send request error sending request for url (http://127.0.0.1/payload): error trying to connect: tcp connect error: Connection refused (os error 111)","level":"WARN","ts":"2024-03-19T19:14:54.195252238Z"}
{"msg":"unable to send request error sending request for url (http://127.0.0.1/payload): error trying to connect: tcp connect error: Connection refused (os error 111)","level":"WARN","ts":"2024-03-19T19:14:57.195482023Z"}
{"msg":"unable to send request error sending request for url (http://127.0.0.1/payload): error trying to connect: tcp connect error: Connection refused (os error 111)","level":"WARN","ts":"2024-03-19T19:15:00.195754499Z"}
{"msg":"unable to send request error sending request for url (http://127.0.0.1/payload): error trying to connect: tcp connect error: Connection refused (os error 111)","level":"WARN","ts":"2024-03-19T19:15:03.195985277Z"}
{"msg":"unable to send request error sending request for url (http://127.0.0.1/payload): error trying to connect: tcp connect error: Connection refused (os error 111)","level":"WARN","ts":"2024-03-19T19:15:06.196223049Z"}
{"msg":"unable to send request error sending request for url (http://127.0.0.1/payload): error trying to connect: tcp connect error: Connection refused (os error 111)","level":"WARN","ts":"2024-03-19T19:15:09.196478901Z"}
{"msg":"unable to send request error sending request for url (http://127.0.0.1/payload): error trying to connect: tcp connect error: Connection refused (os error 111)","level":"WARN","ts":"2024-03-19T19:15:12.196730628Z"}
{"msg":"unable to send request error sending request for url (http://127.0.0.1/payload): error trying to connect: tcp connect error: Connection refused (os error 111)","level":"WARN","ts":"2024-03-19T19:15:15.196975277Z"}
{"msg":"unable to send request after several retries. most recent error: unable to send request error sending request for url (http://127.0.0.1/payload): error trying to connect: tcp connect error: Connection refused (os error 111)","level":"WARN","ts":"2024-03-19T19:15:15.197013398Z"}
{"msg":"Unable to evaluate predicate on Bitcoin chainstate: Scan aborted (consecutive action errors >= 3)","level":"ERRO","ts":"2024-03-19T19:15:15.20517833Z"}

I've searched ordhook repository and these seem to originate from chainhook with the source being: components/chainhook-cli/src/service/runloops.rs invoking: scan_bitcoin_chainstate_via_rpc_using_predicate of components/chainhook-cli/src/scan/bitcoin.rs using: process_block_with_predicates , execute_predicates_action and probably failing at handle_bitcoin_hook_action with the http client being misconfigured here for some reason.

For the context: we're still trying to set up an ordinals-api which would connect to this ordhook instance.

Any tips will be greatly appreciated.

chetanbothra commented 5 months ago

@4ker-dep I'm getting same exact issue on my hetzner servers. Did you get a solution for this ?

4ker-dep commented 5 months ago

Hi @chetanbothra . It's funny that you've asked because I've left alone this issue until yesterday. I've added these two envs:

EXTERNAL_HOSTNAME=xxx
CHAINHOOK_AUTO_PREDICATE_REGISTRATION=true

to the my instance of oridnals-api and both are communicating. I still need to workout the final configuration though.

4ker-dep commented 5 months ago

This was misconfiguration of my ordhook / ordinals-api on my part. and this was the solution for me: https://github.com/hirosystems/ordhook/issues/283#issuecomment-2151685403