Closed tenonedesign closed 2 years ago
Chain crashes on startup due to nonce issue while indexing.
(chain.Koinos) [main.cpp:268]: Established connection to mempool (chain.Koinos) [indexer.cpp:66]: Retrieving highest block from block store (chain.Koinos) [indexer.cpp:91]: Indexing to target block - Height: 1998713, ID: 0x1220b9473bc4603f03bd32addee436f71c99aeadd1b5791e62a0558e601523817a41 (chain.Koinos) [controller.cpp:472]: Block application failed - Height: 1998654 ID: 0x122094f9e9a1b7ddd4c169c4b79859c1fc3429f60e87dec06d18366a2321f85f2670, with reason: invalid transaction nonce - account: 1KwM9wFJ9u9qBgeAEs5v32GyB1EQ2bZQBB, nonce: 0x28f4b1b801, current nonce: 0x28f6b0b801 (chain.Koinos) [main.cpp:290]: An unexpected error has occurred: invalid transaction nonce - account: 1KwM9wFJ9u9qBgeAEs5v32GyB1EQ2bZQBB, nonce: 0x28f4b1b801, current nonce: 0x28f6b0b801
scriptbot@koinos:~$ ./start [+] Running 10/0 ⠿ Container koinos-amqp-1 Created 0.0s ⠿ Container koinos-config-1 Created 0.0s ⠿ Container koinos-mempool-1 Created 0.0s ⠿ Container koinos-jsonrpc-1 Created 0.0s ⠿ Container koinos-block_store-1 Created 0.0s ⠿ Container koinos-chain-1 Created 0.0s ⠿ Container koinos-block_producer-1 Created 0.0s ⠿ Container koinos-transaction_store-1 Created 0.0s ⠿ Container koinos-p2p-1 Created 0.0s ⠿ Container koinos-contract_meta_store-1 Created 0.0s Attaching to koinos-amqp-1, koinos-block_producer-1, koinos-block_store-1, koinos-chain-1, koinos-config-1, koinos-contract_meta_store-1, koinos-jsonrpc-1, koinos-mempool-1, koinos-p2p-1, koinos-transaction_store-1 koinos-config-1 | + mkdir -p /koinos/block_producer koinos-config-1 | sending incremental file list koinos-config-1 | + [[ ! -f /koinos/block_producer/private.key ]] koinos-config-1 | + mkdir -p /koinos/chain koinos-config-1 | + rsync -a -v --ignore-existing /koinos-config/genesis_data.json /koinos/chain/genesis_data.json koinos-config-1 | koinos-config-1 | sent 67 bytes received 12 bytes 158.00 bytes/sec koinos-config-1 | total size is 702,260 speedup is 8,889.37 koinos-config-1 | + mkdir -p /koinos/jsonrpc/descriptors koinos-config-1 | + pushd /koinos/jsonrpc/descriptors koinos-config-1 | + wget https://github.com/koinos/koinos-proto-descriptors/raw/master/koinos_descriptors.pb koinos-config-1 | /koinos/jsonrpc/descriptors / koinos-config-1 | Connecting to github.com (140.82.114.4:443) koinos-config-1 | Connecting to raw.githubusercontent.com (185.199.108.133:443) koinos-config-1 | wget: can't open 'koinos_descriptors.pb': File exists koinos-config-1 exited with code 1 koinos-mempool-1 | 2022-11-03 14:34:20.488007 (mempool.Koinos) [main.cpp:136] : Starting mempool... koinos-mempool-1 | 2022-11-03 14:34:20.488303 (mempool.Koinos) [main.cpp:137] : Number of jobs: 2 koinos-mempool-1 | 2022-11-03 14:34:20.488411 (mempool.Koinos) [main.cpp:138] : Transaction expiration: 120s koinos-mempool-1 | 2022-11-03 14:34:20.492879 (mempool.Koinos) [main.cpp:324] : Connecting AMQP client... koinos-mempool-1 | 2022-11-03 14:34:20.494473 (mempool.Koinos) [retryer.cpp:106] : Failure during client connection to AMQP, retrying in 1000ms koinos-jsonrpc-1 | 2022-11-03 14:34:20.818569 (jsonrpc.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/client.go:117] : Connecting client to AMQP server amqp://guest:guest@amqp:5672/ koinos-jsonrpc-1 | 2022-11-03 14:34:20.821803 (jsonrpc.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/connection.go:85] : Dialing AMQP server amqp://guest:guest@amqp:5672/ koinos-block_store-1 | 2022-11-03 14:34:20.821098 (block_store.Koinos) [koinos-block-store/main.go:103] : Opening database at /koinos/block_store/db koinos-jsonrpc-1 | 2022-11-03 14:34:20.831121 (jsonrpc.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/connection.go:88] : AMQP error dialing server: dial tcp 172.19.0.3:5672: connect: connection refused koinos-jsonrpc-1 | 2022-11-03 14:34:20.862458 (jsonrpc.Koinos) [internal/jsonrpc.go:359] : Registered descriptor package: koinos.rpc.p2p koinos-jsonrpc-1 | 2022-11-03 14:34:20.862543 (jsonrpc.Koinos) [internal/jsonrpc.go:359] : Registered descriptor package: koinos.contracts.claim koinos-jsonrpc-1 | 2022-11-03 14:34:20.862564 (jsonrpc.Koinos) [internal/jsonrpc.go:359] : Registered descriptor package: koinos.rpc.contract_meta_store koinos-jsonrpc-1 | 2022-11-03 14:34:20.862581 (jsonrpc.Koinos) [internal/jsonrpc.go:359] : Registered descriptor package: koinos.rpc.transaction_store koinos-jsonrpc-1 | 2022-11-03 14:34:20.862598 (jsonrpc.Koinos) [internal/jsonrpc.go:359] : Registered descriptor package: koinos.chain koinos-jsonrpc-1 | 2022-11-03 14:34:20.862616 (jsonrpc.Koinos) [internal/jsonrpc.go:359] : Registered descriptor package: koinos.rpc.block_store koinos-jsonrpc-1 | 2022-11-03 14:34:20.862633 (jsonrpc.Koinos) [internal/jsonrpc.go:359] : Registered descriptor package: koinos koinos-jsonrpc-1 | 2022-11-03 14:34:20.862651 (jsonrpc.Koinos) [internal/jsonrpc.go:359] : Registered descriptor package: koinos.contracts.pob koinos-jsonrpc-1 | 2022-11-03 14:34:20.862669 (jsonrpc.Koinos) [internal/jsonrpc.go:359] : Registered descriptor package: koinos.chain koinos-jsonrpc-1 | 2022-11-03 14:34:20.862687 (jsonrpc.Koinos) [internal/jsonrpc.go:359] : Registered descriptor package: koinos.contracts.pow koinos-jsonrpc-1 | 2022-11-03 14:34:20.862705 (jsonrpc.Koinos) [internal/jsonrpc.go:359] : Registered descriptor package: koinos koinos-jsonrpc-1 | 2022-11-03 14:34:20.862724 (jsonrpc.Koinos) [internal/jsonrpc.go:359] : Registered descriptor package: koinos.chain koinos-jsonrpc-1 | 2022-11-03 14:34:20.862741 (jsonrpc.Koinos) [internal/jsonrpc.go:359] : Registered descriptor package: koinos.rpc koinos-jsonrpc-1 | 2022-11-03 14:34:20.862774 (jsonrpc.Koinos) [internal/jsonrpc.go:359] : Registered descriptor package: koinos.protocol koinos-jsonrpc-1 | 2022-11-03 14:34:20.862793 (jsonrpc.Koinos) [internal/jsonrpc.go:359] : Registered descriptor package: koinos.contracts.resources koinos-jsonrpc-1 | 2022-11-03 14:34:20.862811 (jsonrpc.Koinos) [internal/jsonrpc.go:359] : Registered descriptor package: koinos.rpc.mempool koinos-jsonrpc-1 | 2022-11-03 14:34:20.862829 (jsonrpc.Koinos) [internal/jsonrpc.go:359] : Registered descriptor package: koinos.chain koinos-jsonrpc-1 | 2022-11-03 14:34:20.862846 (jsonrpc.Koinos) [internal/jsonrpc.go:359] : Registered descriptor package: koinos.chain koinos-jsonrpc-1 | 2022-11-03 14:34:20.862868 (jsonrpc.Koinos) [internal/jsonrpc.go:359] : Registered descriptor package: google.protobuf koinos-jsonrpc-1 | 2022-11-03 14:34:20.862888 (jsonrpc.Koinos) [internal/jsonrpc.go:359] : Registered descriptor package: koinos.chain koinos-jsonrpc-1 | 2022-11-03 14:34:20.862905 (jsonrpc.Koinos) [internal/jsonrpc.go:359] : Registered descriptor package: koinos.contract_meta_store koinos-jsonrpc-1 | 2022-11-03 14:34:20.862937 (jsonrpc.Koinos) [internal/jsonrpc.go:359] : Registered descriptor package: koinos.transaction_store koinos-jsonrpc-1 | 2022-11-03 14:34:20.863001 (jsonrpc.Koinos) [internal/jsonrpc.go:359] : Registered descriptor package: koinos.chain koinos-jsonrpc-1 | 2022-11-03 14:34:20.863017 (jsonrpc.Koinos) [internal/jsonrpc.go:359] : Registered descriptor package: koinos.broadcast koinos-jsonrpc-1 | 2022-11-03 14:34:20.867276 (jsonrpc.Koinos) [internal/jsonrpc.go:359] : Registered descriptor package: google.protobuf koinos-jsonrpc-1 | 2022-11-03 14:34:20.867320 (jsonrpc.Koinos) [internal/jsonrpc.go:359] : Registered descriptor package: koinos.chain koinos-jsonrpc-1 | 2022-11-03 14:34:20.867349 (jsonrpc.Koinos) [internal/jsonrpc.go:359] : Registered descriptor package: koinos.contracts.token koinos-jsonrpc-1 | 2022-11-03 14:34:20.867375 (jsonrpc.Koinos) [internal/jsonrpc.go:359] : Registered descriptor package: koinos.rpc.chain koinos-jsonrpc-1 | 2022-11-03 14:34:20.867402 (jsonrpc.Koinos) [internal/jsonrpc.go:359] : Registered descriptor package: koinos.block_store koinos-jsonrpc-1 | 2022-11-03 14:34:20.867428 (jsonrpc.Koinos) [internal/jsonrpc.go:359] : Registered descriptor package: koinos.contracts.governance koinos-jsonrpc-1 | 2022-11-03 14:34:20.867532 (jsonrpc.Koinos) [koinos-jsonrpc/main.go:280] : Listening on :8080/ koinos-block_store-1 | 2022-11-03 14:34:21.046157 (block_store.Koinos) [v3@v3.2103.2/logger.go:46] : All 497 tables opened in 114ms koinos-block_store-1 | 2022-11-03 14:34:21.049882 (block_store.Koinos) [v3@v3.2103.2/logger.go:46] : Discard stats nextEmptySlot: 0 koinos-block_store-1 | 2022-11-03 14:34:21.050772 (block_store.Koinos) [v3@v3.2103.2/logger.go:46] : Set nextTxnTs to 6146807 koinos-block_store-1 | 2022-11-03 14:34:21.056843 (block_store.Koinos) [v3@v3.2103.2/logger.go:46] : Deleting empty file: /koinos/block_store/db/000132.vlog koinos-block_store-1 | 2022-11-03 14:34:21.059407 (block_store.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/request_handler.go:100] : Connecting request handler to AMQP server amqp://guest:guest@amqp:5672/ koinos-block_store-1 | 2022-11-03 14:34:21.059472 (block_store.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/connection.go:85] : Dialing AMQP server amqp://guest:guest@amqp:5672/ koinos-block_store-1 | 2022-11-03 14:34:21.061500 (blockstore.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/connection.go:88] : AMQP error dialing server: dial tcp 172.19.0.3:5672: connect: connection refused koinos-mempool-1 | 2022-11-03 14:34:21.495416 (mempool.Koinos) [retryer.cpp:73] : Failure during client connection to AMQP, retrying in 2000ms koinos-chain-1 | 2022-11-03 14:34:21.543503 (chain.Koinos) [main.cpp:170] : Using fork resolution algorithm: block-time koinos-chain-1 | koinos-chain-1 | koinos-chain-1 | | |/ / () __ koinos-chain-1 | | ' // | | ' \ / _ \/ | koinos-chain-1 | | . \ () | | | | | () _ \ koinos-chain-1 | ||__/||| ||_/|/ koinos-chain-1 | ...launching network koinos-chain-1 | 2022-11-03 14:34:21.713292 (chain.Koinos) [main.cpp:210] : Chain ID: 0x1220ccb9ea70dca318e95df16dd904836ff804a2a5db9a6fa45ce373f05bec0a7849 koinos-chain-1 | 2022-11-03 14:34:21.718671 (chain.Koinos) [main.cpp:211] : Number of jobs: 2 koinos-chain-1 | 2022-11-03 14:34:21.722084 (chain.Koinos) [controller.cpp:126] : Initialized fizzy VM backend koinos-chain-1 | 2022-11-03 14:34:21.775560 (chain.Koinos) [controller.cpp:195] : Opened database at block - Height: 1998653, ID: 0x122080425d1419e96503d1e84da15d924827885eebbf53d555536fc47e120a7d906e koinos-chain-1 | 2022-11-03 14:34:21.779006 (chain.Koinos) [main.cpp:254] : Connecting AMQP client... koinos-chain-1 | 2022-11-03 14:34:21.784225 (chain.Koinos) [retryer.cpp:106] : Failure during client connection to AMQP, retrying in 1000ms koinos-jsonrpc-1 | 2022-11-03 14:34:21.833009 (jsonrpc.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/connection.go:85] : Dialing AMQP server amqp://guest:guest@amqp:5672/ koinos-jsonrpc-1 | 2022-11-03 14:34:21.851319 (jsonrpc.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/connection.go:88] : AMQP error dialing server: dial tcp 172.19.0.3:5672: connect: connection refused koinos-block_store-1 | 2022-11-03 14:34:22.062224 (block_store.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/connection.go:85] : Dialing AMQP server amqp://guest:guest@amqp:5672/ koinos-block_store-1 | 2022-11-03 14:34:22.073246 (block_store.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/connection.go:88] : AMQP error dialing server: dial tcp 172.19.0.3:5672: connect: connection refused koinos-chain-1 | 2022-11-03 14:34:22.787401 (chain.Koinos) [retryer.cpp:73] : Failure during client connection to AMQP, retrying in 2000ms koinos-contract_meta_store-1 | 2022-11-03 14:34:23.253567 (contract_meta_store.Koinos) [koinos-contract-meta-store/main.go:105] : Opening database at /koinos/contract_meta_store/db koinos-contract_meta_store-1 | 2022-11-03 14:34:23.268685 (contract_meta_store.Koinos) [v3@v3.2103.2/logger.go:46] : All 1 tables opened in 0s koinos-contract_meta_store-1 | 2022-11-03 14:34:23.271898 (contract_meta_store.Koinos) [v3@v3.2103.2/logger.go:46] : Discard stats nextEmptySlot: 0 koinos-contract_meta_store-1 | 2022-11-03 14:34:23.272372 (contract_meta_store.Koinos) [v3@v3.2103.2/logger.go:46] : Set nextTxnTs to 742 koinos-contract_meta_store-1 | 2022-11-03 14:34:23.272910 (contract_meta_store.Koinos) [v3@v3.2103.2/logger.go:46] : Deleting empty file: /koinos/contract_meta_store/db/000125.vlog koinos-contract_meta_store-1 | 2022-11-03 14:34:23.273890 (contract_meta_store.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/request_handler.go:100] : Connecting request handler to AMQP server amqp://guest:guest@amqp:5672/ koinos-contract_meta_store-1 | 2022-11-03 14:34:23.274330 (contract_meta_store.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/connection.go:85] : Dialing AMQP server amqp://guest:guest@amqp:5672/ koinos-contract_meta_store-1 | 2022-11-03 14:34:23.276019 (contract_meta_store.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/connection.go:88] : AMQP error dialing server: dial tcp 172.19.0.3:5672: connect: connection refused koinos-p2p-1 | 2022-11-03 14:34:23.409106 (p2p.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/client.go:117] : Connecting client to AMQP server amqp://guest:guest@amqp:5672/ koinos-p2p-1 | 2022-11-03 14:34:23.436144 (p2p.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/connection.go:85] : Dialing AMQP server amqp://guest:guest@amqp:5672/ koinos-p2p-1 | 2022-11-03 14:34:23.447552 (p2p.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/connection.go:88] : AMQP error dialing server: dial tcp 172.19.0.3:5672: connect: connection refused koinos-mempool-1 | 2022-11-03 14:34:23.512744 (mempool.Koinos) [retryer.cpp:73] : Failure during client connection to AMQP, retrying in 4000ms koinos-block_producer-1 | 2022-11-03 14:34:23.648769 (block_producer.Koinos) [main.cpp:195] : Public address: 1ByFWv2tFPB8dXktFWmRyhGL1VhU2Kwuwa koinos-block_producer-1 | 2022-11-03 14:34:23.649509 (block_producer.Koinos) [main.cpp:196] : Public key: ArdeHj2gNUzqtuh8Rq5mA2i2JzlNo54TprR3SRsthVqN koinos-block_producer-1 | 2022-11-03 14:34:23.649731 (block_producer.Koinos) [main.cpp:198] : Producer address: 15a6ZZR9M52Vt5ppW4iqoQEUfuNxrasgjF koinos-block_producer-1 | 2022-11-03 14:34:23.653254 (block_producer.Koinos) [main.cpp:199] : Block resource utilization lower bound: 75%, upper bound: 90% koinos-block_producer-1 | 2022-11-03 14:34:23.653681 (block_producer.Koinos) [main.cpp:200] : Maximum transaction inclusion attempts per block: 2000 koinos-block_producer-1 | 2022-11-03 14:34:23.654697 (block_producer.Koinos) [main.cpp:244] : Connecting AMQP client... koinos-block_producer-1 | 2022-11-03 14:34:23.655712 (block_producer.Koinos) [retryer.cpp:106] : Failure during client connection to AMQP, retrying in 1000ms koinos-transaction_store-1 | 2022-11-03 14:34:23.667450 (transaction_store.Koinos) [koinos-transaction-store/main.go:99] : Opening database at /koinos/transaction_store/db koinos-transaction_store-1 | 2022-11-03 14:34:23.873577 (transaction_store.Koinos) [v3@v3.2103.2/logger.go:46] : All 507 tables opened in 150ms koinos-transaction_store-1 | 2022-11-03 14:34:23.877067 (transaction_store.Koinos) [v3@v3.2103.2/logger.go:46] : Discard stats nextEmptySlot: 0 koinos-transaction_store-1 | 2022-11-03 14:34:23.877756 (transaction_store.Koinos) [v3@v3.2103.2/logger.go:46] : Set nextTxnTs to 24721138 koinos-transaction_store-1 | 2022-11-03 14:34:23.881803 (transaction_store.Koinos) [v3@v3.2103.2/logger.go:46] : Deleting empty file: /koinos/transaction_store/db/000125.vlog koinos-transaction_store-1 | 2022-11-03 14:34:23.883511 (transaction_store.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/request_handler.go:100] : Connecting request handler to AMQP server amqp://guest:guest@amqp:5672/ koinos-transaction_store-1 | 2022-11-03 14:34:23.883972 (transaction_store.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/connection.go:85] : Dialing AMQP server amqp://guest:guest@amqp:5672/ koinos-transaction_store-1 | 2022-11-03 14:34:23.888223 (transaction_store.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/connection.go:88] : AMQP error dialing server: dial tcp 172.19.0.3:5672: connect: connection refused koinos-contract_meta_store-1 | 2022-11-03 14:34:24.276394 (contract_meta_store.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/connection.go:85] : Dialing AMQP server amqp://guest:guest@amqp:5672/ koinos-contract_meta_store-1 | 2022-11-03 14:34:24.277832 (contract_meta_store.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/connection.go:88] : AMQP error dialing server: dial tcp 172.19.0.3:5672: connect: connection refused koinos-p2p-1 | 2022-11-03 14:34:24.449432 (p2p.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/connection.go:85] : Dialing AMQP server amqp://guest:guest@amqp:5672/ koinos-p2p-1 | 2022-11-03 14:34:24.450747 (p2p.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/connection.go:88] : AMQP error dialing server: dial tcp 172.19.0.3:5672: connect: connection refused koinos-block_producer-1 | 2022-11-03 14:34:24.656681 (block_producer.Koinos) [retryer.cpp:73] : Failure during client connection to AMQP, retrying in 2000ms koinos-chain-1 | 2022-11-03 14:34:24.789233 (chain.Koinos) [retryer.cpp:73] : Failure during client connection to AMQP, retrying in 4000ms koinos-jsonrpc-1 | 2022-11-03 14:34:24.852650 (jsonrpc.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/connection.go:85] : Dialing AMQP server amqp://guest:guest@amqp:5672/ koinos-jsonrpc-1 | 2022-11-03 14:34:24.853982 (jsonrpc.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/connection.go:88] : AMQP error dialing server: dial tcp 172.19.0.3:5672: connect: connection refused koinos-transaction_store-1 | 2022-11-03 14:34:24.889162 (transaction_store.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/connection.go:85] : Dialing AMQP server amqp://guest:guest@amqp:5672/ koinos-transaction_store-1 | 2022-11-03 14:34:24.890306 (transaction_store.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/connection.go:88] : AMQP error dialing server: dial tcp 172.19.0.3:5672: connect: connection refused koinos-block_store-1 | 2022-11-03 14:34:25.074158 (block_store.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/connection.go:85] : Dialing AMQP server amqp://guest:guest@amqp:5672/ koinos-block_store-1 | 2022-11-03 14:34:25.075451 (block_store.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/connection.go:88] : AMQP error dialing server: dial tcp 172.19.0.3:5672: connect: connection refused koinos-amqp-1 | 2022-11-03 14:34:25.478285+00:00 [notice] <0.44.0> Application syslog exited with reason: stopped koinos-amqp-1 | 2022-11-03 14:34:25.483577+00:00 [notice] <0.229.0> Logging: switching to configured handler(s); following messages may not be visible in this log output koinos-amqp-1 | 2022-11-03 14:34:25.499080+00:00 [notice] <0.229.0> Logging: configured log handlers are now ACTIVE koinos-block_producer-1 | 2022-11-03 14:34:26.658504 (block_producer.Koinos) [retryer.cpp:73] : Failure during client connection to AMQP, retrying in 4000ms koinos-contract_meta_store-1 | 2022-11-03 14:34:27.278947 (contract_meta_store.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/connection.go:85] : Dialing AMQP server amqp://guest:guest@amqp:5672/ koinos-contract_meta_store-1 | 2022-11-03 14:34:27.279887 (contract_meta_store.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/connection.go:88] : AMQP error dialing server: dial tcp 172.19.0.3:5672: connect: connection refused koinos-p2p-1 | 2022-11-03 14:34:27.452024 (p2p.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/connection.go:85] : Dialing AMQP server amqp://guest:guest@amqp:5672/ koinos-p2p-1 | 2022-11-03 14:34:27.453525 (p2p.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/connection.go:88] : AMQP error dialing server: dial tcp 172.19.0.3:5672: connect: connection refused koinos-mempool-1 | 2022-11-03 14:34:27.514315 (mempool.Koinos) [retryer.cpp:73] : Failure during client connection to AMQP, retrying in 8000ms koinos-transaction_store-1 | 2022-11-03 14:34:27.890442 (transaction_store.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/connection.go:85] : Dialing AMQP server amqp://guest:guest@amqp:5672/ koinos-transaction_store-1 | 2022-11-03 14:34:27.891622 (transaction_store.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/connection.go:88] : AMQP error dialing server: dial tcp 172.19.0.3:5672: connect: connection refused koinos-amqp-1 | 2022-11-03 14:34:28.339480+00:00 [info] <0.229.0> ra: starting system quorum_queues koinos-amqp-1 | 2022-11-03 14:34:28.339618+00:00 [info] <0.229.0> starting Ra system: quorum_queues in directory: /var/lib/rabbitmq/mnesia/rabbit@bb48e0941744/quorum/rabbit@bb48e0941744 koinos-amqp-1 | 2022-11-03 14:34:28.343160+00:00 [info] <0.297.0> ra system 'quorum_queues' running pre init for 0 registered servers koinos-amqp-1 | 2022-11-03 14:34:28.345058+00:00 [info] <0.300.0> ra: meta data store initialised for system quorum_queues. 0 record(s) recovered koinos-amqp-1 | 2022-11-03 14:34:28.345660+00:00 [notice] <0.308.0> WAL: ra_log_wal init, open tbls: ra_log_open_mem_tables, closed tbls: ra_log_closed_mem_tables koinos-amqp-1 | 2022-11-03 14:34:28.352619+00:00 [info] <0.229.0> ra: starting system coordination koinos-amqp-1 | 2022-11-03 14:34:28.352797+00:00 [info] <0.229.0> starting Ra system: coordination in directory: /var/lib/rabbitmq/mnesia/rabbit@bb48e0941744/coordination/rabbit@bb48e0941744 koinos-amqp-1 | 2022-11-03 14:34:28.356346+00:00 [info] <0.326.0> ra system 'coordination' running pre init for 0 registered servers koinos-amqp-1 | 2022-11-03 14:34:28.358255+00:00 [info] <0.327.0> ra: meta data store initialised for system coordination. 0 record(s) recovered koinos-amqp-1 | 2022-11-03 14:34:28.358675+00:00 [notice] <0.336.0> WAL: ra_coordination_log_wal init, open tbls: ra_coordination_log_open_mem_tables, closed tbls: ra_coordination_log_closed_mem_tables koinos-amqp-1 | 2022-11-03 14:34:28.365588+00:00 [info] <0.229.0> koinos-amqp-1 | 2022-11-03 14:34:28.365588+00:00 [info] <0.229.0> Starting RabbitMQ 3.11.2 on Erlang 25.1.2 [jit] koinos-amqp-1 | 2022-11-03 14:34:28.365588+00:00 [info] <0.229.0> Copyright (c) 2007-2022 VMware, Inc. or its affiliates. koinos-amqp-1 | 2022-11-03 14:34:28.365588+00:00 [info] <0.229.0> Licensed under the MPL 2.0. Website: https://rabbitmq.com koinos-amqp-1 | koinos-amqp-1 | ## ## RabbitMQ 3.11.2 koinos-amqp-1 | ## ## koinos-amqp-1 | ########## Copyright (c) 2007-2022 VMware, Inc. or its affiliates. koinos-amqp-1 | ###### ## koinos-amqp-1 | ########## Licensed under the MPL 2.0. Website: https://rabbitmq.com koinos-amqp-1 | koinos-amqp-1 | Erlang: 25.1.2 [jit] koinos-amqp-1 | TLS Library: OpenSSL - OpenSSL 1.1.1q 5 Jul 2022 koinos-amqp-1 | Release series support status: supported koinos-amqp-1 | koinos-amqp-1 | Doc guides: https://rabbitmq.com/documentation.html koinos-amqp-1 | Support: https://rabbitmq.com/contact.html koinos-amqp-1 | Tutorials: https://rabbitmq.com/getstarted.html koinos-amqp-1 | Monitoring: https://rabbitmq.com/monitoring.html koinos-amqp-1 | koinos-amqp-1 | Logs: /var/log/rabbitmq/rabbit@bb48e0941744_upgrade.log koinos-amqp-1 | koinos-amqp-1 | koinos-amqp-1 | Config file(s): /etc/rabbitmq/rabbitmq.conf koinos-amqp-1 | /etc/rabbitmq/conf.d/10-defaults.conf koinos-amqp-1 | koinos-amqp-1 | Starting broker...2022-11-03 14:34:28.369996+00:00 [info] <0.229.0> koinos-amqp-1 | 2022-11-03 14:34:28.369996+00:00 [info] <0.229.0> node : rabbit@bb48e0941744 koinos-amqp-1 | 2022-11-03 14:34:28.369996+00:00 [info] <0.229.0> home dir : /var/lib/rabbitmq koinos-amqp-1 | 2022-11-03 14:34:28.369996+00:00 [info] <0.229.0> config file(s) : /etc/rabbitmq/rabbitmq.conf koinos-amqp-1 | 2022-11-03 14:34:28.369996+00:00 [info] <0.229.0> : /etc/rabbitmq/conf.d/10-defaults.conf koinos-amqp-1 | 2022-11-03 14:34:28.369996+00:00 [info] <0.229.0> cookie hash : tZM7Wcrd15OH6BUHMxBRkg== koinos-amqp-1 | 2022-11-03 14:34:28.369996+00:00 [info] <0.229.0> log(s) : /var/log/rabbitmq/rabbit@bb48e0941744_upgrade.log koinos-amqp-1 | 2022-11-03 14:34:28.369996+00:00 [info] <0.229.0> : koinos-amqp-1 | 2022-11-03 14:34:28.369996+00:00 [info] <0.229.0> database dir : /var/lib/rabbitmq/mnesia/rabbit@bb48e0941744 koinos-chain-1 | 2022-11-03 14:34:28.790601 (chain.Koinos) [retryer.cpp:73] : Failure during client connection to AMQP, retrying in 8000ms koinos-amqp-1 | 2022-11-03 14:34:28.982825+00:00 [info] <0.229.0> Running boot step pre_boot defined by app rabbit koinos-amqp-1 | 2022-11-03 14:34:28.982917+00:00 [info] <0.229.0> Running boot step rabbit_global_counters defined by app rabbit koinos-amqp-1 | 2022-11-03 14:34:28.984116+00:00 [info] <0.229.0> Running boot step rabbit_osiris_metrics defined by app rabbit koinos-amqp-1 | 2022-11-03 14:34:28.984308+00:00 [info] <0.229.0> Running boot step rabbit_core_metrics defined by app rabbit koinos-amqp-1 | 2022-11-03 14:34:28.985103+00:00 [info] <0.229.0> Running boot step rabbit_alarm defined by app rabbit koinos-amqp-1 | 2022-11-03 14:34:28.990234+00:00 [info] <0.386.0> Memory high watermark set to 3984 MiB (4177851187 bytes) of 9960 MiB (10444627968 bytes) total koinos-amqp-1 | 2022-11-03 14:34:28.994482+00:00 [info] <0.388.0> Enabling free disk space monitoring koinos-amqp-1 | 2022-11-03 14:34:28.994562+00:00 [info] <0.388.0> Disk free limit set to 50MB koinos-amqp-1 | 2022-11-03 14:34:28.997304+00:00 [info] <0.229.0> Running boot step code_server_cache defined by app rabbit koinos-amqp-1 | 2022-11-03 14:34:28.997478+00:00 [info] <0.229.0> Running boot step file_handle_cache defined by app rabbit koinos-amqp-1 | 2022-11-03 14:34:28.998035+00:00 [info] <0.391.0> Limiting to approx 1048479 file handles (943629 sockets) koinos-amqp-1 | 2022-11-03 14:34:28.998284+00:00 [info] <0.392.0> FHC read buffering: OFF koinos-amqp-1 | 2022-11-03 14:34:28.998546+00:00 [info] <0.392.0> FHC write buffering: ON koinos-amqp-1 | 2022-11-03 14:34:28.999693+00:00 [info] <0.229.0> Running boot step worker_pool defined by app rabbit koinos-amqp-1 | 2022-11-03 14:34:28.999813+00:00 [info] <0.350.0> Will use 2 processes for default worker pool koinos-amqp-1 | 2022-11-03 14:34:28.999923+00:00 [info] <0.350.0> Starting worker pool 'worker_pool' with 2 processes in it koinos-amqp-1 | 2022-11-03 14:34:29.001038+00:00 [info] <0.229.0> Running boot step database defined by app rabbit koinos-amqp-1 | 2022-11-03 14:34:29.003676+00:00 [info] <0.229.0> Waiting for Mnesia tables for 30000 ms, 9 retries left koinos-amqp-1 | 2022-11-03 14:34:29.006754+00:00 [info] <0.229.0> Successfully synced tables from a peer koinos-amqp-1 | 2022-11-03 14:34:29.022126+00:00 [info] <0.229.0> Waiting for Mnesia tables for 30000 ms, 9 retries left koinos-amqp-1 | 2022-11-03 14:34:29.022666+00:00 [info] <0.229.0> Successfully synced tables from a peer koinos-amqp-1 | 2022-11-03 14:34:29.022766+00:00 [info] <0.229.0> Peer discovery backend rabbit_peer_discovery_classic_config does not support registration, skipping registration. koinos-amqp-1 | 2022-11-03 14:34:29.022922+00:00 [info] <0.229.0> Running boot step tracking_metadata_store defined by app rabbit koinos-amqp-1 | 2022-11-03 14:34:29.023195+00:00 [info] <0.407.0> Setting up a table for connection tracking on this node: tracked_connection koinos-amqp-1 | 2022-11-03 14:34:29.023351+00:00 [info] <0.407.0> Setting up a table for per-vhost connection counting on this node: tracked_connection_per_vhost koinos-amqp-1 | 2022-11-03 14:34:29.023473+00:00 [info] <0.407.0> Setting up a table for per-user connection counting on this node: tracked_connection_per_user koinos-amqp-1 | 2022-11-03 14:34:29.023594+00:00 [info] <0.407.0> Setting up a table for channel tracking on this node: tracked_channel koinos-amqp-1 | 2022-11-03 14:34:29.023681+00:00 [info] <0.407.0> Setting up a table for channel tracking on this node: tracked_channel_per_user koinos-amqp-1 | 2022-11-03 14:34:29.024584+00:00 [info] <0.229.0> Running boot step networking_metadata_store defined by app rabbit koinos-amqp-1 | 2022-11-03 14:34:29.025092+00:00 [info] <0.229.0> Running boot step database_sync defined by app rabbit koinos-amqp-1 | 2022-11-03 14:34:29.025668+00:00 [info] <0.229.0> Running boot step feature_flags defined by app rabbit koinos-amqp-1 | 2022-11-03 14:34:29.026192+00:00 [info] <0.229.0> Running boot step codec_correctness_check defined by app rabbit koinos-amqp-1 | 2022-11-03 14:34:29.026268+00:00 [info] <0.229.0> Running boot step external_infrastructure defined by app rabbit koinos-amqp-1 | 2022-11-03 14:34:29.026399+00:00 [info] <0.229.0> Running boot step rabbit_event defined by app rabbit koinos-amqp-1 | 2022-11-03 14:34:29.026798+00:00 [info] <0.229.0> Running boot step rabbit_registry defined by app rabbit koinos-amqp-1 | 2022-11-03 14:34:29.027500+00:00 [info] <0.229.0> Running boot step rabbit_auth_mechanism_amqplain defined by app rabbit koinos-amqp-1 | 2022-11-03 14:34:29.027835+00:00 [info] <0.229.0> Running boot step rabbit_auth_mechanism_cr_demo defined by app rabbit koinos-amqp-1 | 2022-11-03 14:34:29.028657+00:00 [info] <0.229.0> Running boot step rabbit_auth_mechanism_plain defined by app rabbit koinos-amqp-1 | 2022-11-03 14:34:29.028993+00:00 [info] <0.229.0> Running boot step rabbit_exchange_type_direct defined by app rabbit koinos-amqp-1 | 2022-11-03 14:34:29.029266+00:00 [info] <0.229.0> Running boot step rabbit_exchange_type_fanout defined by app rabbit koinos-amqp-1 | 2022-11-03 14:34:29.029627+00:00 [info] <0.229.0> Running boot step rabbit_exchange_type_headers defined by app rabbit koinos-amqp-1 | 2022-11-03 14:34:29.030126+00:00 [info] <0.229.0> Running boot step rabbit_exchange_type_topic defined by app rabbit koinos-amqp-1 | 2022-11-03 14:34:29.030503+00:00 [info] <0.229.0> Running boot step rabbit_mirror_queue_mode_all defined by app rabbit koinos-amqp-1 | 2022-11-03 14:34:29.030883+00:00 [info] <0.229.0> Running boot step rabbit_mirror_queue_mode_exactly defined by app rabbit koinos-amqp-1 | 2022-11-03 14:34:29.031165+00:00 [info] <0.229.0> Running boot step rabbit_mirror_queue_mode_nodes defined by app rabbit koinos-amqp-1 | 2022-11-03 14:34:29.031908+00:00 [info] <0.229.0> Running boot step rabbit_priority_queue defined by app rabbit koinos-amqp-1 | 2022-11-03 14:34:29.032005+00:00 [info] <0.229.0> Priority queues enabled, real BQ is rabbit_variable_queue koinos-amqp-1 | 2022-11-03 14:34:29.032239+00:00 [info] <0.229.0> Running boot step rabbit_queue_location_client_local defined by app rabbit koinos-amqp-1 | 2022-11-03 14:34:29.032809+00:00 [info] <0.229.0> Running boot step rabbit_queue_location_min_masters defined by app rabbit koinos-amqp-1 | 2022-11-03 14:34:29.033088+00:00 [info] <0.229.0> Running boot step rabbit_queue_location_random defined by app rabbit koinos-amqp-1 | 2022-11-03 14:34:29.033353+00:00 [info] <0.229.0> Running boot step kernel_ready defined by app rabbit koinos-amqp-1 | 2022-11-03 14:34:29.033537+00:00 [info] <0.229.0> Running boot step rabbit_sysmon_minder defined by app rabbit koinos-amqp-1 | 2022-11-03 14:34:29.034085+00:00 [info] <0.229.0> Running boot step rabbit_epmd_monitor defined by app rabbit koinos-amqp-1 | 2022-11-03 14:34:29.035696+00:00 [info] <0.416.0> epmd monitor knows us, inter-node communication (distribution) port: 25672 koinos-amqp-1 | 2022-11-03 14:34:29.035850+00:00 [info] <0.229.0> Running boot step guid_generator defined by app rabbit koinos-amqp-1 | 2022-11-03 14:34:29.043376+00:00 [info] <0.229.0> Running boot step rabbit_node_monitor defined by app rabbit koinos-amqp-1 | 2022-11-03 14:34:29.043999+00:00 [info] <0.420.0> Starting rabbit_node_monitor koinos-amqp-1 | 2022-11-03 14:34:29.044470+00:00 [info] <0.229.0> Running boot step delegate_sup defined by app rabbit koinos-amqp-1 | 2022-11-03 14:34:29.045404+00:00 [info] <0.229.0> Running boot step rabbit_memory_monitor defined by app rabbit koinos-amqp-1 | 2022-11-03 14:34:29.045853+00:00 [info] <0.229.0> Running boot step rabbit_fifo_dlx_sup defined by app rabbit koinos-amqp-1 | 2022-11-03 14:34:29.046011+00:00 [info] <0.229.0> Running boot step core_initialized defined by app rabbit koinos-amqp-1 | 2022-11-03 14:34:29.046293+00:00 [info] <0.229.0> Running boot step upgrade_queues defined by app rabbit koinos-amqp-1 | 2022-11-03 14:34:29.056866+00:00 [info] <0.229.0> Running boot step channel_tracking defined by app rabbit koinos-amqp-1 | 2022-11-03 14:34:29.056977+00:00 [info] <0.229.0> Running boot step rabbit_channel_tracking_handler defined by app rabbit koinos-amqp-1 | 2022-11-03 14:34:29.057069+00:00 [info] <0.229.0> Running boot step connection_tracking defined by app rabbit koinos-amqp-1 | 2022-11-03 14:34:29.057378+00:00 [info] <0.229.0> Running boot step rabbit_connection_tracking_handler defined by app rabbit koinos-amqp-1 | 2022-11-03 14:34:29.057619+00:00 [info] <0.229.0> Running boot step rabbit_definitions_hashing defined by app rabbit koinos-amqp-1 | 2022-11-03 14:34:29.057759+00:00 [info] <0.229.0> Running boot step rabbit_exchange_parameters defined by app rabbit koinos-amqp-1 | 2022-11-03 14:34:29.058262+00:00 [info] <0.229.0> Running boot step rabbit_mirror_queue_misc defined by app rabbit koinos-amqp-1 | 2022-11-03 14:34:29.059180+00:00 [info] <0.229.0> Running boot step rabbit_policies defined by app rabbit koinos-amqp-1 | 2022-11-03 14:34:29.059910+00:00 [info] <0.229.0> Running boot step rabbit_policy defined by app rabbit koinos-amqp-1 | 2022-11-03 14:34:29.060093+00:00 [info] <0.229.0> Running boot step rabbit_queue_location_validator defined by app rabbit koinos-amqp-1 | 2022-11-03 14:34:29.060508+00:00 [info] <0.229.0> Running boot step rabbit_quorum_memory_manager defined by app rabbit koinos-amqp-1 | 2022-11-03 14:34:29.060843+00:00 [info] <0.229.0> Running boot step rabbit_stream_coordinator defined by app rabbit koinos-amqp-1 | 2022-11-03 14:34:29.061716+00:00 [info] <0.229.0> Running boot step rabbit_vhost_limit defined by app rabbit koinos-amqp-1 | 2022-11-03 14:34:29.062173+00:00 [info] <0.229.0> Running boot step rabbit_mgmt_reset_handler defined by app rabbitmq_management koinos-amqp-1 | 2022-11-03 14:34:29.062303+00:00 [info] <0.229.0> Running boot step rabbit_mgmt_db_handler defined by app rabbitmq_management_agent koinos-amqp-1 | 2022-11-03 14:34:29.062625+00:00 [info] <0.229.0> Management plugin: using rates mode 'basic' koinos-amqp-1 | 2022-11-03 14:34:29.063688+00:00 [info] <0.229.0> Running boot step recovery defined by app rabbit koinos-amqp-1 | 2022-11-03 14:34:29.065551+00:00 [info] <0.454.0> Making sure data directory '/var/lib/rabbitmq/mnesia/rabbit@bb48e0941744/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L' for vhost '/' exists koinos-amqp-1 | 2022-11-03 14:34:29.071144+00:00 [info] <0.454.0> Starting message stores for vhost '/' koinos-amqp-1 | 2022-11-03 14:34:29.071861+00:00 [info] <0.459.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_transient": using rabbit_msg_store_ets_index to provide index koinos-amqp-1 | 2022-11-03 14:34:29.075092+00:00 [info] <0.454.0> Started message store of type transient for vhost '/' koinos-amqp-1 | 2022-11-03 14:34:29.075705+00:00 [info] <0.463.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": using rabbit_msg_store_ets_index to provide index koinos-amqp-1 | 2022-11-03 14:34:29.079201+00:00 [info] <0.454.0> Started message store of type persistent for vhost '/' koinos-amqp-1 | 2022-11-03 14:34:29.089301+00:00 [info] <0.454.0> Recovering 4 queues of type rabbit_classic_queue took 22ms koinos-amqp-1 | 2022-11-03 14:34:29.089442+00:00 [info] <0.454.0> Recovering 0 queues of type rabbit_quorum_queue took 0ms koinos-amqp-1 | 2022-11-03 14:34:29.089506+00:00 [info] <0.454.0> Recovering 0 queues of type rabbit_stream_queue took 0ms koinos-amqp-1 | 2022-11-03 14:34:29.094322+00:00 [info] <0.229.0> Running boot step empty_db_check defined by app rabbit koinos-amqp-1 | 2022-11-03 14:34:29.094593+00:00 [info] <0.229.0> Will not seed default virtual host and user: have definitions to load... koinos-amqp-1 | 2022-11-03 14:34:29.094768+00:00 [info] <0.229.0> Running boot step rabbit_observer_cli defined by app rabbit koinos-amqp-1 | 2022-11-03 14:34:29.095773+00:00 [info] <0.229.0> Running boot step rabbit_looking_glass defined by app rabbit koinos-amqp-1 | 2022-11-03 14:34:29.095867+00:00 [info] <0.229.0> Running boot step rabbit_core_metrics_gc defined by app rabbit koinos-amqp-1 | 2022-11-03 14:34:29.096775+00:00 [info] <0.229.0> Running boot step background_gc defined by app rabbit koinos-amqp-1 | 2022-11-03 14:34:29.097308+00:00 [info] <0.229.0> Running boot step routing_ready defined by app rabbit koinos-amqp-1 | 2022-11-03 14:34:29.097394+00:00 [info] <0.229.0> Running boot step pre_flight defined by app rabbit koinos-amqp-1 | 2022-11-03 14:34:29.097565+00:00 [info] <0.229.0> Running boot step notify_cluster defined by app rabbit koinos-amqp-1 | 2022-11-03 14:34:29.097789+00:00 [info] <0.229.0> Running boot step networking defined by app rabbit koinos-amqp-1 | 2022-11-03 14:34:29.097917+00:00 [info] <0.229.0> Running boot step definition_import_worker_pool defined by app rabbit koinos-amqp-1 | 2022-11-03 14:34:29.098449+00:00 [info] <0.350.0> Starting worker pool 'definition_import_pool' with 2 processes in it koinos-amqp-1 | 2022-11-03 14:34:29.099807+00:00 [info] <0.229.0> Running boot step cluster_name defined by app rabbit koinos-amqp-1 | 2022-11-03 14:34:29.100218+00:00 [info] <0.229.0> Running boot step direct_client defined by app rabbit koinos-amqp-1 | 2022-11-03 14:34:29.100836+00:00 [info] <0.229.0> Running boot step rabbit_maintenance_mode_state defined by app rabbit koinos-amqp-1 | 2022-11-03 14:34:29.101223+00:00 [info] <0.229.0> Creating table rabbit_node_maintenance_states for maintenance mode status koinos-amqp-1 | 2022-11-03 14:34:29.102334+00:00 [info] <0.229.0> Running boot step rabbit_management_load_definitions defined by app rabbitmq_management koinos-amqp-1 | 2022-11-03 14:34:29.102715+00:00 [info] <0.512.0> Resetting node maintenance status koinos-amqp-1 | 2022-11-03 14:34:29.117189+00:00 [info] <0.571.0> Management plugin: HTTP (non-TLS) listener started on port 15672 koinos-amqp-1 | 2022-11-03 14:34:29.117450+00:00 [info] <0.599.0> Statistics database started. koinos-amqp-1 | 2022-11-03 14:34:29.117943+00:00 [info] <0.598.0> Starting worker pool 'management_worker_pool' with 3 processes in it koinos-amqp-1 | 2022-11-03 14:34:29.129547+00:00 [info] <0.613.0> Prometheus metrics: HTTP (non-TLS) listener started on port 15692 koinos-amqp-1 | 2022-11-03 14:34:29.129725+00:00 [info] <0.512.0> Ready to start client connection listeners koinos-amqp-1 | 2022-11-03 14:34:29.132064+00:00 [info] <0.657.0> started TCP listener on [::]:5672 koinos-amqp-1 | completed with 4 plugins. koinos-amqp-1 | 2022-11-03 14:34:29.256186+00:00 [info] <0.512.0> Server startup complete; 4 plugins started. koinos-amqp-1 | 2022-11-03 14:34:29.256186+00:00 [info] <0.512.0> rabbitmq_prometheus koinos-amqp-1 | 2022-11-03 14:34:29.256186+00:00 [info] <0.512.0> rabbitmq_management koinos-amqp-1 | 2022-11-03 14:34:29.256186+00:00 [info] <0.512.0> rabbitmq_web_dispatch koinos-amqp-1 | 2022-11-03 14:34:29.256186+00:00 [info] <0.512.0> rabbitmq_management_agent koinos-jsonrpc-1 | 2022-11-03 14:34:29.856671 (jsonrpc.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/connection.go:85] : Dialing AMQP server amqp://guest:guest@amqp:5672/ koinos-amqp-1 | 2022-11-03 14:34:29.858761+00:00 [info] <0.660.0> accepting AMQP connection <0.660.0> (172.19.0.4:49964 -> 172.19.0.3:5672) koinos-amqp-1 | 2022-11-03 14:34:29.862618+00:00 [info] <0.660.0> connection <0.660.0> (172.19.0.4:49964 -> 172.19.0.3:5672): user 'guest' authenticated and granted access to vhost '/' koinos-jsonrpc-1 | 2022-11-03 14:34:29.870522 (jsonrpc.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/client.go:136] : Client connected koinos-block_store-1 | 2022-11-03 14:34:30.076254 (block_store.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/connection.go:85] : Dialing AMQP server amqp://guest:guest@amqp:5672/ koinos-amqp-1 | 2022-11-03 14:34:30.078415+00:00 [info] <0.679.0> accepting AMQP connection <0.679.0> (172.19.0.6:53072 -> 172.19.0.3:5672) koinos-amqp-1 | 2022-11-03 14:34:30.081356+00:00 [info] <0.679.0> connection <0.679.0> (172.19.0.6:53072 -> 172.19.0.3:5672): user 'guest' authenticated and granted access to vhost '/' koinos-block_store-1 | 2022-11-03 14:34:30.088267 (block_store.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/request_handler.go:132] : Request handler connected koinos-amqp-1 | 2022-11-03 14:34:30.662169+00:00 [info] <0.699.0> accepting AMQP connection <0.699.0> (172.19.0.10:56962 -> 172.19.0.3:5672) koinos-amqp-1 | 2022-11-03 14:34:30.664831+00:00 [info] <0.699.0> connection <0.699.0> (172.19.0.10:56962 -> 172.19.0.3:5672): user 'guest' authenticated and granted access to vhost '/' koinos-amqp-1 | 2022-11-03 14:34:30.666689+00:00 [info] <0.710.0> accepting AMQP connection <0.710.0> (172.19.0.10:56972 -> 172.19.0.3:5672) koinos-amqp-1 | 2022-11-03 14:34:30.668855+00:00 [info] <0.710.0> connection <0.710.0> (172.19.0.10:56972 -> 172.19.0.3:5672): user 'guest' authenticated and granted access to vhost '/' koinos-block_producer-1 | 2022-11-03 14:34:30.673598 (block_producer.Koinos) [main.cpp:246] : Established AMQP client connection to the server koinos-block_producer-1 | 2022-11-03 14:34:30.673804 (block_producer.Koinos) [main.cpp:248] : Attempting to connect to chain... koinos-block_producer-1 | 2022-11-03 14:34:31.675241 (block_producer.Koinos) [client.cpp:437] : No response to client request with correlation ID: EpZ57SuOZLwoJ0XvL30KSd0eShs7PTPJ, within 1000ms koinos-contract_meta_store-1 | 2022-11-03 14:34:32.279994 (contract_meta_store.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/connection.go:85] : Dialing AMQP server amqp://guest:guest@amqp:5672/ koinos-amqp-1 | 2022-11-03 14:34:32.281518+00:00 [info] <0.727.0> accepting AMQP connection <0.727.0> (172.19.0.8:34906 -> 172.19.0.3:5672) koinos-amqp-1 | 2022-11-03 14:34:32.283851+00:00 [info] <0.727.0> connection <0.727.0> (172.19.0.8:34906 -> 172.19.0.3:5672): user 'guest' authenticated and granted access to vhost '/' koinos-contract_meta_store-1 | 2022-11-03 14:34:32.290715 (contract_meta_store.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/request_handler.go:132] : Request handler connected koinos-p2p-1 | 2022-11-03 14:34:32.457717 (p2p.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/connection.go:85] : Dialing AMQP server amqp://guest:guest@amqp:5672/ koinos-amqp-1 | 2022-11-03 14:34:32.459403+00:00 [info] <0.747.0> accepting AMQP connection <0.747.0> (172.19.0.7:40238 -> 172.19.0.3:5672) koinos-amqp-1 | 2022-11-03 14:34:32.461876+00:00 [info] <0.747.0> connection <0.747.0> (172.19.0.7:40238 -> 172.19.0.3:5672): user 'guest' authenticated and granted access to vhost '/' koinos-p2p-1 | 2022-11-03 14:34:32.467202 (p2p.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/client.go:136] : Client connected koinos-p2p-1 | 2022-11-03 14:34:32.467296 (p2p.Koinos) [koinos-p2p/main.go:161] : Attempting to connect to block_store... koinos-p2p-1 | 2022-11-03 14:34:32.469957 (p2p.Koinos) [koinos-p2p/main.go:167] : Connected koinos-p2p-1 | 2022-11-03 14:34:32.470029 (p2p.Koinos) [koinos-p2p/main.go:172] : Attempting to connect to chain... koinos-transaction_store-1 | 2022-11-03 14:34:32.891996 (transaction_store.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/connection.go:85] : Dialing AMQP server amqp://guest:guest@amqp:5672/ koinos-amqp-1 | 2022-11-03 14:34:32.894004+00:00 [info] <0.766.0> accepting AMQP connection <0.766.0> (172.19.0.9:56070 -> 172.19.0.3:5672) koinos-amqp-1 | 2022-11-03 14:34:32.896501+00:00 [info] <0.766.0> connection <0.766.0> (172.19.0.9:56070 -> 172.19.0.3:5672): user 'guest' authenticated and granted access to vhost '/' koinos-transaction_store-1 | 2022-11-03 14:34:32.903669 (transaction_store.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/request_handler.go:132] : Request handler connected koinos-block_producer-1 | 2022-11-03 14:34:33.676624 (block_producer.Koinos) [client.cpp:437] : No response to client request with correlation ID: XdmbccQ6b4dLv6IdeaHdGZQE94gvuFI9, within 2000ms koinos-amqp-1 | 2022-11-03 14:34:35.521737+00:00 [info] <0.786.0> accepting AMQP connection <0.786.0> (172.19.0.5:49772 -> 172.19.0.3:5672) koinos-amqp-1 | 2022-11-03 14:34:35.524333+00:00 [info] <0.786.0> connection <0.786.0> (172.19.0.5:49772 -> 172.19.0.3:5672): user 'guest' authenticated and granted access to vhost '/' koinos-amqp-1 | 2022-11-03 14:34:35.526345+00:00 [info] <0.797.0> accepting AMQP connection <0.797.0> (172.19.0.5:49774 -> 172.19.0.3:5672) koinos-amqp-1 | 2022-11-03 14:34:35.528429+00:00 [info] <0.797.0> connection <0.797.0> (172.19.0.5:49774 -> 172.19.0.3:5672): user 'guest' authenticated and granted access to vhost '/' koinos-mempool-1 | 2022-11-03 14:34:35.532970 (mempool.Koinos) [main.cpp:326] : Established AMQP client connection to the server koinos-mempool-1 | 2022-11-03 14:34:35.533136 (mempool.Koinos) [main.cpp:328] : Connecting AMQP request handler... koinos-amqp-1 | 2022-11-03 14:34:35.534274+00:00 [info] <0.814.0> accepting AMQP connection <0.814.0> (172.19.0.5:49778 -> 172.19.0.3:5672) koinos-amqp-1 | 2022-11-03 14:34:35.536355+00:00 [info] <0.814.0> connection <0.814.0> (172.19.0.5:49778 -> 172.19.0.3:5672): user 'guest' authenticated and granted access to vhost '/' koinos-amqp-1 | 2022-11-03 14:34:35.538111+00:00 [info] <0.825.0> accepting AMQP connection <0.825.0> (172.19.0.5:49782 -> 172.19.0.3:5672) koinos-amqp-1 | 2022-11-03 14:34:35.540201+00:00 [info] <0.825.0> connection <0.825.0> (172.19.0.5:49782 -> 172.19.0.3:5672): user 'guest' authenticated and granted access to vhost '/' koinos-mempool-1 | 2022-11-03 14:34:35.555219 (mempool.Koinos) [main.cpp:330] : Established request handler connection to the AMQP server koinos-mempool-1 | 2022-11-03 14:34:35.555384 (mempool.Koinos) [main.cpp:332] : Listening for requests over AMQP koinos-amqp-1 | 2022-11-03 14:34:36.799099+00:00 [info] <0.855.0> accepting AMQP connection <0.855.0> (172.19.0.2:56204 -> 172.19.0.3:5672) koinos-amqp-1 | 2022-11-03 14:34:36.801761+00:00 [info] <0.855.0> connection <0.855.0> (172.19.0.2:56204 -> 172.19.0.3:5672): user 'guest' authenticated and granted access to vhost '/' koinos-amqp-1 | 2022-11-03 14:34:36.803743+00:00 [info] <0.866.0> accepting AMQP connection <0.866.0> (172.19.0.2:56216 -> 172.19.0.3:5672) koinos-amqp-1 | 2022-11-03 14:34:36.805935+00:00 [info] <0.866.0> connection <0.866.0> (172.19.0.2:56216 -> 172.19.0.3:5672): user 'guest' authenticated and granted access to vhost '/' koinos-chain-1 | 2022-11-03 14:34:36.811006 (chain.Koinos) [main.cpp:256] : Established AMQP client connection to the server koinos-chain-1 | 2022-11-03 14:34:36.811191 (chain.Koinos) [main.cpp:258] : Attempting to connect to block_store... koinos-chain-1 | 2022-11-03 14:34:36.812936 (chain.Koinos) [main.cpp:262] : Established connection to block_store koinos-chain-1 | 2022-11-03 14:34:36.813088 (chain.Koinos) [main.cpp:264] : Attempting to connect to mempool... koinos-chain-1 | 2022-11-03 14:34:36.814523 (chain.Koinos) [main.cpp:268] : Established connection to mempool koinos-chain-1 | 2022-11-03 14:34:36.814897 (chain.Koinos) [indexer.cpp:66] : Retrieving highest block from block store koinos-chain-1 | 2022-11-03 14:34:36.817234 (chain.Koinos) [indexer.cpp:91] : Indexing to target block - Height: 1998713, ID: 0x1220b9473bc4603f03bd32addee436f71c99aeadd1b5791e62a0558e601523817a41 koinos-chain-1 | 2022-11-03 14:34:37.610352 (chain.Koinos) [controller.cpp:472] : Block application failed - Height: 1998654 ID: 0x122094f9e9a1b7ddd4c169c4b79859c1fc3429f60e87dec06d18366a2321f85f2670, with reason: invalid transaction nonce - account: 1KwM9wFJ9u9qBgeAEs5v32GyB1EQ2bZQBB, nonce: 0x28f4b1b801, current nonce: 0x28f6b0b801 koinos-chain-1 | 2022-11-03 14:34:37.632631 (chain.Koinos) [main.cpp:239] : Caught signal, shutting down... koinos-block_producer-1 | 2022-11-03 14:34:37.677799 (block_producer.Koinos) [client.cpp:437] : No response to client request with correlation ID: 6M4V8QqpnwqwtMxrjFzixtjdXiqWq4h0, within 4000ms koinos-block_producer-1 | 2022-11-03 14:34:45.679309 (block_producer.Koinos) [client.cpp:437] : No response to client request with correlation ID: IW2yannkGeABim5t6gJm3QDWs0I4bo0Q, within 8000ms koinos-block_producer-1 | 2022-11-03 14:35:01.679987 (block_producer.Koinos) [client.cpp:437] : No response to client request with correlation ID: KBvyDEpfvbLfPwrcCWxwNC8s6cDqEYH7, within 16000ms koinos-block_producer-1 | 2022-11-03 14:35:31.680683 (block_producer.Koinos) [client.cpp:437] : No response to client request with correlation ID: QgWq5qwwpDSGKyEMz8yA4PMNHefyQjT4, within 30000ms koinos-block_producer-1 | 2022-11-03 14:36:01.681645 (block_producer.Koinos) [client.cpp:437] : No response to client request with correlation ID: UGP0b4kJXninbpEJWSFI9ai8CwQlOvC8, within 30000ms koinos-block_producer-1 | 2022-11-03 14:36:31.682490 (block_producer.Koinos) [client.cpp:437] : No response to client request with correlation ID: UJxXVJqFr64twEls0quookUVohNvLLtI, within 30000ms koinos-block_producer-1 | 2022-11-03 14:37:01.683907 (block_producer.Koinos) [client.cpp:437] : No response to client request with correlation ID: AVseFkNg5yBhMu6Zzv3Z0C0HHhvgy4vr, within 30000ms koinos-block_producer-1 | 2022-11-03 14:37:31.684976 (block_producer.Koinos) [client.cpp:437] : No response to client request with correlation ID: gsMmKkG0q8YaxuzJsOOjk57vWlUUsZ3H, within 30000ms koinos-block_producer-1 | 2022-11-03 14:38:01.686483 (block_producer.Koinos) [client.cpp:437] : No response to client request with correlation ID: nC1UGfNRZfkTcaQ2PUhx0LvdzQbBqdRE, within 30000ms koinos-block_producer-1 | 2022-11-03 14:38:31.687192 (block_producer.Koinos) [client.cpp:437] : No response to client request with correlation ID: DwDE1cbxhwx0Kin7bxK9RyNmIq0bdPAV, within 30000ms koinos-block_producer-1 | 2022-11-03 14:39:01.688276 (block_producer.Koinos) [client.cpp:437] : No response to client request with correlation ID: tE1tZES3wXNQLx4MLnXZ21SwKEY0zTyG, within 30000ms koinos-block_producer-1 | 2022-11-03 14:39:31.689344 (block_producer.Koinos) [client.cpp:437] : No response to client request with correlation ID: VZUMJOFRkMuGEdsnmUd0SzQba1uFqVCX, within 30000ms koinos-block_producer-1 | 2022-11-03 14:40:01.689801 (block_producer.Koinos) [client.cpp:437] : No response to client request with correlation ID: 9WA0U9b86zQYmCxmeTRyPsxtdJJLGSUN, within 30000ms
No response
- OS: Debian and macOS with latest docker
Expected behavior
tenonedesign
commented
2 years ago tenonedesign
commented
2 years ago
Is there an existing issue for this?
Current behavior
Chain crashes on startup due to nonce issue while indexing.
Log from macOS node
(chain.Koinos) [main.cpp:268]: Established connection to mempool (chain.Koinos) [indexer.cpp:66]: Retrieving highest block from block store (chain.Koinos) [indexer.cpp:91]: Indexing to target block - Height: 1998713, ID: 0x1220b9473bc4603f03bd32addee436f71c99aeadd1b5791e62a0558e601523817a41 (chain.Koinos) [controller.cpp:472]: Block application failed - Height: 1998654 ID: 0x122094f9e9a1b7ddd4c169c4b79859c1fc3429f60e87dec06d18366a2321f85f2670, with reason: invalid transaction nonce - account: 1KwM9wFJ9u9qBgeAEs5v32GyB1EQ2bZQBB, nonce: 0x28f4b1b801, current nonce: 0x28f6b0b801 (chain.Koinos) [main.cpp:290]: An unexpected error has occurred: invalid transaction nonce - account: 1KwM9wFJ9u9qBgeAEs5v32GyB1EQ2bZQBB, nonce: 0x28f4b1b801, current nonce: 0x28f6b0b801
Full log from a debian node using the same data folder
scriptbot@koinos:~$ ./start [+] Running 10/0 ⠿ Container koinos-amqp-1 Created 0.0s ⠿ Container koinos-config-1 Created 0.0s ⠿ Container koinos-mempool-1 Created 0.0s ⠿ Container koinos-jsonrpc-1 Created 0.0s ⠿ Container koinos-block_store-1 Created 0.0s ⠿ Container koinos-chain-1 Created 0.0s ⠿ Container koinos-block_producer-1 Created 0.0s ⠿ Container koinos-transaction_store-1 Created 0.0s ⠿ Container koinos-p2p-1 Created 0.0s ⠿ Container koinos-contract_meta_store-1 Created 0.0s Attaching to koinos-amqp-1, koinos-block_producer-1, koinos-block_store-1, koinos-chain-1, koinos-config-1, koinos-contract_meta_store-1, koinos-jsonrpc-1, koinos-mempool-1, koinos-p2p-1, koinos-transaction_store-1 koinos-config-1 | + mkdir -p /koinos/block_producer koinos-config-1 | sending incremental file list koinos-config-1 | + [[ ! -f /koinos/block_producer/private.key ]] koinos-config-1 | + mkdir -p /koinos/chain koinos-config-1 | + rsync -a -v --ignore-existing /koinos-config/genesis_data.json /koinos/chain/genesis_data.json koinos-config-1 | koinos-config-1 | sent 67 bytes received 12 bytes 158.00 bytes/sec koinos-config-1 | total size is 702,260 speedup is 8,889.37 koinos-config-1 | + mkdir -p /koinos/jsonrpc/descriptors koinos-config-1 | + pushd /koinos/jsonrpc/descriptors koinos-config-1 | + wget https://github.com/koinos/koinos-proto-descriptors/raw/master/koinos_descriptors.pb koinos-config-1 | /koinos/jsonrpc/descriptors / koinos-config-1 | Connecting to github.com (140.82.114.4:443) koinos-config-1 | Connecting to raw.githubusercontent.com (185.199.108.133:443) koinos-config-1 | wget: can't open 'koinos_descriptors.pb': File exists koinos-config-1 exited with code 1 koinos-mempool-1 | 2022-11-03 14:34:20.488007 (mempool.Koinos) [main.cpp:136]: Starting mempool...
koinos-mempool-1 | 2022-11-03 14:34:20.488303 (mempool.Koinos) [main.cpp:137] : Number of jobs: 2
koinos-mempool-1 | 2022-11-03 14:34:20.488411 (mempool.Koinos) [main.cpp:138] : Transaction expiration: 120s
koinos-mempool-1 | 2022-11-03 14:34:20.492879 (mempool.Koinos) [main.cpp:324] : Connecting AMQP client...
koinos-mempool-1 | 2022-11-03 14:34:20.494473 (mempool.Koinos) [retryer.cpp:106] : Failure during client connection to AMQP, retrying in 1000ms
koinos-jsonrpc-1 | 2022-11-03 14:34:20.818569 (jsonrpc.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/client.go:117] : Connecting client to AMQP server amqp://guest:guest@amqp:5672/
koinos-jsonrpc-1 | 2022-11-03 14:34:20.821803 (jsonrpc.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/connection.go:85] : Dialing AMQP server amqp://guest:guest@amqp:5672/
koinos-block_store-1 | 2022-11-03 14:34:20.821098 (block_store.Koinos) [koinos-block-store/main.go:103] : Opening database at /koinos/block_store/db
koinos-jsonrpc-1 | 2022-11-03 14:34:20.831121 (jsonrpc.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/connection.go:88] : AMQP error dialing server: dial tcp 172.19.0.3:5672: connect: connection refused
koinos-jsonrpc-1 | 2022-11-03 14:34:20.862458 (jsonrpc.Koinos) [internal/jsonrpc.go:359] : Registered descriptor package: koinos.rpc.p2p
koinos-jsonrpc-1 | 2022-11-03 14:34:20.862543 (jsonrpc.Koinos) [internal/jsonrpc.go:359] : Registered descriptor package: koinos.contracts.claim
koinos-jsonrpc-1 | 2022-11-03 14:34:20.862564 (jsonrpc.Koinos) [internal/jsonrpc.go:359] : Registered descriptor package: koinos.rpc.contract_meta_store
koinos-jsonrpc-1 | 2022-11-03 14:34:20.862581 (jsonrpc.Koinos) [internal/jsonrpc.go:359] : Registered descriptor package: koinos.rpc.transaction_store
koinos-jsonrpc-1 | 2022-11-03 14:34:20.862598 (jsonrpc.Koinos) [internal/jsonrpc.go:359] : Registered descriptor package: koinos.chain
koinos-jsonrpc-1 | 2022-11-03 14:34:20.862616 (jsonrpc.Koinos) [internal/jsonrpc.go:359] : Registered descriptor package: koinos.rpc.block_store
koinos-jsonrpc-1 | 2022-11-03 14:34:20.862633 (jsonrpc.Koinos) [internal/jsonrpc.go:359] : Registered descriptor package: koinos
koinos-jsonrpc-1 | 2022-11-03 14:34:20.862651 (jsonrpc.Koinos) [internal/jsonrpc.go:359] : Registered descriptor package: koinos.contracts.pob
koinos-jsonrpc-1 | 2022-11-03 14:34:20.862669 (jsonrpc.Koinos) [internal/jsonrpc.go:359] : Registered descriptor package: koinos.chain
koinos-jsonrpc-1 | 2022-11-03 14:34:20.862687 (jsonrpc.Koinos) [internal/jsonrpc.go:359] : Registered descriptor package: koinos.contracts.pow
koinos-jsonrpc-1 | 2022-11-03 14:34:20.862705 (jsonrpc.Koinos) [internal/jsonrpc.go:359] : Registered descriptor package: koinos
koinos-jsonrpc-1 | 2022-11-03 14:34:20.862724 (jsonrpc.Koinos) [internal/jsonrpc.go:359] : Registered descriptor package: koinos.chain
koinos-jsonrpc-1 | 2022-11-03 14:34:20.862741 (jsonrpc.Koinos) [internal/jsonrpc.go:359] : Registered descriptor package: koinos.rpc
koinos-jsonrpc-1 | 2022-11-03 14:34:20.862774 (jsonrpc.Koinos) [internal/jsonrpc.go:359] : Registered descriptor package: koinos.protocol
koinos-jsonrpc-1 | 2022-11-03 14:34:20.862793 (jsonrpc.Koinos) [internal/jsonrpc.go:359] : Registered descriptor package: koinos.contracts.resources
koinos-jsonrpc-1 | 2022-11-03 14:34:20.862811 (jsonrpc.Koinos) [internal/jsonrpc.go:359] : Registered descriptor package: koinos.rpc.mempool
koinos-jsonrpc-1 | 2022-11-03 14:34:20.862829 (jsonrpc.Koinos) [internal/jsonrpc.go:359] : Registered descriptor package: koinos.chain
koinos-jsonrpc-1 | 2022-11-03 14:34:20.862846 (jsonrpc.Koinos) [internal/jsonrpc.go:359] : Registered descriptor package: koinos.chain
koinos-jsonrpc-1 | 2022-11-03 14:34:20.862868 (jsonrpc.Koinos) [internal/jsonrpc.go:359] : Registered descriptor package: google.protobuf
koinos-jsonrpc-1 | 2022-11-03 14:34:20.862888 (jsonrpc.Koinos) [internal/jsonrpc.go:359] : Registered descriptor package: koinos.chain
koinos-jsonrpc-1 | 2022-11-03 14:34:20.862905 (jsonrpc.Koinos) [internal/jsonrpc.go:359] : Registered descriptor package: koinos.contract_meta_store
koinos-jsonrpc-1 | 2022-11-03 14:34:20.862937 (jsonrpc.Koinos) [internal/jsonrpc.go:359] : Registered descriptor package: koinos.transaction_store
koinos-jsonrpc-1 | 2022-11-03 14:34:20.863001 (jsonrpc.Koinos) [internal/jsonrpc.go:359] : Registered descriptor package: koinos.chain
koinos-jsonrpc-1 | 2022-11-03 14:34:20.863017 (jsonrpc.Koinos) [internal/jsonrpc.go:359] : Registered descriptor package: koinos.broadcast
koinos-jsonrpc-1 | 2022-11-03 14:34:20.867276 (jsonrpc.Koinos) [internal/jsonrpc.go:359] : Registered descriptor package: google.protobuf
koinos-jsonrpc-1 | 2022-11-03 14:34:20.867320 (jsonrpc.Koinos) [internal/jsonrpc.go:359] : Registered descriptor package: koinos.chain
koinos-jsonrpc-1 | 2022-11-03 14:34:20.867349 (jsonrpc.Koinos) [internal/jsonrpc.go:359] : Registered descriptor package: koinos.contracts.token
koinos-jsonrpc-1 | 2022-11-03 14:34:20.867375 (jsonrpc.Koinos) [internal/jsonrpc.go:359] : Registered descriptor package: koinos.rpc.chain
koinos-jsonrpc-1 | 2022-11-03 14:34:20.867402 (jsonrpc.Koinos) [internal/jsonrpc.go:359] : Registered descriptor package: koinos.block_store
koinos-jsonrpc-1 | 2022-11-03 14:34:20.867428 (jsonrpc.Koinos) [internal/jsonrpc.go:359] : Registered descriptor package: koinos.contracts.governance
koinos-jsonrpc-1 | 2022-11-03 14:34:20.867532 (jsonrpc.Koinos) [koinos-jsonrpc/main.go:280] : Listening on :8080/
koinos-block_store-1 | 2022-11-03 14:34:21.046157 (block_store.Koinos) [v3@v3.2103.2/logger.go:46] : All 497 tables opened in 114ms
koinos-block_store-1 | 2022-11-03 14:34:21.049882 (block_store.Koinos) [v3@v3.2103.2/logger.go:46] : Discard stats nextEmptySlot: 0
koinos-block_store-1 | 2022-11-03 14:34:21.050772 (block_store.Koinos) [v3@v3.2103.2/logger.go:46] : Set nextTxnTs to 6146807
koinos-block_store-1 | 2022-11-03 14:34:21.056843 (block_store.Koinos) [v3@v3.2103.2/logger.go:46] : Deleting empty file: /koinos/block_store/db/000132.vlog
koinos-block_store-1 | 2022-11-03 14:34:21.059407 (block_store.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/request_handler.go:100] : Connecting request handler to AMQP server amqp://guest:guest@amqp:5672/
koinos-block_store-1 | 2022-11-03 14:34:21.059472 (block_store.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/connection.go:85] : Dialing AMQP server amqp://guest:guest@amqp:5672/
koinos-block_store-1 | 2022-11-03 14:34:21.061500 (blockstore.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/connection.go:88] : AMQP error dialing server: dial tcp 172.19.0.3:5672: connect: connection refused
koinos-mempool-1 | 2022-11-03 14:34:21.495416 (mempool.Koinos) [retryer.cpp:73] : Failure during client connection to AMQP, retrying in 2000ms
koinos-chain-1 | 2022-11-03 14:34:21.543503 (chain.Koinos) [main.cpp:170] : Using fork resolution algorithm: block-time
koinos-chain-1 |
koinos-chain-1 |
koinos-chain-1 | | |/ / () __
koinos-chain-1 | | ' // | | ' \ / _ \/ |
koinos-chain-1 | | . \ () | | | | | () _ \
koinos-chain-1 | ||__/||| ||_/|/
koinos-chain-1 | ...launching network
koinos-chain-1 | 2022-11-03 14:34:21.713292 (chain.Koinos) [main.cpp:210] : Chain ID: 0x1220ccb9ea70dca318e95df16dd904836ff804a2a5db9a6fa45ce373f05bec0a7849
koinos-chain-1 | 2022-11-03 14:34:21.718671 (chain.Koinos) [main.cpp:211] : Number of jobs: 2
koinos-chain-1 | 2022-11-03 14:34:21.722084 (chain.Koinos) [controller.cpp:126] : Initialized fizzy VM backend
koinos-chain-1 | 2022-11-03 14:34:21.775560 (chain.Koinos) [controller.cpp:195] : Opened database at block - Height: 1998653, ID: 0x122080425d1419e96503d1e84da15d924827885eebbf53d555536fc47e120a7d906e
koinos-chain-1 | 2022-11-03 14:34:21.779006 (chain.Koinos) [main.cpp:254] : Connecting AMQP client...
koinos-chain-1 | 2022-11-03 14:34:21.784225 (chain.Koinos) [retryer.cpp:106] : Failure during client connection to AMQP, retrying in 1000ms
koinos-jsonrpc-1 | 2022-11-03 14:34:21.833009 (jsonrpc.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/connection.go:85] : Dialing AMQP server amqp://guest:guest@amqp:5672/
koinos-jsonrpc-1 | 2022-11-03 14:34:21.851319 (jsonrpc.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/connection.go:88] : AMQP error dialing server: dial tcp 172.19.0.3:5672: connect: connection refused
koinos-block_store-1 | 2022-11-03 14:34:22.062224 (block_store.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/connection.go:85] : Dialing AMQP server amqp://guest:guest@amqp:5672/
koinos-block_store-1 | 2022-11-03 14:34:22.073246 (block_store.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/connection.go:88] : AMQP error dialing server: dial tcp 172.19.0.3:5672: connect: connection refused
koinos-chain-1 | 2022-11-03 14:34:22.787401 (chain.Koinos) [retryer.cpp:73] : Failure during client connection to AMQP, retrying in 2000ms
koinos-contract_meta_store-1 | 2022-11-03 14:34:23.253567 (contract_meta_store.Koinos) [koinos-contract-meta-store/main.go:105] : Opening database at /koinos/contract_meta_store/db
koinos-contract_meta_store-1 | 2022-11-03 14:34:23.268685 (contract_meta_store.Koinos) [v3@v3.2103.2/logger.go:46] : All 1 tables opened in 0s
koinos-contract_meta_store-1 | 2022-11-03 14:34:23.271898 (contract_meta_store.Koinos) [v3@v3.2103.2/logger.go:46] : Discard stats nextEmptySlot: 0
koinos-contract_meta_store-1 | 2022-11-03 14:34:23.272372 (contract_meta_store.Koinos) [v3@v3.2103.2/logger.go:46] : Set nextTxnTs to 742
koinos-contract_meta_store-1 | 2022-11-03 14:34:23.272910 (contract_meta_store.Koinos) [v3@v3.2103.2/logger.go:46] : Deleting empty file: /koinos/contract_meta_store/db/000125.vlog
koinos-contract_meta_store-1 | 2022-11-03 14:34:23.273890 (contract_meta_store.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/request_handler.go:100] : Connecting request handler to AMQP server amqp://guest:guest@amqp:5672/
koinos-contract_meta_store-1 | 2022-11-03 14:34:23.274330 (contract_meta_store.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/connection.go:85] : Dialing AMQP server amqp://guest:guest@amqp:5672/
koinos-contract_meta_store-1 | 2022-11-03 14:34:23.276019 (contract_meta_store.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/connection.go:88] : AMQP error dialing server: dial tcp 172.19.0.3:5672: connect: connection refused
koinos-p2p-1 | 2022-11-03 14:34:23.409106 (p2p.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/client.go:117] : Connecting client to AMQP server amqp://guest:guest@amqp:5672/
koinos-p2p-1 | 2022-11-03 14:34:23.436144 (p2p.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/connection.go:85] : Dialing AMQP server amqp://guest:guest@amqp:5672/
koinos-p2p-1 | 2022-11-03 14:34:23.447552 (p2p.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/connection.go:88] : AMQP error dialing server: dial tcp 172.19.0.3:5672: connect: connection refused
koinos-mempool-1 | 2022-11-03 14:34:23.512744 (mempool.Koinos) [retryer.cpp:73] : Failure during client connection to AMQP, retrying in 4000ms
koinos-block_producer-1 | 2022-11-03 14:34:23.648769 (block_producer.Koinos) [main.cpp:195] : Public address: 1ByFWv2tFPB8dXktFWmRyhGL1VhU2Kwuwa
koinos-block_producer-1 | 2022-11-03 14:34:23.649509 (block_producer.Koinos) [main.cpp:196] : Public key: ArdeHj2gNUzqtuh8Rq5mA2i2JzlNo54TprR3SRsthVqN
koinos-block_producer-1 | 2022-11-03 14:34:23.649731 (block_producer.Koinos) [main.cpp:198] : Producer address: 15a6ZZR9M52Vt5ppW4iqoQEUfuNxrasgjF
koinos-block_producer-1 | 2022-11-03 14:34:23.653254 (block_producer.Koinos) [main.cpp:199] : Block resource utilization lower bound: 75%, upper bound: 90%
koinos-block_producer-1 | 2022-11-03 14:34:23.653681 (block_producer.Koinos) [main.cpp:200] : Maximum transaction inclusion attempts per block: 2000
koinos-block_producer-1 | 2022-11-03 14:34:23.654697 (block_producer.Koinos) [main.cpp:244] : Connecting AMQP client...
koinos-block_producer-1 | 2022-11-03 14:34:23.655712 (block_producer.Koinos) [retryer.cpp:106] : Failure during client connection to AMQP, retrying in 1000ms
koinos-transaction_store-1 | 2022-11-03 14:34:23.667450 (transaction_store.Koinos) [koinos-transaction-store/main.go:99] : Opening database at /koinos/transaction_store/db
koinos-transaction_store-1 | 2022-11-03 14:34:23.873577 (transaction_store.Koinos) [v3@v3.2103.2/logger.go:46] : All 507 tables opened in 150ms
koinos-transaction_store-1 | 2022-11-03 14:34:23.877067 (transaction_store.Koinos) [v3@v3.2103.2/logger.go:46] : Discard stats nextEmptySlot: 0
koinos-transaction_store-1 | 2022-11-03 14:34:23.877756 (transaction_store.Koinos) [v3@v3.2103.2/logger.go:46] : Set nextTxnTs to 24721138
koinos-transaction_store-1 | 2022-11-03 14:34:23.881803 (transaction_store.Koinos) [v3@v3.2103.2/logger.go:46] : Deleting empty file: /koinos/transaction_store/db/000125.vlog
koinos-transaction_store-1 | 2022-11-03 14:34:23.883511 (transaction_store.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/request_handler.go:100] : Connecting request handler to AMQP server amqp://guest:guest@amqp:5672/
koinos-transaction_store-1 | 2022-11-03 14:34:23.883972 (transaction_store.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/connection.go:85] : Dialing AMQP server amqp://guest:guest@amqp:5672/
koinos-transaction_store-1 | 2022-11-03 14:34:23.888223 (transaction_store.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/connection.go:88] : AMQP error dialing server: dial tcp 172.19.0.3:5672: connect: connection refused
koinos-contract_meta_store-1 | 2022-11-03 14:34:24.276394 (contract_meta_store.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/connection.go:85] : Dialing AMQP server amqp://guest:guest@amqp:5672/
koinos-contract_meta_store-1 | 2022-11-03 14:34:24.277832 (contract_meta_store.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/connection.go:88] : AMQP error dialing server: dial tcp 172.19.0.3:5672: connect: connection refused
koinos-p2p-1 | 2022-11-03 14:34:24.449432 (p2p.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/connection.go:85] : Dialing AMQP server amqp://guest:guest@amqp:5672/
koinos-p2p-1 | 2022-11-03 14:34:24.450747 (p2p.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/connection.go:88] : AMQP error dialing server: dial tcp 172.19.0.3:5672: connect: connection refused
koinos-block_producer-1 | 2022-11-03 14:34:24.656681 (block_producer.Koinos) [retryer.cpp:73] : Failure during client connection to AMQP, retrying in 2000ms
koinos-chain-1 | 2022-11-03 14:34:24.789233 (chain.Koinos) [retryer.cpp:73] : Failure during client connection to AMQP, retrying in 4000ms
koinos-jsonrpc-1 | 2022-11-03 14:34:24.852650 (jsonrpc.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/connection.go:85] : Dialing AMQP server amqp://guest:guest@amqp:5672/
koinos-jsonrpc-1 | 2022-11-03 14:34:24.853982 (jsonrpc.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/connection.go:88] : AMQP error dialing server: dial tcp 172.19.0.3:5672: connect: connection refused
koinos-transaction_store-1 | 2022-11-03 14:34:24.889162 (transaction_store.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/connection.go:85] : Dialing AMQP server amqp://guest:guest@amqp:5672/
koinos-transaction_store-1 | 2022-11-03 14:34:24.890306 (transaction_store.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/connection.go:88] : AMQP error dialing server: dial tcp 172.19.0.3:5672: connect: connection refused
koinos-block_store-1 | 2022-11-03 14:34:25.074158 (block_store.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/connection.go:85] : Dialing AMQP server amqp://guest:guest@amqp:5672/
koinos-block_store-1 | 2022-11-03 14:34:25.075451 (block_store.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/connection.go:88] : AMQP error dialing server: dial tcp 172.19.0.3:5672: connect: connection refused
koinos-amqp-1 | 2022-11-03 14:34:25.478285+00:00 [notice] <0.44.0> Application syslog exited with reason: stopped
koinos-amqp-1 | 2022-11-03 14:34:25.483577+00:00 [notice] <0.229.0> Logging: switching to configured handler(s); following messages may not be visible in this log output
koinos-amqp-1 | 2022-11-03 14:34:25.499080+00:00 [notice] <0.229.0> Logging: configured log handlers are now ACTIVE
koinos-block_producer-1 | 2022-11-03 14:34:26.658504 (block_producer.Koinos) [retryer.cpp:73] : Failure during client connection to AMQP, retrying in 4000ms
koinos-contract_meta_store-1 | 2022-11-03 14:34:27.278947 (contract_meta_store.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/connection.go:85] : Dialing AMQP server amqp://guest:guest@amqp:5672/
koinos-contract_meta_store-1 | 2022-11-03 14:34:27.279887 (contract_meta_store.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/connection.go:88] : AMQP error dialing server: dial tcp 172.19.0.3:5672: connect: connection refused
koinos-p2p-1 | 2022-11-03 14:34:27.452024 (p2p.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/connection.go:85] : Dialing AMQP server amqp://guest:guest@amqp:5672/
koinos-p2p-1 | 2022-11-03 14:34:27.453525 (p2p.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/connection.go:88] : AMQP error dialing server: dial tcp 172.19.0.3:5672: connect: connection refused
koinos-mempool-1 | 2022-11-03 14:34:27.514315 (mempool.Koinos) [retryer.cpp:73] : Failure during client connection to AMQP, retrying in 8000ms
koinos-transaction_store-1 | 2022-11-03 14:34:27.890442 (transaction_store.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/connection.go:85] : Dialing AMQP server amqp://guest:guest@amqp:5672/
koinos-transaction_store-1 | 2022-11-03 14:34:27.891622 (transaction_store.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/connection.go:88] : AMQP error dialing server: dial tcp 172.19.0.3:5672: connect: connection refused
koinos-amqp-1 | 2022-11-03 14:34:28.339480+00:00 [info] <0.229.0> ra: starting system quorum_queues
koinos-amqp-1 | 2022-11-03 14:34:28.339618+00:00 [info] <0.229.0> starting Ra system: quorum_queues in directory: /var/lib/rabbitmq/mnesia/rabbit@bb48e0941744/quorum/rabbit@bb48e0941744
koinos-amqp-1 | 2022-11-03 14:34:28.343160+00:00 [info] <0.297.0> ra system 'quorum_queues' running pre init for 0 registered servers
koinos-amqp-1 | 2022-11-03 14:34:28.345058+00:00 [info] <0.300.0> ra: meta data store initialised for system quorum_queues. 0 record(s) recovered
koinos-amqp-1 | 2022-11-03 14:34:28.345660+00:00 [notice] <0.308.0> WAL: ra_log_wal init, open tbls: ra_log_open_mem_tables, closed tbls: ra_log_closed_mem_tables
koinos-amqp-1 | 2022-11-03 14:34:28.352619+00:00 [info] <0.229.0> ra: starting system coordination
koinos-amqp-1 | 2022-11-03 14:34:28.352797+00:00 [info] <0.229.0> starting Ra system: coordination in directory: /var/lib/rabbitmq/mnesia/rabbit@bb48e0941744/coordination/rabbit@bb48e0941744
koinos-amqp-1 | 2022-11-03 14:34:28.356346+00:00 [info] <0.326.0> ra system 'coordination' running pre init for 0 registered servers
koinos-amqp-1 | 2022-11-03 14:34:28.358255+00:00 [info] <0.327.0> ra: meta data store initialised for system coordination. 0 record(s) recovered
koinos-amqp-1 | 2022-11-03 14:34:28.358675+00:00 [notice] <0.336.0> WAL: ra_coordination_log_wal init, open tbls: ra_coordination_log_open_mem_tables, closed tbls: ra_coordination_log_closed_mem_tables
koinos-amqp-1 | 2022-11-03 14:34:28.365588+00:00 [info] <0.229.0>
koinos-amqp-1 | 2022-11-03 14:34:28.365588+00:00 [info] <0.229.0> Starting RabbitMQ 3.11.2 on Erlang 25.1.2 [jit]
koinos-amqp-1 | 2022-11-03 14:34:28.365588+00:00 [info] <0.229.0> Copyright (c) 2007-2022 VMware, Inc. or its affiliates.
koinos-amqp-1 | 2022-11-03 14:34:28.365588+00:00 [info] <0.229.0> Licensed under the MPL 2.0. Website: https://rabbitmq.com
koinos-amqp-1 |
koinos-amqp-1 | ## ## RabbitMQ 3.11.2
koinos-amqp-1 | ## ##
koinos-amqp-1 | ########## Copyright (c) 2007-2022 VMware, Inc. or its affiliates.
koinos-amqp-1 | ###### ##
koinos-amqp-1 | ########## Licensed under the MPL 2.0. Website: https://rabbitmq.com
koinos-amqp-1 |
koinos-amqp-1 | Erlang: 25.1.2 [jit]
koinos-amqp-1 | TLS Library: OpenSSL - OpenSSL 1.1.1q 5 Jul 2022
koinos-amqp-1 | Release series support status: supported
koinos-amqp-1 |
koinos-amqp-1 | Doc guides: https://rabbitmq.com/documentation.html
koinos-amqp-1 | Support: https://rabbitmq.com/contact.html
koinos-amqp-1 | Tutorials: https://rabbitmq.com/getstarted.html
koinos-amqp-1 | Monitoring: https://rabbitmq.com/monitoring.html
koinos-amqp-1 |
koinos-amqp-1 | Logs: /var/log/rabbitmq/rabbit@bb48e0941744_upgrade.log
koinos-amqp-1 |
koinos-amqp-1 |
koinos-amqp-1 | Config file(s): /etc/rabbitmq/rabbitmq.conf
koinos-amqp-1 | /etc/rabbitmq/conf.d/10-defaults.conf
koinos-amqp-1 |
koinos-amqp-1 | Starting broker...2022-11-03 14:34:28.369996+00:00 [info] <0.229.0>
koinos-amqp-1 | 2022-11-03 14:34:28.369996+00:00 [info] <0.229.0> node : rabbit@bb48e0941744
koinos-amqp-1 | 2022-11-03 14:34:28.369996+00:00 [info] <0.229.0> home dir : /var/lib/rabbitmq
koinos-amqp-1 | 2022-11-03 14:34:28.369996+00:00 [info] <0.229.0> config file(s) : /etc/rabbitmq/rabbitmq.conf
koinos-amqp-1 | 2022-11-03 14:34:28.369996+00:00 [info] <0.229.0> : /etc/rabbitmq/conf.d/10-defaults.conf
koinos-amqp-1 | 2022-11-03 14:34:28.369996+00:00 [info] <0.229.0> cookie hash : tZM7Wcrd15OH6BUHMxBRkg==
koinos-amqp-1 | 2022-11-03 14:34:28.369996+00:00 [info] <0.229.0> log(s) : /var/log/rabbitmq/rabbit@bb48e0941744_upgrade.log
koinos-amqp-1 | 2022-11-03 14:34:28.369996+00:00 [info] <0.229.0> :
koinos-amqp-1 | 2022-11-03 14:34:28.369996+00:00 [info] <0.229.0> database dir : /var/lib/rabbitmq/mnesia/rabbit@bb48e0941744
koinos-chain-1 | 2022-11-03 14:34:28.790601 (chain.Koinos) [retryer.cpp:73] : Failure during client connection to AMQP, retrying in 8000ms
koinos-amqp-1 | 2022-11-03 14:34:28.982825+00:00 [info] <0.229.0> Running boot step pre_boot defined by app rabbit
koinos-amqp-1 | 2022-11-03 14:34:28.982917+00:00 [info] <0.229.0> Running boot step rabbit_global_counters defined by app rabbit
koinos-amqp-1 | 2022-11-03 14:34:28.984116+00:00 [info] <0.229.0> Running boot step rabbit_osiris_metrics defined by app rabbit
koinos-amqp-1 | 2022-11-03 14:34:28.984308+00:00 [info] <0.229.0> Running boot step rabbit_core_metrics defined by app rabbit
koinos-amqp-1 | 2022-11-03 14:34:28.985103+00:00 [info] <0.229.0> Running boot step rabbit_alarm defined by app rabbit
koinos-amqp-1 | 2022-11-03 14:34:28.990234+00:00 [info] <0.386.0> Memory high watermark set to 3984 MiB (4177851187 bytes) of 9960 MiB (10444627968 bytes) total
koinos-amqp-1 | 2022-11-03 14:34:28.994482+00:00 [info] <0.388.0> Enabling free disk space monitoring
koinos-amqp-1 | 2022-11-03 14:34:28.994562+00:00 [info] <0.388.0> Disk free limit set to 50MB
koinos-amqp-1 | 2022-11-03 14:34:28.997304+00:00 [info] <0.229.0> Running boot step code_server_cache defined by app rabbit
koinos-amqp-1 | 2022-11-03 14:34:28.997478+00:00 [info] <0.229.0> Running boot step file_handle_cache defined by app rabbit
koinos-amqp-1 | 2022-11-03 14:34:28.998035+00:00 [info] <0.391.0> Limiting to approx 1048479 file handles (943629 sockets)
koinos-amqp-1 | 2022-11-03 14:34:28.998284+00:00 [info] <0.392.0> FHC read buffering: OFF
koinos-amqp-1 | 2022-11-03 14:34:28.998546+00:00 [info] <0.392.0> FHC write buffering: ON
koinos-amqp-1 | 2022-11-03 14:34:28.999693+00:00 [info] <0.229.0> Running boot step worker_pool defined by app rabbit
koinos-amqp-1 | 2022-11-03 14:34:28.999813+00:00 [info] <0.350.0> Will use 2 processes for default worker pool
koinos-amqp-1 | 2022-11-03 14:34:28.999923+00:00 [info] <0.350.0> Starting worker pool 'worker_pool' with 2 processes in it
koinos-amqp-1 | 2022-11-03 14:34:29.001038+00:00 [info] <0.229.0> Running boot step database defined by app rabbit
koinos-amqp-1 | 2022-11-03 14:34:29.003676+00:00 [info] <0.229.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
koinos-amqp-1 | 2022-11-03 14:34:29.006754+00:00 [info] <0.229.0> Successfully synced tables from a peer
koinos-amqp-1 | 2022-11-03 14:34:29.022126+00:00 [info] <0.229.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
koinos-amqp-1 | 2022-11-03 14:34:29.022666+00:00 [info] <0.229.0> Successfully synced tables from a peer
koinos-amqp-1 | 2022-11-03 14:34:29.022766+00:00 [info] <0.229.0> Peer discovery backend rabbit_peer_discovery_classic_config does not support registration, skipping registration.
koinos-amqp-1 | 2022-11-03 14:34:29.022922+00:00 [info] <0.229.0> Running boot step tracking_metadata_store defined by app rabbit
koinos-amqp-1 | 2022-11-03 14:34:29.023195+00:00 [info] <0.407.0> Setting up a table for connection tracking on this node: tracked_connection
koinos-amqp-1 | 2022-11-03 14:34:29.023351+00:00 [info] <0.407.0> Setting up a table for per-vhost connection counting on this node: tracked_connection_per_vhost
koinos-amqp-1 | 2022-11-03 14:34:29.023473+00:00 [info] <0.407.0> Setting up a table for per-user connection counting on this node: tracked_connection_per_user
koinos-amqp-1 | 2022-11-03 14:34:29.023594+00:00 [info] <0.407.0> Setting up a table for channel tracking on this node: tracked_channel
koinos-amqp-1 | 2022-11-03 14:34:29.023681+00:00 [info] <0.407.0> Setting up a table for channel tracking on this node: tracked_channel_per_user
koinos-amqp-1 | 2022-11-03 14:34:29.024584+00:00 [info] <0.229.0> Running boot step networking_metadata_store defined by app rabbit
koinos-amqp-1 | 2022-11-03 14:34:29.025092+00:00 [info] <0.229.0> Running boot step database_sync defined by app rabbit
koinos-amqp-1 | 2022-11-03 14:34:29.025668+00:00 [info] <0.229.0> Running boot step feature_flags defined by app rabbit
koinos-amqp-1 | 2022-11-03 14:34:29.026192+00:00 [info] <0.229.0> Running boot step codec_correctness_check defined by app rabbit
koinos-amqp-1 | 2022-11-03 14:34:29.026268+00:00 [info] <0.229.0> Running boot step external_infrastructure defined by app rabbit
koinos-amqp-1 | 2022-11-03 14:34:29.026399+00:00 [info] <0.229.0> Running boot step rabbit_event defined by app rabbit
koinos-amqp-1 | 2022-11-03 14:34:29.026798+00:00 [info] <0.229.0> Running boot step rabbit_registry defined by app rabbit
koinos-amqp-1 | 2022-11-03 14:34:29.027500+00:00 [info] <0.229.0> Running boot step rabbit_auth_mechanism_amqplain defined by app rabbit
koinos-amqp-1 | 2022-11-03 14:34:29.027835+00:00 [info] <0.229.0> Running boot step rabbit_auth_mechanism_cr_demo defined by app rabbit
koinos-amqp-1 | 2022-11-03 14:34:29.028657+00:00 [info] <0.229.0> Running boot step rabbit_auth_mechanism_plain defined by app rabbit
koinos-amqp-1 | 2022-11-03 14:34:29.028993+00:00 [info] <0.229.0> Running boot step rabbit_exchange_type_direct defined by app rabbit
koinos-amqp-1 | 2022-11-03 14:34:29.029266+00:00 [info] <0.229.0> Running boot step rabbit_exchange_type_fanout defined by app rabbit
koinos-amqp-1 | 2022-11-03 14:34:29.029627+00:00 [info] <0.229.0> Running boot step rabbit_exchange_type_headers defined by app rabbit
koinos-amqp-1 | 2022-11-03 14:34:29.030126+00:00 [info] <0.229.0> Running boot step rabbit_exchange_type_topic defined by app rabbit
koinos-amqp-1 | 2022-11-03 14:34:29.030503+00:00 [info] <0.229.0> Running boot step rabbit_mirror_queue_mode_all defined by app rabbit
koinos-amqp-1 | 2022-11-03 14:34:29.030883+00:00 [info] <0.229.0> Running boot step rabbit_mirror_queue_mode_exactly defined by app rabbit
koinos-amqp-1 | 2022-11-03 14:34:29.031165+00:00 [info] <0.229.0> Running boot step rabbit_mirror_queue_mode_nodes defined by app rabbit
koinos-amqp-1 | 2022-11-03 14:34:29.031908+00:00 [info] <0.229.0> Running boot step rabbit_priority_queue defined by app rabbit
koinos-amqp-1 | 2022-11-03 14:34:29.032005+00:00 [info] <0.229.0> Priority queues enabled, real BQ is rabbit_variable_queue
koinos-amqp-1 | 2022-11-03 14:34:29.032239+00:00 [info] <0.229.0> Running boot step rabbit_queue_location_client_local defined by app rabbit
koinos-amqp-1 | 2022-11-03 14:34:29.032809+00:00 [info] <0.229.0> Running boot step rabbit_queue_location_min_masters defined by app rabbit
koinos-amqp-1 | 2022-11-03 14:34:29.033088+00:00 [info] <0.229.0> Running boot step rabbit_queue_location_random defined by app rabbit
koinos-amqp-1 | 2022-11-03 14:34:29.033353+00:00 [info] <0.229.0> Running boot step kernel_ready defined by app rabbit
koinos-amqp-1 | 2022-11-03 14:34:29.033537+00:00 [info] <0.229.0> Running boot step rabbit_sysmon_minder defined by app rabbit
koinos-amqp-1 | 2022-11-03 14:34:29.034085+00:00 [info] <0.229.0> Running boot step rabbit_epmd_monitor defined by app rabbit
koinos-amqp-1 | 2022-11-03 14:34:29.035696+00:00 [info] <0.416.0> epmd monitor knows us, inter-node communication (distribution) port: 25672
koinos-amqp-1 | 2022-11-03 14:34:29.035850+00:00 [info] <0.229.0> Running boot step guid_generator defined by app rabbit
koinos-amqp-1 | 2022-11-03 14:34:29.043376+00:00 [info] <0.229.0> Running boot step rabbit_node_monitor defined by app rabbit
koinos-amqp-1 | 2022-11-03 14:34:29.043999+00:00 [info] <0.420.0> Starting rabbit_node_monitor
koinos-amqp-1 | 2022-11-03 14:34:29.044470+00:00 [info] <0.229.0> Running boot step delegate_sup defined by app rabbit
koinos-amqp-1 | 2022-11-03 14:34:29.045404+00:00 [info] <0.229.0> Running boot step rabbit_memory_monitor defined by app rabbit
koinos-amqp-1 | 2022-11-03 14:34:29.045853+00:00 [info] <0.229.0> Running boot step rabbit_fifo_dlx_sup defined by app rabbit
koinos-amqp-1 | 2022-11-03 14:34:29.046011+00:00 [info] <0.229.0> Running boot step core_initialized defined by app rabbit
koinos-amqp-1 | 2022-11-03 14:34:29.046293+00:00 [info] <0.229.0> Running boot step upgrade_queues defined by app rabbit
koinos-amqp-1 | 2022-11-03 14:34:29.056866+00:00 [info] <0.229.0> Running boot step channel_tracking defined by app rabbit
koinos-amqp-1 | 2022-11-03 14:34:29.056977+00:00 [info] <0.229.0> Running boot step rabbit_channel_tracking_handler defined by app rabbit
koinos-amqp-1 | 2022-11-03 14:34:29.057069+00:00 [info] <0.229.0> Running boot step connection_tracking defined by app rabbit
koinos-amqp-1 | 2022-11-03 14:34:29.057378+00:00 [info] <0.229.0> Running boot step rabbit_connection_tracking_handler defined by app rabbit
koinos-amqp-1 | 2022-11-03 14:34:29.057619+00:00 [info] <0.229.0> Running boot step rabbit_definitions_hashing defined by app rabbit
koinos-amqp-1 | 2022-11-03 14:34:29.057759+00:00 [info] <0.229.0> Running boot step rabbit_exchange_parameters defined by app rabbit
koinos-amqp-1 | 2022-11-03 14:34:29.058262+00:00 [info] <0.229.0> Running boot step rabbit_mirror_queue_misc defined by app rabbit
koinos-amqp-1 | 2022-11-03 14:34:29.059180+00:00 [info] <0.229.0> Running boot step rabbit_policies defined by app rabbit
koinos-amqp-1 | 2022-11-03 14:34:29.059910+00:00 [info] <0.229.0> Running boot step rabbit_policy defined by app rabbit
koinos-amqp-1 | 2022-11-03 14:34:29.060093+00:00 [info] <0.229.0> Running boot step rabbit_queue_location_validator defined by app rabbit
koinos-amqp-1 | 2022-11-03 14:34:29.060508+00:00 [info] <0.229.0> Running boot step rabbit_quorum_memory_manager defined by app rabbit
koinos-amqp-1 | 2022-11-03 14:34:29.060843+00:00 [info] <0.229.0> Running boot step rabbit_stream_coordinator defined by app rabbit
koinos-amqp-1 | 2022-11-03 14:34:29.061716+00:00 [info] <0.229.0> Running boot step rabbit_vhost_limit defined by app rabbit
koinos-amqp-1 | 2022-11-03 14:34:29.062173+00:00 [info] <0.229.0> Running boot step rabbit_mgmt_reset_handler defined by app rabbitmq_management
koinos-amqp-1 | 2022-11-03 14:34:29.062303+00:00 [info] <0.229.0> Running boot step rabbit_mgmt_db_handler defined by app rabbitmq_management_agent
koinos-amqp-1 | 2022-11-03 14:34:29.062625+00:00 [info] <0.229.0> Management plugin: using rates mode 'basic'
koinos-amqp-1 | 2022-11-03 14:34:29.063688+00:00 [info] <0.229.0> Running boot step recovery defined by app rabbit
koinos-amqp-1 | 2022-11-03 14:34:29.065551+00:00 [info] <0.454.0> Making sure data directory '/var/lib/rabbitmq/mnesia/rabbit@bb48e0941744/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L' for vhost '/' exists
koinos-amqp-1 | 2022-11-03 14:34:29.071144+00:00 [info] <0.454.0> Starting message stores for vhost '/'
koinos-amqp-1 | 2022-11-03 14:34:29.071861+00:00 [info] <0.459.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_transient": using rabbit_msg_store_ets_index to provide index
koinos-amqp-1 | 2022-11-03 14:34:29.075092+00:00 [info] <0.454.0> Started message store of type transient for vhost '/'
koinos-amqp-1 | 2022-11-03 14:34:29.075705+00:00 [info] <0.463.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": using rabbit_msg_store_ets_index to provide index
koinos-amqp-1 | 2022-11-03 14:34:29.079201+00:00 [info] <0.454.0> Started message store of type persistent for vhost '/'
koinos-amqp-1 | 2022-11-03 14:34:29.089301+00:00 [info] <0.454.0> Recovering 4 queues of type rabbit_classic_queue took 22ms
koinos-amqp-1 | 2022-11-03 14:34:29.089442+00:00 [info] <0.454.0> Recovering 0 queues of type rabbit_quorum_queue took 0ms
koinos-amqp-1 | 2022-11-03 14:34:29.089506+00:00 [info] <0.454.0> Recovering 0 queues of type rabbit_stream_queue took 0ms
koinos-amqp-1 | 2022-11-03 14:34:29.094322+00:00 [info] <0.229.0> Running boot step empty_db_check defined by app rabbit
koinos-amqp-1 | 2022-11-03 14:34:29.094593+00:00 [info] <0.229.0> Will not seed default virtual host and user: have definitions to load...
koinos-amqp-1 | 2022-11-03 14:34:29.094768+00:00 [info] <0.229.0> Running boot step rabbit_observer_cli defined by app rabbit
koinos-amqp-1 | 2022-11-03 14:34:29.095773+00:00 [info] <0.229.0> Running boot step rabbit_looking_glass defined by app rabbit
koinos-amqp-1 | 2022-11-03 14:34:29.095867+00:00 [info] <0.229.0> Running boot step rabbit_core_metrics_gc defined by app rabbit
koinos-amqp-1 | 2022-11-03 14:34:29.096775+00:00 [info] <0.229.0> Running boot step background_gc defined by app rabbit
koinos-amqp-1 | 2022-11-03 14:34:29.097308+00:00 [info] <0.229.0> Running boot step routing_ready defined by app rabbit
koinos-amqp-1 | 2022-11-03 14:34:29.097394+00:00 [info] <0.229.0> Running boot step pre_flight defined by app rabbit
koinos-amqp-1 | 2022-11-03 14:34:29.097565+00:00 [info] <0.229.0> Running boot step notify_cluster defined by app rabbit
koinos-amqp-1 | 2022-11-03 14:34:29.097789+00:00 [info] <0.229.0> Running boot step networking defined by app rabbit
koinos-amqp-1 | 2022-11-03 14:34:29.097917+00:00 [info] <0.229.0> Running boot step definition_import_worker_pool defined by app rabbit
koinos-amqp-1 | 2022-11-03 14:34:29.098449+00:00 [info] <0.350.0> Starting worker pool 'definition_import_pool' with 2 processes in it
koinos-amqp-1 | 2022-11-03 14:34:29.099807+00:00 [info] <0.229.0> Running boot step cluster_name defined by app rabbit
koinos-amqp-1 | 2022-11-03 14:34:29.100218+00:00 [info] <0.229.0> Running boot step direct_client defined by app rabbit
koinos-amqp-1 | 2022-11-03 14:34:29.100836+00:00 [info] <0.229.0> Running boot step rabbit_maintenance_mode_state defined by app rabbit
koinos-amqp-1 | 2022-11-03 14:34:29.101223+00:00 [info] <0.229.0> Creating table rabbit_node_maintenance_states for maintenance mode status
koinos-amqp-1 | 2022-11-03 14:34:29.102334+00:00 [info] <0.229.0> Running boot step rabbit_management_load_definitions defined by app rabbitmq_management
koinos-amqp-1 | 2022-11-03 14:34:29.102715+00:00 [info] <0.512.0> Resetting node maintenance status
koinos-amqp-1 | 2022-11-03 14:34:29.117189+00:00 [info] <0.571.0> Management plugin: HTTP (non-TLS) listener started on port 15672
koinos-amqp-1 | 2022-11-03 14:34:29.117450+00:00 [info] <0.599.0> Statistics database started.
koinos-amqp-1 | 2022-11-03 14:34:29.117943+00:00 [info] <0.598.0> Starting worker pool 'management_worker_pool' with 3 processes in it
koinos-amqp-1 | 2022-11-03 14:34:29.129547+00:00 [info] <0.613.0> Prometheus metrics: HTTP (non-TLS) listener started on port 15692
koinos-amqp-1 | 2022-11-03 14:34:29.129725+00:00 [info] <0.512.0> Ready to start client connection listeners
koinos-amqp-1 | 2022-11-03 14:34:29.132064+00:00 [info] <0.657.0> started TCP listener on [::]:5672
koinos-amqp-1 | completed with 4 plugins.
koinos-amqp-1 | 2022-11-03 14:34:29.256186+00:00 [info] <0.512.0> Server startup complete; 4 plugins started.
koinos-amqp-1 | 2022-11-03 14:34:29.256186+00:00 [info] <0.512.0> rabbitmq_prometheus
koinos-amqp-1 | 2022-11-03 14:34:29.256186+00:00 [info] <0.512.0> rabbitmq_management
koinos-amqp-1 | 2022-11-03 14:34:29.256186+00:00 [info] <0.512.0> rabbitmq_web_dispatch
koinos-amqp-1 | 2022-11-03 14:34:29.256186+00:00 [info] <0.512.0> rabbitmq_management_agent
koinos-jsonrpc-1 | 2022-11-03 14:34:29.856671 (jsonrpc.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/connection.go:85] : Dialing AMQP server amqp://guest:guest@amqp:5672/
koinos-amqp-1 | 2022-11-03 14:34:29.858761+00:00 [info] <0.660.0> accepting AMQP connection <0.660.0> (172.19.0.4:49964 -> 172.19.0.3:5672)
koinos-amqp-1 | 2022-11-03 14:34:29.862618+00:00 [info] <0.660.0> connection <0.660.0> (172.19.0.4:49964 -> 172.19.0.3:5672): user 'guest' authenticated and granted access to vhost '/'
koinos-jsonrpc-1 | 2022-11-03 14:34:29.870522 (jsonrpc.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/client.go:136] : Client connected
koinos-block_store-1 | 2022-11-03 14:34:30.076254 (block_store.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/connection.go:85] : Dialing AMQP server amqp://guest:guest@amqp:5672/
koinos-amqp-1 | 2022-11-03 14:34:30.078415+00:00 [info] <0.679.0> accepting AMQP connection <0.679.0> (172.19.0.6:53072 -> 172.19.0.3:5672)
koinos-amqp-1 | 2022-11-03 14:34:30.081356+00:00 [info] <0.679.0> connection <0.679.0> (172.19.0.6:53072 -> 172.19.0.3:5672): user 'guest' authenticated and granted access to vhost '/'
koinos-block_store-1 | 2022-11-03 14:34:30.088267 (block_store.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/request_handler.go:132] : Request handler connected
koinos-amqp-1 | 2022-11-03 14:34:30.662169+00:00 [info] <0.699.0> accepting AMQP connection <0.699.0> (172.19.0.10:56962 -> 172.19.0.3:5672)
koinos-amqp-1 | 2022-11-03 14:34:30.664831+00:00 [info] <0.699.0> connection <0.699.0> (172.19.0.10:56962 -> 172.19.0.3:5672): user 'guest' authenticated and granted access to vhost '/'
koinos-amqp-1 | 2022-11-03 14:34:30.666689+00:00 [info] <0.710.0> accepting AMQP connection <0.710.0> (172.19.0.10:56972 -> 172.19.0.3:5672)
koinos-amqp-1 | 2022-11-03 14:34:30.668855+00:00 [info] <0.710.0> connection <0.710.0> (172.19.0.10:56972 -> 172.19.0.3:5672): user 'guest' authenticated and granted access to vhost '/'
koinos-block_producer-1 | 2022-11-03 14:34:30.673598 (block_producer.Koinos) [main.cpp:246] : Established AMQP client connection to the server
koinos-block_producer-1 | 2022-11-03 14:34:30.673804 (block_producer.Koinos) [main.cpp:248] : Attempting to connect to chain...
koinos-block_producer-1 | 2022-11-03 14:34:31.675241 (block_producer.Koinos) [client.cpp:437] : No response to client request with correlation ID: EpZ57SuOZLwoJ0XvL30KSd0eShs7PTPJ, within 1000ms
koinos-contract_meta_store-1 | 2022-11-03 14:34:32.279994 (contract_meta_store.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/connection.go:85] : Dialing AMQP server amqp://guest:guest@amqp:5672/
koinos-amqp-1 | 2022-11-03 14:34:32.281518+00:00 [info] <0.727.0> accepting AMQP connection <0.727.0> (172.19.0.8:34906 -> 172.19.0.3:5672)
koinos-amqp-1 | 2022-11-03 14:34:32.283851+00:00 [info] <0.727.0> connection <0.727.0> (172.19.0.8:34906 -> 172.19.0.3:5672): user 'guest' authenticated and granted access to vhost '/'
koinos-contract_meta_store-1 | 2022-11-03 14:34:32.290715 (contract_meta_store.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/request_handler.go:132] : Request handler connected
koinos-p2p-1 | 2022-11-03 14:34:32.457717 (p2p.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/connection.go:85] : Dialing AMQP server amqp://guest:guest@amqp:5672/
koinos-amqp-1 | 2022-11-03 14:34:32.459403+00:00 [info] <0.747.0> accepting AMQP connection <0.747.0> (172.19.0.7:40238 -> 172.19.0.3:5672)
koinos-amqp-1 | 2022-11-03 14:34:32.461876+00:00 [info] <0.747.0> connection <0.747.0> (172.19.0.7:40238 -> 172.19.0.3:5672): user 'guest' authenticated and granted access to vhost '/'
koinos-p2p-1 | 2022-11-03 14:34:32.467202 (p2p.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/client.go:136] : Client connected
koinos-p2p-1 | 2022-11-03 14:34:32.467296 (p2p.Koinos) [koinos-p2p/main.go:161] : Attempting to connect to block_store...
koinos-p2p-1 | 2022-11-03 14:34:32.469957 (p2p.Koinos) [koinos-p2p/main.go:167] : Connected
koinos-p2p-1 | 2022-11-03 14:34:32.470029 (p2p.Koinos) [koinos-p2p/main.go:172] : Attempting to connect to chain...
koinos-transaction_store-1 | 2022-11-03 14:34:32.891996 (transaction_store.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/connection.go:85] : Dialing AMQP server amqp://guest:guest@amqp:5672/
koinos-amqp-1 | 2022-11-03 14:34:32.894004+00:00 [info] <0.766.0> accepting AMQP connection <0.766.0> (172.19.0.9:56070 -> 172.19.0.3:5672)
koinos-amqp-1 | 2022-11-03 14:34:32.896501+00:00 [info] <0.766.0> connection <0.766.0> (172.19.0.9:56070 -> 172.19.0.3:5672): user 'guest' authenticated and granted access to vhost '/'
koinos-transaction_store-1 | 2022-11-03 14:34:32.903669 (transaction_store.Koinos) [koinos-mq-golang@v0.0.0-20221027220937-6ac042578e2f/request_handler.go:132] : Request handler connected
koinos-block_producer-1 | 2022-11-03 14:34:33.676624 (block_producer.Koinos) [client.cpp:437] : No response to client request with correlation ID: XdmbccQ6b4dLv6IdeaHdGZQE94gvuFI9, within 2000ms
koinos-amqp-1 | 2022-11-03 14:34:35.521737+00:00 [info] <0.786.0> accepting AMQP connection <0.786.0> (172.19.0.5:49772 -> 172.19.0.3:5672)
koinos-amqp-1 | 2022-11-03 14:34:35.524333+00:00 [info] <0.786.0> connection <0.786.0> (172.19.0.5:49772 -> 172.19.0.3:5672): user 'guest' authenticated and granted access to vhost '/'
koinos-amqp-1 | 2022-11-03 14:34:35.526345+00:00 [info] <0.797.0> accepting AMQP connection <0.797.0> (172.19.0.5:49774 -> 172.19.0.3:5672)
koinos-amqp-1 | 2022-11-03 14:34:35.528429+00:00 [info] <0.797.0> connection <0.797.0> (172.19.0.5:49774 -> 172.19.0.3:5672): user 'guest' authenticated and granted access to vhost '/'
koinos-mempool-1 | 2022-11-03 14:34:35.532970 (mempool.Koinos) [main.cpp:326] : Established AMQP client connection to the server
koinos-mempool-1 | 2022-11-03 14:34:35.533136 (mempool.Koinos) [main.cpp:328] : Connecting AMQP request handler...
koinos-amqp-1 | 2022-11-03 14:34:35.534274+00:00 [info] <0.814.0> accepting AMQP connection <0.814.0> (172.19.0.5:49778 -> 172.19.0.3:5672)
koinos-amqp-1 | 2022-11-03 14:34:35.536355+00:00 [info] <0.814.0> connection <0.814.0> (172.19.0.5:49778 -> 172.19.0.3:5672): user 'guest' authenticated and granted access to vhost '/'
koinos-amqp-1 | 2022-11-03 14:34:35.538111+00:00 [info] <0.825.0> accepting AMQP connection <0.825.0> (172.19.0.5:49782 -> 172.19.0.3:5672)
koinos-amqp-1 | 2022-11-03 14:34:35.540201+00:00 [info] <0.825.0> connection <0.825.0> (172.19.0.5:49782 -> 172.19.0.3:5672): user 'guest' authenticated and granted access to vhost '/'
koinos-mempool-1 | 2022-11-03 14:34:35.555219 (mempool.Koinos) [main.cpp:330] : Established request handler connection to the AMQP server
koinos-mempool-1 | 2022-11-03 14:34:35.555384 (mempool.Koinos) [main.cpp:332] : Listening for requests over AMQP
koinos-amqp-1 | 2022-11-03 14:34:36.799099+00:00 [info] <0.855.0> accepting AMQP connection <0.855.0> (172.19.0.2:56204 -> 172.19.0.3:5672)
koinos-amqp-1 | 2022-11-03 14:34:36.801761+00:00 [info] <0.855.0> connection <0.855.0> (172.19.0.2:56204 -> 172.19.0.3:5672): user 'guest' authenticated and granted access to vhost '/'
koinos-amqp-1 | 2022-11-03 14:34:36.803743+00:00 [info] <0.866.0> accepting AMQP connection <0.866.0> (172.19.0.2:56216 -> 172.19.0.3:5672)
koinos-amqp-1 | 2022-11-03 14:34:36.805935+00:00 [info] <0.866.0> connection <0.866.0> (172.19.0.2:56216 -> 172.19.0.3:5672): user 'guest' authenticated and granted access to vhost '/'
koinos-chain-1 | 2022-11-03 14:34:36.811006 (chain.Koinos) [main.cpp:256] : Established AMQP client connection to the server
koinos-chain-1 | 2022-11-03 14:34:36.811191 (chain.Koinos) [main.cpp:258] : Attempting to connect to block_store...
koinos-chain-1 | 2022-11-03 14:34:36.812936 (chain.Koinos) [main.cpp:262] : Established connection to block_store
koinos-chain-1 | 2022-11-03 14:34:36.813088 (chain.Koinos) [main.cpp:264] : Attempting to connect to mempool...
koinos-chain-1 | 2022-11-03 14:34:36.814523 (chain.Koinos) [main.cpp:268] : Established connection to mempool
koinos-chain-1 | 2022-11-03 14:34:36.814897 (chain.Koinos) [indexer.cpp:66] : Retrieving highest block from block store
koinos-chain-1 | 2022-11-03 14:34:36.817234 (chain.Koinos) [indexer.cpp:91] : Indexing to target block - Height: 1998713, ID: 0x1220b9473bc4603f03bd32addee436f71c99aeadd1b5791e62a0558e601523817a41
koinos-chain-1 | 2022-11-03 14:34:37.610352 (chain.Koinos) [controller.cpp:472] : Block application failed - Height: 1998654 ID: 0x122094f9e9a1b7ddd4c169c4b79859c1fc3429f60e87dec06d18366a2321f85f2670, with reason: invalid transaction nonce - account: 1KwM9wFJ9u9qBgeAEs5v32GyB1EQ2bZQBB, nonce: 0x28f4b1b801, current nonce: 0x28f6b0b801
koinos-chain-1 | 2022-11-03 14:34:37.632631 (chain.Koinos) [main.cpp:239] : Caught signal, shutting down...
koinos-block_producer-1 | 2022-11-03 14:34:37.677799 (block_producer.Koinos) [client.cpp:437] : No response to client request with correlation ID: 6M4V8QqpnwqwtMxrjFzixtjdXiqWq4h0, within 4000ms
koinos-block_producer-1 | 2022-11-03 14:34:45.679309 (block_producer.Koinos) [client.cpp:437] : No response to client request with correlation ID: IW2yannkGeABim5t6gJm3QDWs0I4bo0Q, within 8000ms
koinos-block_producer-1 | 2022-11-03 14:35:01.679987 (block_producer.Koinos) [client.cpp:437] : No response to client request with correlation ID: KBvyDEpfvbLfPwrcCWxwNC8s6cDqEYH7, within 16000ms
koinos-block_producer-1 | 2022-11-03 14:35:31.680683 (block_producer.Koinos) [client.cpp:437] : No response to client request with correlation ID: QgWq5qwwpDSGKyEMz8yA4PMNHefyQjT4, within 30000ms
koinos-block_producer-1 | 2022-11-03 14:36:01.681645 (block_producer.Koinos) [client.cpp:437] : No response to client request with correlation ID: UGP0b4kJXninbpEJWSFI9ai8CwQlOvC8, within 30000ms
koinos-block_producer-1 | 2022-11-03 14:36:31.682490 (block_producer.Koinos) [client.cpp:437] : No response to client request with correlation ID: UJxXVJqFr64twEls0quookUVohNvLLtI, within 30000ms
koinos-block_producer-1 | 2022-11-03 14:37:01.683907 (block_producer.Koinos) [client.cpp:437] : No response to client request with correlation ID: AVseFkNg5yBhMu6Zzv3Z0C0HHhvgy4vr, within 30000ms
koinos-block_producer-1 | 2022-11-03 14:37:31.684976 (block_producer.Koinos) [client.cpp:437] : No response to client request with correlation ID: gsMmKkG0q8YaxuzJsOOjk57vWlUUsZ3H, within 30000ms
koinos-block_producer-1 | 2022-11-03 14:38:01.686483 (block_producer.Koinos) [client.cpp:437] : No response to client request with correlation ID: nC1UGfNRZfkTcaQ2PUhx0LvdzQbBqdRE, within 30000ms
koinos-block_producer-1 | 2022-11-03 14:38:31.687192 (block_producer.Koinos) [client.cpp:437] : No response to client request with correlation ID: DwDE1cbxhwx0Kin7bxK9RyNmIq0bdPAV, within 30000ms
koinos-block_producer-1 | 2022-11-03 14:39:01.688276 (block_producer.Koinos) [client.cpp:437] : No response to client request with correlation ID: tE1tZES3wXNQLx4MLnXZ21SwKEY0zTyG, within 30000ms
koinos-block_producer-1 | 2022-11-03 14:39:31.689344 (block_producer.Koinos) [client.cpp:437] : No response to client request with correlation ID: VZUMJOFRkMuGEdsnmUd0SzQba1uFqVCX, within 30000ms
koinos-block_producer-1 | 2022-11-03 14:40:01.689801 (block_producer.Koinos) [client.cpp:437] : No response to client request with correlation ID: 9WA0U9b86zQYmCxmeTRyPsxtdJJLGSUN, within 30000ms
Expected behavior
No response
Steps to reproduce
No response
Environment
Anything else?
No response