search

helium / miner

Miner for the helium blockchain
Apache License 2.0
609 stars 266 forks source link

validator implodes upon restoring snapshot #968

Closed densone closed 3 years ago

densone commented 3 years ago

Notes

I have tried on several different machines. All manifested after 1.1.4. I've tried without the use of sync_pause. I have tried on FreeBSD 11 and 13. 

## System 
- Intel processor 
- FreeBSD 13
- 16GB RAM
- NVME

## Commands run on snapshot host
```bash
#snapshot memory needed to be increased for snapshot to succeed
> miner snapshot take /chain
> scp /chain root@new_host:/

commands run on new host

> rm -rf /usr/local/miner/data
> miner genesis load /genesis.mainnet
> miner repair. sync_pause
> miner snapshot load /chain

Error On Screen

root@hot_dinosaur:~ # miner repair sync_pause
ok
root@hot_dinosaur:~ # miner snapshot load /chain
Logger - error: {removed_failing_handler,default}
"escript: exception error: terminated\n  in function  io:format/3\n     called as io:format(<0.69.0>,\"RPC to ~p failed: ~p~n\",\n                         ['val_miner@127.0.0.1',\n                          {'EXIT',\n                           {{{badmatch,\n                              {error,{bad_nonce,{payment,8042,8042}}}},\n                             [{blockchain_ledger_snapshot_v1,\n                               '-load_blocks/3-fun-0-',4,\n                               [{file,\n                                 \"/root/miner/_build/default/lib/blockchain/src/ledger/v1/blockchain_ledger_snapshot_v1.erl\"},\n                                {line,542}]},\n                              {lists,foreach,2,\n                               [{file,\"lists.erl\"},{line,1342}]},\n                              {blockchain_ledger_snapshot_v1,import,3,\n                               [{file,\n                                 \"/root/miner/_build/default/lib/blockchain/src/ledger/v1/blockchain_ledger_snapshot_v1.erl\"},\n                                {line,411}]},\n                              {blockchain_worker,handle_call,3,\n                               [{file,\n                                 \"/root/miner/_build/default/lib/blockchain/src/blockchain_worker.erl\"},\n                                {line,387}]},\n                              {gen_server,try_handle_call,4,\n                               [{file,\"gen_server.erl\"},{line,715}]},\n                              {gen_server,handle_msg,6,\n                               [{file,\"gen_server.erl\"},{line,744}]},\n                              {proc_lib,wake_up,3,\n                               [{file,\"proc_lib.erl\"},{line,236}]}]},\n                            {gen_server,call,\n                             [blockchain_worker,\n                              {install_snapshot,\n                               <<172,161,43,230,199,166,58,186,186,196,88,147,\n                                 133,1,186,252,53,1,31,154,97,254,229,97,87,\n                                 152,86,104,134,221,53,120>>,\n                               #{accounts =>\n                                  <<33,0,0,0,0,0,56,11,97,223,171,33,53,81,211,\n                                    31,234,34,160,150,24,247,187,145,231,83,33,\n                                    109,235,43,142,...>>,\n                                 blocks =>\n                                  <<131,108,0,0,1,96,109,0,3,146,15,10,139,164,\n                                    14,10,32,239,32,13,94,231,189,126,241,202,\n                                    200,106,38,231,83,...>>,\n                                 consensus_members =>\n                                  <<131,108,0,0,0,43,109,0,0,0,33,0,255,172,\n                                    249,8,125,152,93,253,218,106,163,67,42,112,\n                                    146,234,178,29,90,...>>,\n                                 current_height => <<131,98,0,14,157,3>>,\n                                 dc_accounts =>\n                                  <<33,0,0,0,0,0,56,11,97,223,171,33,53,81,211,\n                                    31,234,34,160,150,24,247,187,145,231,83,33,\n                                    109,235,43,142,...>>,\n                                 delayed_hnt =>\n                                  <<131,108,0,0,0,1,104,2,109,0,0,0,17,36,99,\n                                    100,95,98,108,111,99,107,95,49,49,54,50,48,\n                                    50,57,109,...>>,\n                                 delayed_vars => <<131,106>>,\n                                 election_epoch => <<131,98,0,0,95,222>>,\n                                 election_height => <<131,98,0,14,156,245>>,\n                                 gateways =>\n                                  <<33,0,0,0,0,0,1,26,119,96,147,61,116,24,18,\n                                    82,246,56,198,1,237,111,157,113,196,254,\n                                    181,223,200,121,220,...>>,\n                                 h3dex =>\n                                  <<131,108,0,1,160,218,104,2,110,8,0,255,13,\n                                    215,43,54,116,192,8,108,0,0,0,1,109,0,0,0,\n                                    33,0,229,...>>,\n                                 hexes =>\n                                  <<131,108,0,0,27,102,104,2,110,8,0,255,255,\n                                    255,63,54,116,80,8,108,0,0,0,1,109,0,0,0,\n                                    33,0,229,...>>,\n                                 htlcs =>\n                                  <<131,108,0,0,0,11,104,2,109,0,0,0,33,1,100,\n                                    203,32,171,134,186,234,144,149,92,232,57,\n                                    128,76,68,195,174,...>>,\n                                 master_key =>\n                                  <<131,109,0,0,0,33,0,223,191,171,128,182,185,\n                                    208,20,185,247,22,115,143,28,6,51,75,181,\n                                    65,82,29,47,254,120,...>>,\n                                 multi_keys => <<131,106>>,\n                                 oracle_price => <<131,98,86,171,188,140>>,\n                                 oracle_price_list =>\n                                  <<131,108,0,0,0,131,104,5,100,0,18,111,114,\n                                    97,99,108,101,95,112,114,105,99,101,95,101,\n                                    110,116,114,121,98,97,...>>,\n                                 oui_counter => <<131,97,12>>,\n                                 ouis =>\n                                  <<131,108,0,0,0,12,104,2,97,1,104,7,100,0,10,\n                                    114,111,117,116,105,110,103,95,118,49,97,1,\n                                    109,0,0,0,...>>,\n                                 pocs =>\n                                  <<32,0,0,0,0,0,77,11,223,76,76,145,18,128,\n                                    235,44,12,39,89,175,175,78,204,214,161,197,\n                                    21,211,158,153,88,...>>,\n                                 security_accounts =>\n                                  <<33,0,0,0,1,4,241,54,165,156,232,123,15,255,\n                                    171,178,79,11,164,131,127,42,106,43,8,11,\n                                    148,165,127,15,177,...>>,\n                                 state_channels =>\n                                  <<131,108,0,0,0,30,104,2,109,0,0,0,65,0,77,\n                                    17,156,210,89,30,33,218,218,22,68,16,247,\n                                    97,1,249,107,...>>,\n                                 subnets =>\n                                  <<131,108,0,0,0,12,104,2,109,0,0,0,6,0,0,0,\n                                    127,255,0,97,1,104,2,109,0,0,0,6,0,2,0,...>>,\n                                 threshold_txns => <<131,106>>,\n                                 transaction_fee => <<131,97,0>>,\n                                 upgrades =>\n                                  <<131,108,0,0,0,7,109,0,0,0,10,103,97,116,\n                                    101,119,97,121,95,118,50,109,0,0,0,11,104,\n                                    101,120,95,116,...>>,\n                                 validators =>\n                                  <<131,108,0,0,8,208,104,2,109,0,0,0,33,0,0,\n                                    61,1,214,151,81,202,247,217,189,140,152,58,\n                                    4,67,181,83,...>>,\n                                 vars =>\n                                  <<131,108,0,0,0,141,104,2,109,0,0,0,22,97,\n                                    108,108,111,119,95,112,97,121,109,101,110,\n                                    116,95,118,50,95,109,...>>,\n                                 vars_nonce => <<131,97,81>>,\n                                 version => v6}},\n                              infinity]}}}])\n  in call from erl_eval:do_apply/6 (erl_eval.erl, line 680)\n  in call from erl_eval:exprs/5 (erl_eval.erl, line 126)\n  in call from escript:eval_exprs/5 (escript.erl, line 872)\n  in call from erl_eval:local_func/6 (erl_eval.erl, line 567)\n  in call from escript:interpret/4 (escript.erl, line 788)\n  in call from escript:start/1 (escript.erl, line 277)\n  in call from init:start_em/1 \n"
root@hot_dinosaur:~ #

Errors in console.log

2021-08-09 03:26:59.411 [warning] <0.5358.0>@blockchain_txn:separate_res:373 invalid txn blockchain_txn_poc_receipts_v1 : {error,not_found} / type=poc_receipts_v1 hash="148VpZomXv5wWM5dwDkjcy1bBS5VFDbS6pHepimG7BVXK3LJ8q" challenger="hot-hemp-hyena" onion="12wGX9YBy8tHQ7ireq64dUcu3MxjDzkXcQNTjWZzLqWs7fLR1JR" path:
    type=element challengee: sunny-walnut-opossum, receipt: type=receipt gateway: sunny-walnut-opossum timestamp: 1628457489560939516 signal: 0 origin: p2p
        witnesses:
2021-08-09 03:26:59.411 [warning] <0.5358.0>@blockchain_txn:separate_res:373 invalid txn blockchain_txn_poc_receipts_v1 : {error,not_found} / type=poc_receipts_v1 hash="12NGoFWDZcV2CFR1dFBsu8i2RwDRBgCtQ6HKe74k9iC91fWiRbn" challenger="hidden-citron-bat" onion="1whtNGXP635A1YX5Qiyud9sTqZHCrPhEsABXhwqMzygCftVFAz" path:
    type=element challengee: powerful-sage-puma, receipt: type=receipt gateway: powerful-sage-puma timestamp: 1628457449137622015 signal: 0 origin: p2p
        witnesses: type=witness gateway: "dandy-alabaster-lobster" timestamp: 1628457450564564915 signal: -107
2021-08-09 03:26:59.411 [warning] <0.5358.0>@blockchain_txn:separate_res:373 invalid txn blockchain_txn_poc_receipts_v1 : {error,not_found} / type=poc_receipts_v1 hash="1Lib8RHmBuGx7wTnut9YA5nHMVeWgFN4tG7LP81jCS5zeYzbxB" challenger="spicy-ash-wombat" onion="1m2csXPSQxhCvSptLHQZfRqJK2HkC2D4p1cbdmzTKTPuv1jaKt" path:
    type=element challengee: witty-navy-mouse, receipt: type=receipt gateway: witty-navy-mouse timestamp: 1628457419308500013 signal: 0 origin: p2p
        witnesses:
2021-08-09 03:26:59.411 [warning] <0.5358.0>@blockchain_txn:separate_res:373 invalid txn blockchain_txn_poc_receipts_v1 : {error,not_found} / type=poc_receipts_v1 hash="1hrXwZc6TUwwA2xpij61EgfH7h5D1osc9PtWotUogn54dYozQK" challenger="clean-peanut-vulture" onion="12i1f6WX3Vo5keEGYcHtBzTx5yrn8MFV1DQVK9Epd2JebX5pat2" path:
    type=element challengee: virtual-rose-koala, receipt: type=receipt gateway: virtual-rose-koala timestamp: 1628457545406890573 signal: 0 origin: p2p
        witnesses:
2021-08-09 03:26:59.413 [warning] <0.5358.0>@blockchain_txn:separate_res:373 invalid txn blockchain_txn_poc_receipts_v1 : {error,not_found} / type=poc_receipts_v1 hash="1SzgTrryR9VdGPw25zMaRccTGK74xAZxZxog3bgNt2ZNzspom1" challenger="tricky-wool-mongoose" onion="12C3A2ijGhCWSBpG4EzpqcRUyK9okESecNbkrZ1Q7V5hPYeMjKa" path:
    type=element challengee: plain-opal-wren, receipt: type=receipt gateway: plain-opal-wren timestamp: 1628457475106845313 signal: 0 origin: p2p
        witnesses: type=witness gateway: "calm-glossy-parakeet" timestamp: 1628457476528735340 signal: -114
        type=witness gateway: "brisk-bamboo-flamingo" timestamp: 1628457474513336368 signal: -83
        type=witness gateway: "petite-mango-hedgehog" timestamp: 1628457474641263373 signal: -103
        type=witness gateway: "loud-jetblack-owl" timestamp: 1628457474414535746 signal: -103
        type=witness gateway: "droll-rouge-fox" timestamp: 1628457476453116284 signal: -104
        type=witness gateway: "rare-cedar-condor" timestamp: 1628457476516256694 signal: -80
        type=witness gateway: "odd-seafoam-manatee" timestamp: 1628457476518234052 signal: -111
        type=witness gateway: "salty-brick-swift" timestamp: 1628457474525487868 signal: -113
        type=witness gateway: "narrow-ebony-giraffe" timestamp: 1628457476528301874 signal: -111
        type=witness gateway: "fresh-mustard-fox" timestamp: 1628457476513711176 signal: -113
        type=witness gateway: "small-mauve-elk" timestamp: 1628457476578037346 signal: -109
        type=witness gateway: "unique-metal-cat" timestamp: 1628457474401730284 signal: -121
        type=witness gateway: "fancy-purple-beetle" timestamp: 1628457476521520248 signal: -103
        type=witness gateway: "melted-ultraviolet-mole" timestamp: 1628457476517873881 signal: -115
        type=witness gateway: "energetic-macaroon-rattlesnake" timestamp: 1628457473873068163 signal: -109
        type=witness gateway: "festive-punch-liger" timestamp: 1628457476520555780 signal: -102
        type=witness gateway: "dandy-chambray-chipmunk" timestamp: 1628457476528027640 signal: -121
        type=witness gateway: "lucky-mahogany-starling" timestamp: 1628457474766763330 signal: -111
        type=witness gateway: "tangy-ash-goldfish" timestamp: 1628457476527306299 signal: -117
        type=witness gateway: "bubbly-daffodil-python" timestamp: 1628457476526412799 signal: -104
        type=witness gateway: "striped-leather-squirrel" timestamp: 1628457476517437796 signal: -112
        type=witness gateway: "stable-iris-sealion" timestamp: 1628457476517369633 signal: -111
        type=witness gateway: "daring-blonde-otter" timestamp: 1628457474109780579 signal: -121
        type=witness gateway: "rapid-violet-baboon" timestamp: 1628457474785419373 signal: -118
2021-08-09 03:26:59.414 [warning] <0.5358.0>@blockchain_txn:separate_res:373 invalid txn blockchain_txn_poc_receipts_v1 : {error,not_found} / type=poc_receipts_v1 hash="12Q8tP2hJqY34869bHFsPeQS8pruGYmLWquy8cxNKt6W43myT6R" challenger="merry-nylon-goat" onion="12c8QFiQuvELoEvNAzXxZr3zmF2ahpAtbZkYNPwchP1xF3nvLZ" path:
    type=element challengee: bumpy-cerulean-vulture, receipt: type=receipt gateway: bumpy-cerulean-vulture timestamp: 1628457572913600178 signal: 0 origin: p2p
        witnesses:
2021-08-09 03:26:59.414 [warning] <0.5358.0>@blockchain_txn:separate_res:373 invalid txn blockchain_txn_poc_receipts_v1 : {error,not_found} / type=poc_receipts_v1 hash="1zFXfBiyrDKjh2eniFQdnoSkP2346dKVFmTATVGqEe6j95aRq6" challenger="little-alabaster-mandrill" onion="19Ed8wxCwk8YofyfvMc548Xh2ichfTCqEdLiYkyCW47qD8pqM4" path:
    type=element challengee: fancy-mocha-otter, receipt: type=receipt gateway: fancy-mocha-otter timestamp: 1628457643131213088 signal: 0 origin: p2p
        witnesses: type=witness gateway: "obedient-myrtle-lynx" timestamp: 1628457644477587623 signal: -110
        type=witness gateway: "zesty-lemonade-salmon" timestamp: 1628457644545809621 signal: -111
        type=witness gateway: "little-macaroon-dragon" timestamp: 1628457644533215976 signal: -109
2021-08-09 03:26:59.415 [warning] <0.5358.0>@blockchain_txn:separate_res:373 invalid txn blockchain_txn_poc_receipts_v1 : {error,not_found} / type=poc_receipts_v1 hash="12ZsUJgRt8wnowHb2cTXN2PLUX6qXNUzfYn3UoVFV2JHRh26Gzm" challenger="quick-spruce-swift" onion="1qJuUFwgcfzDbyJpFRS1yK5d4uF9MJ97E6Gd6FryuH6BRzSHgo" path:
    type=element challengee: shallow-topaz-cuckoo, receipt: type=receipt gateway: shallow-topaz-cuckoo timestamp: 1628457554667445213 signal: 0 origin: p2p
        witnesses: type=witness gateway: "docile-coral-yak" timestamp: 1628457556106059391 signal: -115
        type=witness gateway: "acrobatic-wooden-gazelle" timestamp: 1628457556078770239 signal: -111
        type=witness gateway: "prehistoric-fossilized-eel" timestamp: 1628457553917056202 signal: -115
        type=witness gateway: "rare-indigo-manatee" timestamp: 1628457554444645031 signal: -114
        type=witness gateway: "tall-chambray-orangutan" timestamp: 1628457556083721740 signal: -111
        type=witness gateway: "dry-burlap-dolphin" timestamp: 1628457556085769208 signal: -109
2021-08-09 03:26:59.416 [warning] <0.5358.0>@blockchain_txn:separate_res:373 invalid txn blockchain_txn_poc_receipts_v1 : {error,not_found} / type=poc_receipts_v1 hash="1JmGaEWTfWXZc9gKdGrZro1kjJZESYrkZmgGjW1zpF43jKEQJz" challenger="bent-taffy-duck" onion="12Qp3ABGuDa9b3NUmtdhVE3q5tgCeoSsz1M3GXTQkXkY3evcJgZ" path:
    type=element challengee: formal-snowy-kitten, receipt: type=receipt gateway: formal-snowy-kitten timestamp: 1628457547032124653 signal: 0 origin: p2p
        witnesses: type=witness gateway: "straight-eggshell-donkey" timestamp: 1628457548367629491 signal: -120
        type=witness gateway: "active-mahogany-woodpecker" timestamp: 1628457550106719272 signal: -98
        type=witness gateway: "soaring-carrot-sawfish" timestamp: 1628457548407433370 signal: -117
        type=witness gateway: "rough-silver-sheep" timestamp: 1628457550101623705 signal: -109
        type=witness gateway: "quaint-fuchsia-lobster" timestamp: 1628457549422978872 signal: -117
        type=witness gateway: "fluffy-licorice-donkey" timestamp: 1628457550099635203 signal: -115
        type=witness gateway: "sharp-white-lizard" timestamp: 1628457550104754047 signal: -116
        type=witness gateway: "modern-bone-troll" timestamp: 1628457548210746366 signal: -119
2021-08-09 03:26:59.416 [warning] <0.5358.0>@blockchain_txn:separate_res:373 invalid txn blockchain_txn_poc_receipts_v1 : {error,not_found} / type=poc_receipts_v1 hash="1qxhad9cLZ9puyDCoRSPAB2aTGTohjrUX96aorCanudaykQ6aE" challenger="exotic-mocha-loris" onion="1Y3qQPrw6Dw3oZSr5MW1DVvDbABTs8C4KVyZWGSuDyFyAVRXGJ" path:
    type=element challengee: cheesy-juniper-gorilla, receipt: type=receipt gateway: cheesy-juniper-gorilla timestamp: 1628457524783147686 signal: 0 origin: p2p
        witnesses: type=witness gateway: "quaint-laurel-lizard" timestamp: 1628457524038012813 signal: -114
2021-08-09 03:26:59.418 [warning] <0.5358.0>@blockchain_txn:separate_res:373 invalid txn blockchain_txn_poc_receipts_v1 : {error,not_found} / type=poc_receipts_v1 hash="122MakvpDvgMzL4f9FHpEXE44sriQJHZXh29M4UCHTsybau1V6U" challenger="warm-chiffon-elk" onion="1YvUs5nVyqZdc1tfpuMwwbodL7Ywxb5EmXXJZyGh3LiAYuFHzp" path:
    type=element challengee: gentle-graphite-mockingbird, receipt: type=receipt gateway: gentle-graphite-mockingbird timestamp: 1628457084631856394 signal: 0 origin: p2p
        witnesses: type=witness gateway: "deep-linen-hornet" timestamp: 1628457089768362343 signal: -114
        type=witness gateway: "short-lemonade-salmon" timestamp: 1628457089749175708 signal: -97
        type=witness gateway: "tall-candy-falcon" timestamp: 1628457089750043218 signal: -116
        type=witness gateway: "rare-charcoal-rabbit" timestamp: 1628457089779881539 signal: -107
        type=witness gateway: "sharp-vermilion-shell" timestamp: 1628457089752829628 signal: -113
        type=witness gateway: "tricky-plastic-mustang" timestamp: 1628457089769434318 signal: -114
        type=witness gateway: "trendy-myrtle-hippo" timestamp: 1628457089768283744 signal: -112
        type=witness gateway: "lone-flaxen-narwhal" timestamp: 1628457089755861898 signal: -117
        type=witness gateway: "gentle-merlot-pike" timestamp: 1628457087931958771 signal: -118
        type=witness gateway: "old-pear-vulture" timestamp: 1628457089813462555 signal: -117
        type=witness gateway: "proper-lead-capybara" timestamp: 1628457089779997973 signal: -113
        type=witness gateway: "plain-olive-porpoise" timestamp: 1628457089773549919 signal: -111
        type=witness gateway: "ambitious-denim-tardigrade" timestamp: 1628457087628662979 signal: -118
        type=witness gateway: "handsome-eggplant-shetland" timestamp: 1628457089770207652 signal: -111
        type=witness gateway: "atomic-rusty-sawfish" timestamp: 1628457089101421715 signal: -115
        type=witness gateway: "strong-hazel-starfish" timestamp: 1628457089751365113 signal: -113
        type=witness gateway: "helpful-foggy-stallion" timestamp: 1628457089773331479 signal: -114
        type=witness gateway: "late-walnut-blackbird" timestamp: 1628457087930084354 signal: -108
        type=witness gateway: "ambitious-daffodil-loris" timestamp: 1628457089747304344 signal: -117
        type=witness gateway: "fierce-azure-rook" timestamp: 1628457089747815937 signal: -115
        type=witness gateway: "shiny-walnut-loris" timestamp: 1628457089749641476 signal: -113
        type=witness gateway: "sharp-cream-okapi" timestamp: 1628457103629628127 signal: -114
2021-08-09 03:26:59.418 [warning] <0.5358.0>@blockchain_txn:separate_res:373 invalid txn blockchain_txn_poc_receipts_v1 : {error,not_found} / type=poc_receipts_v1 hash="12qf2ELi5QV1qMS7PCVTjxnrE2aRnCwnSZpJ4MiEeNtXNWvAkKB" challenger="scrawny-mulberry-cow" onion="12DV4Bx4Z5CS8hGWP1fwsonHhAaENMKgJycJ6G7dTWXrZGezpj2" path:
    type=element challengee: bright-tartan-mouse, receipt: type=receipt gateway: bright-tartan-mouse timestamp: 1628456991412773418 signal: 0 origin: p2p
        witnesses: type=witness gateway: "stale-bone-goose" timestamp: 1628456992821302128 signal: -97
        type=witness gateway: "curved-crimson-puma" timestamp: 1628456992828523430 signal: -111
        type=witness gateway: "droll-aqua-pheasant" timestamp: 1628456992824599233 signal: -112
        type=witness gateway: "atomic-watermelon-marmot" timestamp: 1628456990703305142 signal: -118
        type=witness gateway: "electric-crimson-pig" timestamp: 1628456992766187855 signal: -113
        type=witness gateway: "eager-flaxen-canary" timestamp: 1628456992823252236 signal: -110
2021-08-09 03:26:59.418 [warning] <0.5358.0>@blockchain_txn:separate_res:373 invalid txn blockchain_txn_poc_receipts_v1 : {error,not_found} / type=poc_receipts_v1 hash="1MvuuxB5dUawPvfx5V5sZ8TEPwxZnTj9odVtDktDWEPcAsJuWm" challenger="hollow-aqua-lynx" onion="12RBfcJFeGqzBWiHMCHzhmt8ShjPFMhhXT117orVQiBw3tS8iqv" path:
    type=element challengee: damaged-infrared-otter, receipt: type=receipt gateway: damaged-infrared-otter timestamp: 1628457428129268024 signal: 0 origin: p2p
        witnesses:
2021-08-09 03:26:59.419 [warning] <0.5358.0>@blockchain_txn:separate_res:373 invalid txn blockchain_txn_poc_receipts_v1 : {error,not_found} / type=poc_receipts_v1 hash="1fZbe3s6hEhB6cNdTkJKbjMX2v1N84wsjdKqV4eEpQoED7ZDFj" challenger="brisk-garnet-jaguar" onion="1gxEV2PQhAL1w98eoUrdtYbDf7vuwHnYF1TJHNhDC83LUgT2jM" path:
    type=element challengee: low-dijon-narwhal, receipt: type=receipt gateway: low-dijon-narwhal timestamp: 1628457428611691184 signal: 0 origin: p2p
        witnesses: type=witness gateway: "polished-jetblack-salmon" timestamp: 1628457429959831268 signal: -113
        type=witness gateway: "early-hickory-squid" timestamp: 1628457429737044267 signal: -114
        type=witness gateway: "wonderful-fuchsia-peacock" timestamp: 1628457429933088393 signal: -93
        type=witness gateway: "faithful-syrup-tadpole" timestamp: 1628457429927587309 signal: -108
        type=witness gateway: "ripe-black-sparrow" timestamp: 1628457365363016826 signal: -108
2021-08-09 03:26:59.420 [warning] <0.5358.0>@blockchain_txn:separate_res:373 invalid txn blockchain_txn_poc_receipts_v1 : {error,not_found} / type=poc_receipts_v1 hash="125SVZTZtnqAczFd1jV4HMwQqUCni97yuegkwd4FgqKWXGzcZtA" challenger="lively-rusty-crab" onion="1z3p9jQTmsVfQXvFpfFQmLCEdAGG1CyZm6sitbZUkNup3JWwkU" path:
    type=element challengee: fluffy-glass-mantaray, receipt: type=receipt gateway: fluffy-glass-mantaray timestamp: 1628457604806953001 signal: 0 origin: p2p
        witnesses: type=witness gateway: "loud-chocolate-mongoose" timestamp: 1628457606222669380 signal: -94
        type=witness gateway: "abundant-misty-bat" timestamp: 1628457606225797954 signal: -109
        type=witness gateway: "alert-gunmetal-spider" timestamp: 1628457604109496017 signal: -113
        type=witness gateway: "refined-ash-wolverine" timestamp: 1628457606234991271 signal: -96
        type=witness gateway: "round-mercurial-newt" timestamp: 1628457606218380118 signal: -107
        type=witness gateway: "quaint-aquamarine-pig" timestamp: 1628457606219792405 signal: -107
2021-08-09 03:26:59.420 [warning] <0.5358.0>@blockchain_txn:separate_res:373 invalid txn blockchain_txn_poc_receipts_v1 : {error,not_found} / type=poc_receipts_v1 hash="15nD4gqz87qnpXEFT6fEQxLhY4TUvJuq3bTEtdtd41LnpHZJ2Q" challenger="beautiful-coral-troll" onion="12L3GBbGJG9juxcwHxFehZFtPUTdEfYoi4DzBfeECY4ohE9eCAk" path:
    type=element challengee: soaring-caramel-bat, receipt: type=receipt gateway: soaring-caramel-bat timestamp: 1628457513034487099 signal: 0 origin: p2p
        witnesses: type=witness gateway: "bouncy-latte-aardvark" timestamp: 1628457516900957686 signal: -112
2021-08-09 03:26:59.421 [warning] <0.5358.0>@blockchain_txn:separate_res:373 invalid txn blockchain_txn_poc_receipts_v1 : {error,not_found} / type=poc_receipts_v1 hash="12ba7PgZsNVyS4MxEPDKDLAbBDaXKg7FF4WJN61f5Aa5VsXD8K2" challenger="quaint-pearl-crane" onion="12Tm2GFfXYHKjGtrNBmsyB7wDU68UTCM8vkXJwcqoxUK5QmX5Ag" path:
    type=element challengee: zesty-burlap-boa, receipt: type=receipt gateway: zesty-burlap-boa timestamp: 1628457626268884195 signal: 0 origin: p2p
        witnesses: type=witness gateway: "quiet-fuchsia-nuthatch" timestamp: 1628457627552930113 signal: -115
        type=witness gateway: "attractive-chrome-gerbil" timestamp: 1628457627352469238 signal: -102
        type=witness gateway: "overt-mustard-puppy" timestamp: 1628457627700116862 signal: -104
        type=witness gateway: "tall-wintergreen-spider" timestamp: 1628457627781014696 signal: -106
2021-08-09 03:26:59.421 [warning] <0.5358.0>@blockchain_txn:separate_res:373 invalid txn blockchain_txn_poc_receipts_v1 : {error,not_found} / type=poc_receipts_v1 hash="12DRPQFi9Pa4zaHtuMyWKBdRJPJTYZTvxtagbffpZyTtVFfe4Ce" challenger="mean-sky-tuna" onion="12JPcnGrTnwXECgkMJ5xDffsz3RNAr89sQhzfyYi4FJDotshxX5" path:
    type=element challengee: acidic-wintergreen-chameleon, receipt: type=receipt gateway: acidic-wintergreen-chameleon timestamp: 1628457460584095813 signal: 0 origin: p2p
        witnesses:
2021-08-09 03:26:59.423 [warning] <0.5358.0>@blockchain_txn:separate_res:373 invalid txn blockchain_txn_poc_receipts_v1 : {error,not_found} / type=poc_receipts_v1 hash="1rras31D3nsLbV5bHpevjgwdZ8rVoTXMcBMnNdSLZhg1sSTthV" challenger="dancing-shadow-gazelle" onion="1MahJLnyY8XopZLrc3NAXgZnAyjuRAPKTFP9yRQ3fSZMrKg1ik" path:
    type=element challengee: damp-plum-crow, receipt: type=receipt gateway: damp-plum-crow timestamp: 1628457425242426505 signal: 0 origin: p2p
        witnesses: type=witness gateway: "funny-gunmetal-lizard" timestamp: 1628457428862896994 signal: -115
        type=witness gateway: "striped-pickle-cyborg" timestamp: 1628457428857150327 signal: -117
        type=witness gateway: "muscular-amber-anteater" timestamp: 1628457428877950204 signal: -108
        type=witness gateway: "beautiful-lava-mule" timestamp: 1628457428862829306 signal: -118
        type=witness gateway: "melodic-ceramic-mouse" timestamp: 1628457426765990058 signal: -114
        type=witness gateway: "howling-mahogany-lobster" timestamp: 1628457428853296692 signal: -117
        type=witness gateway: "deep-wooden-bison" timestamp: 1628457428877822896 signal: -116
        type=witness gateway: "square-khaki-fly" timestamp: 1628457428913363756 signal: -118
        type=witness gateway: "best-menthol-finch" timestamp: 1628457428910209227 signal: -115
        type=witness gateway: "savory-flint-buffalo" timestamp: 1628457428861812910 signal: -116
        type=witness gateway: "brisk-opaque-eagle" timestamp: 1628457428941017122 signal: -119
        type=witness gateway: "gentle-vanilla-wombat" timestamp: 1628457428860114598 signal: -116
        type=witness gateway: "radiant-grey-narwhal" timestamp: 1628457428860273651 signal: -116
        type=witness gateway: "daring-pewter-hamster" timestamp: 1628457428916936339 signal: -116
        type=witness gateway: "rough-eggshell-shrimp" timestamp: 1628457428858041693 signal: -116
        type=witness gateway: "shiny-taffy-rat" timestamp: 1628457428859295570 signal: -118
        type=witness gateway: "blunt-watermelon-coyote" timestamp: 1628457428862484241 signal: -92
        type=witness gateway: "future-laurel-vulture" timestamp: 1628457426725397058 signal: -118
        type=witness gateway: "huge-glossy-scallop" timestamp: 1628457428874851668 signal: -119
        type=witness gateway: "dazzling-misty-shell" timestamp: 1628457428867109484 signal: -116
        type=witness gateway: "soaring-cloud-dolphin" timestamp: 1628457428927175112 signal: -120
        type=witness gateway: "brilliant-pink-goldfish" timestamp: 1628457428861945149 signal: -104
        type=witness gateway: "prehistoric-misty-salmon" timestamp: 1628457426908286308 signal: -113
        type=witness gateway: "fit-mossy-trout" timestamp: 1628457428859545545 signal: -115
2021-08-09 03:26:59.423 [warning] <0.5358.0>@blockchain_txn:separate_res:373 invalid txn blockchain_txn_poc_receipts_v1 : {error,not_found} / type=poc_receipts_v1 hash="19ijB3Go8FJTGNt83EYoV1bTGd6cztZqcDuqktFvhbV3fWGRNP" challenger="odd-leather-oyster" onion="1G98AgH5ZVExnT1UQv2JBmLtMzQBu3VDhxqNx7sRiJ8tTf4waq" path:
    type=element challengee: mammoth-hickory-pelican, receipt: type=receipt gateway: mammoth-hickory-pelican timestamp: 1628457570654107505 signal: 0 origin: p2p
        witnesses: type=witness gateway: "faint-oily-jellyfish" timestamp: 1628457572063224458 signal: -106
2021-08-09 03:26:59.424 [warning] <0.5358.0>@blockchain_txn:separate_res:373 invalid txn blockchain_txn_poc_receipts_v1 : {error,not_found} / type=poc_receipts_v1 hash="1HPYw2jodQ97DcTM72E5QfCaMsxJpaQ6q1BKVBam2Q76M6WnoN" challenger="precise-goldenrod-lynx" onion="1z9jcXjSKhHfM9erARSmmfZpVJoVU9odSxxfXusBGrXPgVsFM6" path:
    type=element challengee: spicy-blush-albatross, receipt: type=receipt gateway: spicy-blush-albatross timestamp: 1628457526585789893 signal: 0 origin: p2p
        witnesses: type=witness gateway: "young-blue-shark" timestamp: 1628457531954659976 signal: -120
2021-08-09 03:26:59.425 [warning] <0.5358.0>@blockchain_txn:separate_res:373 invalid txn blockchain_txn_poc_receipts_v1 : {error,not_found} / type=poc_receipts_v1 hash="1ikKu3ku6yusUVZGuXADWWiwXctJ5yzMB1jcMGz4rY86FYEYLj" challenger="raspy-blue-yak" onion="1KisjXJK4j2HhMcpihDRMVPh1Y667No8TD8Xf1FhKf9kNgxjh5" path:
    type=element challengee: howling-crepe-tardigrade, receipt: type=receipt gateway: howling-crepe-tardigrade timestamp: 1628457656152914721 signal: 0 origin: p2p
        witnesses: type=witness gateway: "swift-red-puppy" timestamp: 1628457657577251997 signal: -99
        type=witness gateway: "mean-tiger-squirrel" timestamp: 1628457657496660692 signal: -56
        type=witness gateway: "dry-saffron-terrier" timestamp: 1628457657566936893 signal: -92
        type=witness gateway: "little-eggshell-chimpanzee" timestamp: 1628457657564748490 signal: -103
        type=witness gateway: "witty-boysenberry-wasp" timestamp: 1628457657508022432 signal: -60
        type=witness gateway: "mammoth-blonde-anteater" timestamp: 1628457657567716301 signal: -107
        type=witness gateway: "sharp-carbon-jaguar" timestamp: 1628457657564360038 signal: -103
        type=witness gateway: "rich-sapphire-lemur" timestamp: 1628457657566191865 signal: -105
        type=witness gateway: "wide-hemp-beetle" timestamp: 1628457655687490667 signal: -115
        type=witness gateway: "curved-crimson-puma" timestamp: 1628457657571019913 signal: -104
        type=witness gateway: "festive-magenta-liger" timestamp: 1628457657562541056 signal: -108
        type=witness gateway: "fancy-iron-pheasant" timestamp: 1628457657574879449 signal: -113
        type=witness gateway: "agreeable-daffodil-dragon" timestamp: 1628457655811154959 signal: -109
        type=witness gateway: "brisk-juniper-hamster" timestamp: 1628457657570702008 signal: -102
        type=witness gateway: "happy-taffy-orca" timestamp: 1628457657566033588 signal: -97
        type=witness gateway: "eager-lemon-jellyfish" timestamp: 1628457655690848291 signal: -99
        type=witness gateway: "refined-khaki-sidewinder" timestamp: 1628457657567293720 signal: -113
        type=witness gateway: "boxy-carrot-cat" timestamp: 1628457657564859725 signal: -111
2021-08-09 03:26:59.425 [warning] <0.5358.0>@blockchain_txn:separate_res:373 invalid txn blockchain_txn_poc_receipts_v1 : {error,not_found} / type=poc_receipts_v1 hash="1wTQyrTbew21DAMzT1dhUjVrPVxvX4odRNcXQ12F8XmuCyAfgh" challenger="feisty-lilac-elephant" onion="121RfYEdmNjKg6YenVggUTKn6nzefmsgkj8W1XZcbxFCZUkob48" path:
    type=element challengee: breezy-jade-pangolin, receipt: type=receipt gateway: breezy-jade-pangolin timestamp: 1628457289315567280 signal: 0 origin: p2p
        witnesses:
2021-08-09 03:26:59.426 [warning] <0.5358.0>@blockchain_txn:separate_res:373 invalid txn blockchain_txn_poc_receipts_v1 : {error,not_found} / type=poc_receipts_v1 hash="12bnCnCzfSJQv9dfLujJsXyffLkMDg1Xo1dhfE4t6D8aPkRU3bt" challenger="restless-pewter-starling" onion="12HW4edAWZVFmk8BEdFNXgBiyEiwPwLapC6XNvr6LoTunwQWWzq" path:
    type=element challengee: upbeat-coconut-dragonfly, receipt: type=receipt gateway: upbeat-coconut-dragonfly timestamp: 1628457624878258817 signal: 0 origin: p2p
        witnesses: type=witness gateway: "recumbent-blood-terrier" timestamp: 1628457628453460655 signal: -113
2021-08-09 03:26:59.426 [warning] <0.5358.0>@blockchain_txn:separate_res:373 invalid txn blockchain_txn_poc_receipts_v1 : {error,not_found} / type=poc_receipts_v1 hash="14bER5gFKDyhEDWxayZ1jJHFnrU8C3a4W1zm6YgwQnWTXZjUv9" challenger="proper-ebony-crane" onion="128mEe4XLMR4SFrPJTqSrYZhnFW8ASS6poEpN7nwJrL3y9w7Qkt" path:
    type=element challengee: lucky-ash-dinosaur, receipt: type=receipt gateway: lucky-ash-dinosaur timestamp: 1628457629317075720 signal: 0 origin: p2p
        witnesses:
2021-08-09 03:26:59.426 [warning] <0.5358.0>@blockchain_txn:separate_res:373 invalid txn blockchain_txn_poc_receipts_v1 : {error,not_found} / type=poc_receipts_v1 hash="18HGYgL5yb9xwmfQS7QNcCK5cc69cUAXf2xvD17RHimTkYzqrR" challenger="faint-hotpink-mole" onion="124FjftPY64FUyyyYs9VFNiRk4AHUxyRAa2Uthc4BHXCPtdgqXY" path:
    type=element challengee: calm-saffron-aphid, receipt: type=receipt gateway: calm-saffron-aphid timestamp: 1628457562658666702 signal: 0 origin: p2p
        witnesses:
2021-08-09 03:26:59.428 [warning] <0.5358.0>@blockchain_txn:separate_res:373 invalid txn blockchain_txn_poc_receipts_v1 : {error,not_found} / type=poc_receipts_v1 hash="1rvC5zLCGEx2Py7cnFHS4oNEx89WyREPNVGteY9UKoVUzRtfnC" challenger="small-red-coyote" onion="12U8NpxFkRh2FrXAaDZXxrpK72WqFRz6cnsTdksWZXaJxD3FvSL" path:
    type=element challengee: clean-grape-leopard, receipt: type=receipt gateway: clean-grape-leopard timestamp: 1628457554551043869 signal: 0 origin: p2p
        witnesses: type=witness gateway: "boxy-cinnabar-dachshund" timestamp: 1628457555771572995 signal: -112
        type=witness gateway: "large-heather-crow" timestamp: 1628457555860209577 signal: -111
        type=witness gateway: "urban-sable-yeti" timestamp: 1628457555875955535 signal: -109
        type=witness gateway: "sharp-raspberry-kookaburra" timestamp: 1628457555970995287 signal: -111
        type=witness gateway: "quiet-smoke-viper" timestamp: 1628457555738099702 signal: -111
        type=witness gateway: "ambitious-inky-viper" timestamp: 1628457555637435661 signal: -108
        type=witness gateway: "fancy-navy-jellyfish" timestamp: 1628457557559476545 signal: -106
        type=witness gateway: "dazzling-rusty-orangutan" timestamp: 1628457555683953453 signal: -113
        type=witness gateway: "recumbent-chrome-raccoon" timestamp: 1628457555503376494 signal: -109
        type=witness gateway: "abundant-shadow-marmot" timestamp: 1628457556038714162 signal: -112
        type=witness gateway: "powerful-arctic-pheasant" timestamp: 1628457555941440995 signal: -109
        type=witness gateway: "colossal-honey-dove" timestamp: 1628457555875285037 signal: -101
        type=witness gateway: "high-ebony-butterfly" timestamp: 1628457555551139078 signal: -112
        type=witness gateway: "silly-coffee-gecko" timestamp: 1628457555661371661 signal: -111
        type=witness gateway: "polished-blonde-octopus" timestamp: 1628457555803988036 signal: -112
        type=witness gateway: "best-marmalade-hamster" timestamp: 1628457555959492453 signal: -111
        type=witness gateway: "sharp-brunette-yeti" timestamp: 1628457556017509912 signal: -113
        type=witness gateway: "tiny-latte-moth" timestamp: 1628457555808420702 signal: -115
        type=witness gateway: "damp-pewter-ram" timestamp: 1628457556384943787 signal: -114
        type=witness gateway: "prehistoric-pink-porcupine" timestamp: 1628457555705935453 signal: -103
        type=witness gateway: "breezy-slate-lizard" timestamp: 1628457555988033912 signal: -113
        type=witness gateway: "salty-cerulean-condor" timestamp: 1628457555826186161 signal: -114
        type=witness gateway: "loud-mulberry-haddock" timestamp: 1628457555677975828 signal: -113
        type=witness gateway: "daring-tortilla-duck" timestamp: 1628457555916283578 signal: -112
        type=witness gateway: "fast-pink-wombat" timestamp: 1628457555993364828 signal: -114
2021-08-09 03:26:59.429 [warning] <0.5358.0>@blockchain_txn:separate_res:373 invalid txn blockchain_txn_poc_receipts_v1 : {error,not_found} / type=poc_receipts_v1 hash="1AF7S3jW7Z8QsLug5bCK5FUockPPgsF2bMtvvevwNzGTdEjfya" challenger="witty-crepe-dinosaur" onion="12D3BqGZyLKSHjXZT9Nip5YzbyjCfjxCP3fC6JpGZrK6sTVi9kh" path:
    type=element challengee: fit-wintergreen-liger, receipt: type=receipt gateway: fit-wintergreen-liger timestamp: 1628457592356942365 signal: 0 origin: p2p
        witnesses:
2021-08-09 03:26:59.430 [warning] <0.5358.0>@blockchain_txn:separate_res:373 invalid txn blockchain_txn_poc_receipts_v1 : {error,not_found} / type=poc_receipts_v1 hash="12HvHstgbMdMjC147SGFHWcH4tHeKNzpJrWQhnz1NUxWopxfnE4" challenger="polite-denim-shrimp" onion="12CooUWkRWtZNcLLwNzZc1hy2JBA8AoMYwcrfyCYoMF8BVNWc97" path:
    type=element challengee: itchy-grey-kangaroo, receipt: type=receipt gateway: itchy-grey-kangaroo timestamp: 1628457648593941418 signal: 0 origin: p2p
        witnesses: type=witness gateway: "bent-azure-grasshopper" timestamp: 1628457652159609345 signal: -105
        type=witness gateway: "flaky-coconut-marmot" timestamp: 1628457652153093237 signal: -112
        type=witness gateway: "damp-ocean-sardine" timestamp: 1628457651975577428 signal: -111
        type=witness gateway: "little-lemon-lynx" timestamp: 1628457652163997396 signal: -117
        type=witness gateway: "gentle-vermilion-gerbil" timestamp: 1628457652198346782 signal: -107
        type=witness gateway: "hollow-sand-crab" timestamp: 1628457652164691607 signal: -108
        type=witness gateway: "immense-stone-badger" timestamp: 1628457650025918159 signal: -120
        type=witness gateway: "bright-cedar-salamander" timestamp: 1628457652312341119 signal: -118
        type=witness gateway: "curly-linen-mandrill" timestamp: 1628457652315383294 signal: -119
        type=witness gateway: "feisty-cinnabar-orca" timestamp: 1628457652162380567 signal: -103
        type=witness gateway: "spare-candy-kookaburra" timestamp: 1628457650311159248 signal: -120
2021-08-09 03:26:59.430 [warning] <0.5358.0>@blockchain_txn:separate_res:373 invalid txn blockchain_txn_poc_receipts_v1 : {error,not_found} / type=poc_receipts_v1 hash="126NE9YcnVxxxvYqT2NN1YdtjK3wsNDatM5EvhGvtmZy4R8FZ64" challenger="cool-magenta-rabbit" onion="1A1qNYziE1qCqrMz6ukA9QoohJp7QkNQeKoWmjUk61i1juNPfM" path:
    type=element challengee: short-sky-reindeer, receipt: type=receipt gateway: short-sky-reindeer timestamp: 1628457565670039216 signal: 0 origin: p2p
        witnesses: type=witness gateway: "brilliant-bamboo-meerkat" timestamp: 1628457569242510550 signal: -101
        type=witness gateway: "narrow-fern-lemur" timestamp: 1628457569237289868 signal: -115
        type=witness gateway: "brisk-bubblegum-sparrow" timestamp: 1628457569244290721 signal: -111
        type=witness gateway: "orbiting-currant-mockingbird" timestamp: 1628457569240591177 signal: -110
        type=witness gateway: "careful-inky-scallop" timestamp: 1628457567326945666 signal: -120
        type=witness gateway: "sticky-coconut-elk" timestamp: 1628457567402982713 signal: -120
        type=witness gateway: "rough-linen-chimpanzee" timestamp: 1628457569246207710 signal: -114
        type=witness gateway: "dandy-lavender-dinosaur" timestamp: 1628457569237064389 signal: -113
2021-08-09 03:26:59.431 [warning] <0.5358.0>@blockchain_txn:separate_res:373 invalid txn blockchain_txn_poc_receipts_v1 : {error,not_found} / type=poc_receipts_v1 hash="12SqdvQsfhBhwsz2YYDsiR8yzRTkAEeCEot9FHyWRM9zC2h37uh" challenger="elegant-vermilion-goldfish" onion="12ZEX19Cpcsg6yFojFPxPEujJ2ssvVotM7aLUP3kRRocEG4HGem" path:
    type=element challengee: proper-heather-carp, receipt: type=receipt gateway: proper-heather-carp timestamp: 1628457600431614650 signal: 0 origin: p2p
        witnesses:
2021-08-09 03:26:59.431 [warning] <0.5358.0>@blockchain_txn:separate_res:373 invalid txn blockchain_txn_poc_receipts_v1 : {error,not_found} / type=poc_receipts_v1 hash="1Vsxsc1FoA28V1L9WexZ2ftztpndUKFrppkiUFTybYg3sAVitj" challenger="fit-strawberry-goose" onion="1xCFyeD1oUgE8553Aoh95yD9tjqZRJrZuFKhu6oh4WnENnJs3t" path:
    type=element challengee: plain-fossilized-quail, receipt: type=receipt gateway: plain-fossilized-quail timestamp: 1628457418325660678 signal: 0 origin: p2p
        witnesses: type=witness gateway: "refined-violet-puppy" timestamp: 1628457424128296521 signal: -93
        type=witness gateway: "fluffy-laurel-goat" timestamp: 1628457422074380014 signal: -115
        type=witness gateway: "little-tweed-frog" timestamp: 1628457424115489317 signal: -118
2021-08-09 03:26:59.431 [warning] <0.5358.0>@blockchain_txn:separate_res:373 invalid txn blockchain_txn_poc_receipts_v1 : {error,not_found} / type=poc_receipts_v1 hash="1rhodWRCm1bMvPLeSMjG6Hc6RUBQtC5e6LYn4T7owAcniAhdKs" challenger="smooth-powder-cheetah" onion="1X2eibPMAiJ5vpd72awvPgTQzWd5NkMVdRDMU8Rdy9Snr9enYn" path:
    type=element challengee: innocent-alabaster-squirrel, receipt: type=receipt gateway: innocent-alabaster-squirrel timestamp: 1628457427930695267 signal: 0 origin: p2p
        witnesses:
2021-08-09 03:26:59.432 [warning] <0.5358.0>@blockchain_txn:separate_res:373 invalid txn blockchain_txn_poc_receipts_v1 : {error,not_found} / type=poc_receipts_v1 hash="1kYxWhRxxpsZmqU5uxNuLGTej4oeqL58MC3VergZUe3iCerwnW" challenger="obedient-umber-sardine" onion="12NUi1GYAWS9eJf1spXRHvhwLQi33bBitM8LN87Tyh5SkmqBxpF" path:
    type=element challengee: fancy-tangelo-dachshund, receipt: type=receipt gateway: fancy-tangelo-dachshund timestamp: 1628457570668043284 signal: 0 origin: p2p
        witnesses: type=witness gateway: "ripe-honeysuckle-pangolin" timestamp: 1628457574259197917 signal: -115
        type=witness gateway: "brilliant-ivory-antelope" timestamp: 1628457574261472998 signal: -111
        type=witness gateway: "long-metal-gibbon" timestamp: 1628457574259563944 signal: -115
        type=witness gateway: "muscular-satin-rabbit" timestamp: 1628457574255551532 signal: -79
        type=witness gateway: "prehistoric-berry-chimpanzee" timestamp: 1628457574259034337 signal: -100
        type=witness gateway: "warm-citron-beetle" timestamp: 1628457574265585514 signal: -113
        type=witness gateway: "main-sky-penguin" timestamp: 1628457574297757989 signal: -106
        type=witness gateway: "shaggy-emerald-sloth" timestamp: 1628457574263026669 signal: -104
2021-08-09 03:26:59.433 [warning] <0.5358.0>@blockchain_txn:separate_res:373 invalid txn blockchain_txn_poc_receipts_v1 : {error,not_found} / type=poc_receipts_v1 hash="1Yf5jJ8hxeEuACK3kBXnFsnRNtX79LEpxPrDDXxoo3taMDaHRU" challenger="salty-zinc-lizard" onion="12mbHiuHQbtqd26E89KfBcy85vecRBGNiwmQ9PhtmhA3bFNcreJ" path:
    type=element challengee: curved-burgundy-hedgehog, receipt: type=receipt gateway: curved-burgundy-hedgehog timestamp: 1628457505930160970 signal: 0 origin: p2p
        witnesses: type=witness gateway: "tall-aqua-sloth" timestamp: 1628457507857585295 signal: -101
2021-08-09 03:26:59.433 [warning] <0.5358.0>@blockchain_txn:separate_res:373 invalid txn blockchain_txn_poc_receipts_v1 : {error,not_found} / type=poc_receipts_v1 hash="12uJ4sZa9PT1DhJAut6Z3L17t7BnxJuXaugLJBTUtiLUvBGSycS" challenger="radiant-mango-alpaca" onion="1LDUPRzBTBibUgeoXy6tbF7aPuj4gjbaHwvmQQTVfnnz6mdwvE" path:
    type=element challengee: spicy-carbon-corgi, receipt: type=receipt gateway: spicy-carbon-corgi timestamp: 1628457577342141505 signal: 0 origin: p2p
        witnesses:
2021-08-09 03:26:59.434 [warning] <0.5358.0>@blockchain_txn:separate_res:373 invalid txn blockchain_txn_poc_receipts_v1 : {error,not_found} / type=poc_receipts_v1 hash="127gqEgsaanhKy1AT8chGcVe7qJdw4LhSCr8GgjEGo8x9pMsmiC" challenger="cheesy-tangelo-stork" onion="12ozi6QdKfF4Wstup655EiaQw14mVsbpsbA764VXUjMACV8SkS2" path:
    type=element challengee: mean-rose-trout, receipt: type=receipt gateway: mean-rose-trout timestamp: 1628457427538632774 signal: 0 origin: p2p
        witnesses: type=witness gateway: "square-plum-spider" timestamp: 1628457431095183633 signal: -116
        type=witness gateway: "acrobatic-tawny-lobster" timestamp: 1628457431102615906 signal: -115
        type=witness gateway: "hot-denim-dolphin" timestamp: 1628457431095123081 signal: -116
        type=witness gateway: "tame-shamrock-baboon" timestamp: 1628457431112572763 signal: -115
        type=witness gateway: "furry-denim-starfish" timestamp: 1628457431110932697 signal: -111
        type=witness gateway: "quaint-seaweed-salmon" timestamp: 1628457431098218858 signal: -102
        type=witness gateway: "proud-cornflower-worm" timestamp: 1628457431098753664 signal: -103
        type=witness gateway: "shallow-lace-alpaca" timestamp: 1628457431122436473 signal: -111
        type=witness gateway: "rhythmic-gingerbread-elephant" timestamp: 1628457431109684282 signal: -115
        type=witness gateway: "rare-brown-manatee" timestamp: 1628457431103327380 signal: -104
2021-08-09 03:26:59.434 [warning] <0.5358.0>@blockchain_txn:separate_res:373 invalid txn blockchain_txn_poc_receipts_v1 : {error,not_found} / type=poc_receipts_v1 hash="12hUe948Jyv8tc2yqSrNk3EFAhW5i2pjW1UPGSwUYhKsjALkrKE" challenger="harsh-coral-capybara" onion="12LzsK5F1K9A9f6ooTRj6r4PmPiR4EuenUisxorzPmAxzFyWAHa" path:
    type=element challengee: custom-blush-sparrow, receipt: type=receipt gateway: custom-blush-sparrow timestamp: 1628457478543663271 signal: 0 origin: p2p
        witnesses:
2021-08-09 03:26:59.434 [warning] <0.5358.0>@blockchain_txn:separate_res:373 invalid txn blockchain_txn_poc_receipts_v1 : {error,not_found} / type=poc_receipts_v1 hash="1f9oDmm6HafUSDYUArcYBP2T5NmDXLUjpMTfA9PWmkZpT7pu8y" challenger="winning-tawny-panda" onion="1rhcz3PJCQv1ihxKh45MPExRvTYAbhWdYRoGyb1HvVW6oAEmc7" path:
    type=element challengee: rural-ultraviolet-mole, receipt: type=receipt gateway: rural-ultraviolet-mole timestamp: 1628457559303150534 signal: 0 origin: p2p
        witnesses: type=witness gateway: "merry-hickory-haddock" timestamp: 1628457560718698861 signal: -95
        type=witness gateway: "itchy-bamboo-snail" timestamp: 1628457560715073324 signal: -104
        type=witness gateway: "striped-latte-swan" timestamp: 1628457560708266177 signal: -103
2021-08-09 03:26:59.436 [warning] <0.5358.0>@blockchain_txn:separate_res:373 invalid txn blockchain_txn_poc_receipts_v1 : {error,not_found} / type=poc_receipts_v1 hash="1doPTcxvVK9JtgD2mJSBmS6pd3SU5TKWk7AwfYGpkSFhfa1RwX" challenger="cheerful-stone-nuthatch" onion="127KQ6xARywxyXzVSEitEKvLXL73yktBK7zjsFTBZqqtMWQz48K" path:
    type=element challengee: unique-paisley-seagull, receipt: type=receipt gateway: unique-paisley-seagull timestamp: 1628457626333292362 signal: 0 origin: p2p
        witnesses: type=witness gateway: "cool-latte-barracuda" timestamp: 1628457632226530688 signal: -91
        type=witness gateway: "blunt-taupe-haddock" timestamp: 1628457632233375770 signal: -117
        type=witness gateway: "gigantic-lava-cow" timestamp: 1628457632237632426 signal: -118
        type=witness gateway: "wonderful-jetblack-elephant" timestamp: 1628457632233529611 signal: -117
        type=witness gateway: "nice-peach-hippo" timestamp: 1628457632241947338 signal: -113
        type=witness gateway: "muscular-garnet-boar" timestamp: 1628457632233718135 signal: -117
        type=witness gateway: "proud-lace-horse" timestamp: 1628457632228647071 signal: -119
        type=witness gateway: "fierce-ebony-gecko" timestamp: 1628457632242705201 signal: -113
        type=witness gateway: "scrawny-tartan-hyena" timestamp: 1628457632239560437 signal: -108
        type=witness gateway: "chilly-cotton-panda" timestamp: 1628457632248882009 signal: -115
        type=witness gateway: "magic-cinnamon-beetle" timestamp: 1628457632375730922 signal: -115
        type=witness gateway: "short-nylon-seal" timestamp: 1628457632226317295 signal: -108
        type=witness gateway: "crazy-smoke-goblin" timestamp: 1628457632237507192 signal: -118
        type=witness gateway: "brief-flaxen-sparrow" timestamp: 1628457632235259867 signal: -98
2021-08-09 03:26:59.437 [warning] <0.5358.0>@blockchain_txn:separate_res:373 invalid txn blockchain_txn_poc_receipts_v1 : {error,not_found} / type=poc_receipts_v1 hash="12Svs8GZxC4hj9m6owVWbRBCrKkyC7jsReCn3hR2rMjsURDbede" challenger="odd-lavender-rat" onion="1FQ2KrWSPds6KppAo8QujJYuKK6ybZjqj7WXTEx48THxRYoZpQ" path:
    type=element challengee: droll-leather-moth, receipt: type=receipt gateway: droll-leather-moth timestamp: 1628457679153524813 signal: 0 origin: p2p
        witnesses: type=witness gateway: "colossal-coral-dolphin" timestamp: 1628457682779666021 signal: -104
        type=witness gateway: "ancient-syrup-pike" timestamp: 1628457680830185967 signal: -118
        type=witness gateway: "energetic-felt-jaguar" timestamp: 1628457682743112477 signal: -80
        type=witness gateway: "funny-blood-deer" timestamp: 1628457682750905761 signal: -112
        type=witness gateway: "faithful-iron-wasp" timestamp: 1628457682759710165 signal: -104
        type=witness gateway: "cheerful-denim-coyote" timestamp: 1628457680504234179 signal: -120
        type=witness gateway: "bent-mint-cormorant" timestamp: 1628457682747541054 signal: -115
        type=witness gateway: "careful-ebony-elk" timestamp: 1628457682737256565 signal: -117
        type=witness gateway: "stable-wool-dinosaur" timestamp: 1628457680962711842 signal: -99
        type=witness gateway: "big-parchment-skunk" timestamp: 1628457680760280095 signal: -120
        type=witness gateway: "digital-opaque-shark" timestamp: 1628457682751924345 signal: -117
        type=witness gateway: "short-blush-dalmatian" timestamp: 1628457680794960804 signal: -119
        type=witness gateway: "custom-honey-octopus" timestamp: 1628457682738076394 signal: -114
        type=witness gateway: "amateur-mossy-python" timestamp: 1628457682757153101 signal: -119
2021-08-09 03:26:59.437 [warning] <0.5358.0>@blockchain_txn:separate_res:373 invalid txn blockchain_txn_poc_receipts_v1 : {error,not_found} / type=poc_receipts_v1 hash="1p2fYMzrwvBEPZUa4toYVniUrErr8ZY89aZTCttKfv4tw82o3P" challenger="macho-beige-wolf" onion="123bGrGByMmKjUh4L2hDv63ie2aBM6gthDEsv4ATL8xbj1XfNaq" path:
    type=element challengee: perfect-currant-grasshopper, receipt: type=receipt gateway: perfect-currant-grasshopper timestamp: 1628457476649602858 signal: 0 origin: p2p
        witnesses:
2021-08-09 03:26:59.437 [warning] <0.5358.0>@blockchain_txn:separate_res:373 invalid txn blockchain_txn_poc_receipts_v1 : {error,not_found} / type=poc_receipts_v1 hash="1CBAFySUnweYGmDRMfm4DaQcQTYDmxUkh9T3d34qhGXmnWpSj7" challenger="cool-bronze-pony" onion="12DLVb3VjLYinLsEANXqgk5kxcBvqkSLHhf5M2xfJc3gJbEiZv3" path:
    type=element challengee: blurry-indigo-quail, receipt: type=receipt gateway: blurry-indigo-quail timestamp: 1628457611191363089 signal: 0 origin: p2p
        witnesses:
2021-08-09 03:26:59.438 [warning] <0.5358.0>@blockchain_txn:separate_res:373 invalid txn blockchain_txn_poc_receipts_v1 : {error,not_found} / type=poc_receipts_v1 hash="12FiXEYSKpct5BXR5PFtTkPNt5RazW1FAfVXayfJXYLxp4F2456" challenger="dizzy-cerulean-opossum" onion="1znURuqiKtqiitjdFa8xsuDyQtxE4oznzinc4kkq1yUQk8S8ZH" path:
    type=element challengee: suave-blue-trout, receipt: type=receipt gateway: suave-blue-trout timestamp: 1628457502422859302 signal: 0 origin: p2p
        witnesses:
2021-08-09 03:26:59.439 [warning] <0.5358.0>@blockchain_txn:separate_res:373 invalid txn blockchain_txn_poc_receipts_v1 : {error,not_found} / type=poc_receipts_v1 hash="1XHCqPMW6pKija6nDjtixTZD96JT5jFTCb8hTT4SEjf4DYu4L3" challenger="steep-crimson-leopard" onion="1nuJwyWNVF3QFUrEQE64TcP4ccAfwdnEAW81JKvQBxXibU7bBz" path:
    type=element challengee: innocent-burgundy-sparrow, receipt: type=receipt gateway: innocent-burgundy-sparrow timestamp: 1628457633862780863 signal: 0 origin: p2p
        witnesses: type=witness gateway: "dazzling-saffron-poodle" timestamp: 1628457635280992876 signal: -110
        type=witness gateway: "fit-parchment-sardine" timestamp: 1628457635215230021 signal: -102
        type=witness gateway: "wobbly-eggshell-caterpillar" timestamp: 1628457635277267833 signal: -81
        type=witness gateway: "square-slate-bison" timestamp: 1628457635276320511 signal: -96
        type=witness gateway: "attractive-silver-armadillo" timestamp: 1628457635271183706 signal: -77
        type=witness gateway: "savory-candy-shetland" timestamp: 1628457635280302708 signal: -108
        type=witness gateway: "blurry-oily-ape" timestamp: 1628457633771799954 signal: -90
        type=witness gateway: "cheery-amethyst-hare" timestamp: 1628457635274559803 signal: -111
        type=witness gateway: "spare-iron-cobra" timestamp: 1628457635272146150 signal: -113
        type=witness gateway: "stable-pebble-dragonfly" timestamp: 1628457635215960079 signal: -92
        type=witness gateway: "generous-yellow-elk" timestamp: 1628457635199712499 signal: -111
        type=witness gateway: "tiny-ivory-sloth" timestamp: 1628457635209399750 signal: -110
        type=witness gateway: "lively-raisin-dachshund" timestamp: 1628457635274596456 signal: -113
        type=witness gateway: "jolly-ceramic-worm" timestamp: 1628457635331465701 signal: -92
        type=witness gateway: "long-rouge-hyena" timestamp: 1628457635272135567 signal: -98
        type=witness gateway: "vast-wintergreen-donkey" timestamp: 1628457635345551286 signal: -115
        type=witness gateway: "mini-dijon-dolphin" timestamp: 1628457635273872588 signal: -104
        type=witness gateway: "mean-white-fox" timestamp: 1628457635276897417 signal: -100
        type=witness gateway: "chilly-mint-starfish" timestamp: 1628457635279008089 signal: -115
        type=witness gateway: "clever-vinyl-cod" timestamp: 1628457635320829599 signal: -118
        type=witness gateway: "wide-pink-weasel" timestamp: 1628457635279208408 signal: -120
        type=witness gateway: "bent-paisley-zebra" timestamp: 1628457635206128362 signal: -105
2021-08-09 03:26:59.440 [warning] <0.5358.0>@blockchain_txn:separate_res:373 invalid txn blockchain_txn_poc_receipts_v1 : {error,not_found} / type=poc_receipts_v1 hash="12TeS5PZLsaqSbJXkUyfK8QUBBExfDrgED5mg2LEdSv3u8XMt37" challenger="suave-berry-squid" onion="1czQ2Ei8nNvTxXcoEX4nYsC2EbgoDzToF79aJADoY9Qtx9vdBB" path:
    type=element challengee: keen-macaroon-hedgehog, receipt: type=receipt gateway: keen-macaroon-hedgehog timestamp: 1628457492627330599 signal: 0 origin: p2p
        witnesses: type=witness gateway: "uneven-glass-bat" timestamp: 1628457495606379238 signal: -115
        type=witness gateway: "refined-rose-pigeon" timestamp: 1628457494141552132 signal: -118
2021-08-09 03:26:59.440 [warning] <0.5358.0>@blockchain_txn:separate_res:373 invalid txn blockchain_txn_poc_receipts_v1 : {error,not_found} / type=poc_receipts_v1 hash="12h62siqX1qYhSzVsPUJ9xUaQZ28SvuzNdKnNLRAeXT7fyENvXi" challenger="skinny-crepe-gazelle" onion="12siH8XEqtTeMvCiaVVPfJTE5pgpnYBsHoAtE42FoYPTJu26iHX" path:
    type=element challengee: skinny-gunmetal-tuna, receipt: type=receipt gateway: skinny-gunmetal-tuna timestamp: 1628457556368638857 signal: 0 origin: p2p
        witnesses:
2021-08-09 03:26:59.440 [warning] <0.5358.0>@blockchain_txn:separate_res:373 invalid txn blockchain_txn_poc_receipts_v1 : {error,not_found} / type=poc_receipts_v1 hash="121R8FXqprdJa1H568BuSGMK2LNnZLtaALz2iZPHQ7b6HZUPuhA" challenger="abundant-paisley-cottonmouth" onion="12UQiQw25MB1nrJcQuoMP65Mei7z8AsWNGDf2DZqXDMZp6Ki9jq" path:
    type=element challengee: early-blonde-wombat, receipt: type=receipt gateway: early-blonde-wombat timestamp: 1628457651219895338 signal: 0 origin: p2p
        witnesses:
2021-08-09 03:26:59.441 [warning] <0.5358.0>@blockchain_txn:separate_res:373 invalid txn blockchain_txn_poc_receipts_v1 : {error,not_found} / type=poc_receipts_v1 hash="12MKT6nyLjQj2w2nVpEK4LwTZFomdMqSnP3Jjte2j2Acdy29sku" challenger="blunt-gunmetal-eel" onion="1mSovfvCVqffXbDVTZ9Sa5TuZKX3AzP8UzKqSADKH5GXJ4uJYP" path:
    type=element challengee: cheesy-pastel-raven, receipt: type=receipt gateway: cheesy-pastel-raven timestamp: 1628457667706125991 signal: 0 origin: p2p
        witnesses:
2021-08-09 03:26:59.441 [warning] <0.5358.0>@blockchain_txn:separate_res:373 invalid txn blockchain_txn_poc_receipts_v1 : {error,not_found} / type=poc_receipts_v1 hash="12P6vJXHicgKbez4fXW1f7wYTxxKLXNEUJGaJZQabn5ckatbR1L" challenger="mythical-peach-cat" onion="1zLtf3BYqXrk1attr5YSNC44NPihZEgG2QRydCVSAwBQy4bWCJ" path:
    type=element challengee: plain-sky-pangolin, receipt: type=receipt gateway: plain-sky-pangolin timestamp: 1628457630016603046 signal: 0 origin: p2p
        witnesses: type=witness gateway: "quaint-seaweed-urchin" timestamp: 1628457630104644053 signal: -111
        type=witness gateway: "virtual-crepe-penguin" timestamp: 1628457630110271815 signal: -113
        type=witness gateway: "narrow-obsidian-giraffe" timestamp: 1628457630112199126 signal: -110
        type=witness gateway: "custom-aqua-bear" timestamp: 1628457630112203201 signal: -114
        type=witness gateway: "puny-teal-hawk" timestamp: 1628457630107100208 signal: -111
        type=witness gateway: "cheerful-mercurial-jay" timestamp: 1628457630105638891 signal: -113
2021-08-09 03:26:59.441 [warning] <0.5358.0>@blockchain_txn:separate_res:373 invalid txn blockchain_txn_poc_receipts_v1 : {error,not_found} / type=poc_receipts_v1 hash="1yhp6391mkn7xcYbcmXySNo9rPtAmL7wpf5z7Nf2jXCKikYoaC" challenger="muscular-lemonade-lark" onion="12jcnAGREJQ4iCRNnZucjM2k3BPEzKLRPUjqTS6EGXqiSJuVWzn" path:
    type=element challengee: large-obsidian-corgi, receipt: type=receipt gateway: large-obsidian-corgi timestamp: 1628457283994027512 signal: 0 origin: p2p
        witnesses:
2021-08-09 03:26:59.442 [warning] <0.5358.0>@blockchain_txn:separate_res:373 invalid txn blockchain_txn_poc_receipts_v1 : {error,not_found} / type=poc_receipts_v1 hash="18BE7tCY5K4t8JKLxWjabDVReQLURDFTHKGET46KerxRZNyvwT" challenger="brilliant-honeysuckle-ram" onion="1yYvVWt36vMJWUYmMAaUkRwCwKStqxStVtzXedvgis9vZ1y5TX" path:
    type=element challengee: rough-slate-goose, receipt: type=receipt gateway: rough-slate-goose timestamp: 1628457552229741851 signal: 0 origin: p2p
        witnesses:
2021-08-09 03:26:59.442 [warning] <0.5358.0>@blockchain_txn:separate_res:373 invalid txn blockchain_txn_poc_receipts_v1 : {error,not_found} / type=poc_receipts_v1 hash="12AReorroh23hpwcDNdj6iDVvLMgdUEZxxgTctiw7ChuNCUvucD" challenger="droll-denim-duck" onion="1PWr9WJ2TzAPgoArJpvjA1iqpQ3kodgpLuA4qzN2HiDLprudGG" path:
    type=element challengee: square-mango-dove, receipt: type=receipt gateway: square-mango-dove timestamp: 1628457641078317830 signal: 0 origin: p2p
        witnesses: type=witness gateway: "tangy-lime-donkey" timestamp: 1628457640317260451 signal: -115
        type=witness gateway: "small-myrtle-piranha" timestamp: 1628457642415787603 signal: -105
2021-08-09 03:26:59.442 [warning] <0.5358.0>@blockchain_txn:separate_res:373 invalid txn blockchain_txn_poc_receipts_v1 : {error,not_found} / type=poc_receipts_v1 hash="1TBfuhKewq9FrNAc3BU4e5MnTo9oWnCxU5e9h3MQ1B4ZxuesPg" challenger="orbiting-pecan-mole" onion="12YHpuGgRerexJ5j8H46gH2wji19bVSQ1cYdxjNSxX8Fn6ihv7y" path:
    type=element challengee: active-tawny-cougar, receipt: type=receipt gateway: active-tawny-cougar timestamp: 1628457082878334944 signal: 0 origin: p2p
        witnesses:
2021-08-09 03:26:59.443 [warning] <0.5358.0>@blockchain_txn:separate_res:373 invalid txn blockchain_txn_poc_receipts_v1 : {error,not_found} / type=poc_receipts_v1 hash="12DAh1M2QgU3dyqMRPXTfYLfqfeJzMRPiuztBnuBDBqnyAFKCd2" challenger="alert-daffodil-iguana" onion="122rqD7PLG4UJDu4ut2KCeymDy1rcykMSrrFtBviRHN8VfDys2E" path:
    type=element challengee: winning-rainbow-cricket, receipt: type=receipt gateway: winning-rainbow-cricket timestamp: 1628457485260925127 signal: 0 origin: p2p
        witnesses: type=witness gateway: "feisty-honeysuckle-hippo" timestamp: 1628457489177190470 signal: -106
        type=witness gateway: "helpful-pewter-rat" timestamp: 1628457489169872397 signal: -108
2021-08-09 03:26:59.444 [warning] <0.5358.0>@blockchain_txn:separate_res:373 invalid txn blockchain_txn_poc_receipts_v1 : {error,not_found} / type=poc_receipts_v1 hash="12rpUANTQKakfapdfm36BNBervgkkRzYRqEj8jceaGt8WEWXVmb" challenger="cool-carbon-tadpole" onion="12LAaszdtpxMSLRYagFScMhryLWsGp1fv5Ys3kjYNQqHRUZMEpC" path:
    type=element challengee: prehistoric-mint-hawk, receipt: type=receipt gateway: prehistoric-mint-hawk timestamp: 1628457566598813299 signal: 0 origin: p2p
        witnesses: type=witness gateway: "damp-aquamarine-tiger" timestamp: 1628457567999738202 signal: -105
        type=witness gateway: "basic-metal-loris" timestamp: 1628457568008689994 signal: -64
        type=witness gateway: "dazzling-porcelain-spider" timestamp: 1628457568008087764 signal: -113
        type=witness gateway: "itchy-graphite-tiger" timestamp: 1628457567928835968 signal: -109
        type=witness gateway: "melodic-menthol-fish" timestamp: 1628457565730368208 signal: -113
        type=witness gateway: "curved-violet-spider" timestamp: 1628457568002784840 signal: -113
        type=witness gateway: "blunt-pink-caribou" timestamp: 1628457568003935894 signal: -104
        type=witness gateway: "breezy-goldenrod-pelican" timestamp: 1628457568011148580 signal: -97
        type=witness gateway: "shaggy-opal-piranha" timestamp: 1628457568013563856 signal: -112
        type=witness gateway: "curved-burgundy-sparrow" timestamp: 1628457568011495672 signal: -108
        type=witness gateway: "rapid-wool-moth" timestamp: 1628457568285808099 signal: -91
        type=witness gateway: "odd-burlap-dragon" timestamp: 1628457568023440656 signal: -107
        type=witness gateway: "calm-jetblack-deer" timestamp: 1628457568009237157 signal: -88
2021-08-09 03:26:59.444 [warning] <0.5358.0>@blockchain_txn:separate_res:373 invalid txn blockchain_txn_poc_receipts_v1 : {error,not_found} / type=poc_receipts_v1 hash="1Rm6srTcJDuuGnmW5bmfdvxcyuPqzzx5M44L18PMGSyTE1S688" challenger="melted-shadow-mouse" onion="1Frqr5s7Y441pRH8pxycyLU6q3kEW5xp7BLyUf9tcNgQrHneeF" path:
    type=element challengee: skinny-lime-wolf, receipt: type=receipt gateway: skinny-lime-wolf timestamp: 1628457501366206189 signal: 0 origin: p2p
        witnesses: type=witness gateway: "polished-pickle-cottonmouth" timestamp: 1628457500498709051 signal: -114
        type=witness gateway: "spicy-pearl-cobra" timestamp: 1628457503338987212 signal: -118
2021-08-09 03:26:59.445 [warning] <0.5358.0>@blockchain_txn:separate_res:373 invalid txn blockchain_txn_poc_receipts_v1 : {error,not_found} / type=poc_receipts_v1 hash="1ayQ2aV7hRJ8VCmNHKg5HfuJZFVdNesFQiv7NQQrHhvojBAB29" challenger="jolly-tangelo-stallion" onion="1hjXMEEV74JzXhYGeKPLPQZkrWfVKaHLjYTixDhGAZMZujJVJ7" path:
    type=element challengee: brief-teal-tuna, receipt: type=receipt gateway: brief-teal-tuna timestamp: 1628457589214287351 signal: 0 origin: p2p
        witnesses: type=witness gateway: "silly-crimson-shell" timestamp: 1628457594342891952 signal: -109
        type=witness gateway: "sharp-quartz-wren" timestamp: 1628457592787007960 signal: -112
        type=witness gateway: "joyful-tartan-anteater" timestamp: 1628457594518133944 signal: -107
2021-08-09 03:26:59.446 [warning] <0.5358.0>@blockchain_txn:separate_res:373 invalid txn blockchain_txn_poc_receipts_v1 : {error,not_found} / type=poc_receipts_v1 hash="1trMUFqKstaEnTLCujonvPJ3CiMmk8pRbZ2ueGcjmiuUWs9ZAh" challenger="innocent-clear-raccoon" onion="12bRfQSrp2FsrT5jPdQqETit1hFxtTH5ttaYB2TEE7F5TRBiXZK" path:
    type=element challengee: best-burgundy-octopus, receipt: type=receipt gateway: best-burgundy-octopus timestamp: 1628457725622736243 signal: 0 origin: p2p
        witnesses: type=witness gateway: "keen-watermelon-wolf" timestamp: 1628457724762947316 signal: -109
        type=witness gateway: "cheery-scarlet-spider" timestamp: 1628457727804129405 signal: -99
        type=witness gateway: "thankful-ginger-kestrel" timestamp: 1628457725872632575 signal: -97
        type=witness gateway: "bald-basil-owl" timestamp: 1628457727283534784 signal: -108
        type=witness gateway: "narrow-cider-mallard" timestamp: 1628457726864904742 signal: -110
        type=witness gateway: "sticky-cobalt-shark" timestamp: 1628457727100706743 signal: -103
        type=witness gateway: "ancient-mint-raven" timestamp: 1628457727131726805 signal: -103
        type=witness gateway: "damaged-stone-turtle" timestamp: 1628457726803884368 signal: -108
        type=witness gateway: "pet-magenta-jay" timestamp: 1628457725129776678 signal: -108
        type=witness gateway: "hot-watermelon-troll" timestamp: 1628457727025731993 signal: -109
        type=witness gateway: "custom-tiger-fly" timestamp: 1628457726852755281 signal: -97
        type=witness gateway: "modern-vinyl-squid" timestamp: 1628457726860543159 signal: -96
        type=witness gateway: "winning-goldenrod-boar" timestamp: 1628457727425190993 signal: -105
        type=witness gateway: "broad-glossy-millipede" timestamp: 1628457724754755940 signal: -97
        type=witness gateway: "low-coconut-opossum" timestamp: 1628457725509469534 signal: -106
        type=witness gateway: "active-taffy-canary" timestamp: 1628457726947500451 signal: -111
        type=witness gateway: "basic-myrtle-duck" timestamp: 1628457726630404722 signal: -101
        type=witness gateway: "short-walnut-alpaca" timestamp: 1628457728195040049 signal: -103
        type=witness gateway: "shambolic-mossy-fox" timestamp: 1628457725963226834 signal: -93
        type=witness gateway: "massive-lemon-sawfish" timestamp: 1628457726858748618 signal: -94
        type=witness gateway: "powerful-bone-mallard" timestamp: 1628457726336952395 signal: -105
2021-08-09 03:26:59.447 [warning] <0.5358.0>@blockchain_txn:separate_res:373 invalid txn blockchain_txn_poc_receipts_v1 : {error,not_found} / type=poc_receipts_v1 hash="1UYouAPcB6hBYLPuiLWFkR3Jbq8GkEAvsAyDqsaky4CP1b3KL2" challenger="shambolic-pickle-sloth" onion="123DktMkn68ZepJVFq1QBLJLTrZvvB3E83DeRG68WXiR2HCTFT5" path:
    type=element challengee: silly-magenta-quail, receipt: type=receipt gateway: silly-magenta-quail timestamp: 1628457574741858004 signal: 0 origin: p2p
        witnesses: type=witness gateway: "cheery-pear-seahorse" timestamp: 1628457576395352879 signal: -120
        type=witness gateway: "funny-lemon-capybara" timestamp: 1628457578145491526 signal: -110
2021-08-09 03:26:59.447 [warning] <0.5358.0>@blockchain_txn:separate_res:373 invalid txn blockchain_txn_poc_receipts_v1 : {error,not_found} / type=poc_receipts_v1 hash="129zAxag5Pc5re6BRdsh9xQMYhMX32kFhFwHcYXXpXgS7vniih5" challenger="fantastic-eggshell-crocodile" onion="12HUH1QedT5d3y9FjZtthpWPQJLAC7QwxuqCRNunsHUsBHkTeB4" path:
    type=element challengee: trendy-taffy-rabbit, receipt: type=receipt gateway: trendy-taffy-rabbit timestamp: 1628457565289842249 signal: 0 origin: p2p
        witnesses: type=witness gateway: "fit-shamrock-chinchilla" timestamp: 1628457567203382792 signal: -109
        type=witness gateway: "harsh-hotpink-tuna" timestamp: 1628457567091504792 signal: -109
        type=witness gateway: "late-tiger-orangutan" timestamp: 1628457567004124667 signal: -108
        type=witness gateway: "zany-white-wombat" timestamp: 1628457567063946292 signal: -107
        type=witness gateway: "sneaky-lace-tadpole" timestamp: 1628457567005579375 signal: -109
2021-08-09 03:26:59.448 [warning] <0.5358.0>@blockchain_txn:separate_res:373 invalid txn blockchain_txn_poc_receipts_v1 : {error,not_found} / type=poc_receipts_v1 hash="12g46kADRhW95eH7Kz1dGyKoFxfVsZepeHzgH3LxseCiEhNcvpJ" challenger="wide-pine-baboon" onion="12AhNZvmTKQrriNwobFr5EWYoYy6PdX1ZbKPx1xxUZAxBb6o5Zp" path:
    type=element challengee: sneaky-mercurial-meerkat, receipt: type=receipt gateway: sneaky-mercurial-meerkat timestamp: 1628457491151984586 signal: 0 origin: p2p
        witnesses: type=witness gateway: "dandy-zinc-tuna" timestamp: 1628457492573036407 signal: -107
2021-08-09 03:26:59.448 [warning] <0.5358.0>@blockchain_txn:separate_res:373 invalid txn blockchain_txn_poc_receipts_v1 : {error,not_found} / type=poc_receipts_v1 hash="12XrBfWTc13NRLynSDQKKyTsUKDRL6RaBHhPMrHC7fGuigF1A6x" challenger="shallow-clay-whale" onion="1z26kpKw67raDczw7mahAoiCgKjUoFcAjtYZy95KBa5bQ3qMA9" path:
    type=element challengee: future-iris-ferret, receipt: type=receipt gateway: future-iris-ferret timestamp: 1628457476642296392 signal: 0 origin: p2p
        witnesses:
2021-08-09 03:26:59.448 [warning] <0.5358.0>@blockchain_txn:separate_res:373 invalid txn blockchain_txn_poc_receipts_v1 : {error,not_found} / type=poc_receipts_v1 hash="12D2pQqcCYTeVrH9EMk1ksPeVeoVGvDmjPthFMLN8jMyEkbRcq9" challenger="muscular-alabaster-finch" onion="129ZMzhyfko1jtXPSHVo9WUgirtgjVbJiA7Nyc4sYYKUk3T78g1" path:
    type=element challengee: sunny-caramel-rooster, receipt: type=receipt gateway: sunny-caramel-rooster timestamp: 1628457683958315222 signal: 0 origin: p2p
        witnesses:
2021-08-09 03:26:59.521 [info] <0.5358.0>@blockchain_txn_state_channel_close_v1:is_valid:190 close state was closed
2021-08-09 03:26:59.626 [warning] <0.5358.0>@blockchain_txn:validate:327 invalid txn blockchain_txn_state_channel_close_v1 : {error,redundant} / type=state_channel_close, state_channel={blockchain_state_channel_v1_pb,
                                         <<241,96,157,164,242,195,109,198,106,
                                           111,195,237,25,240,221,95,252,8,19,
                                           171,243,153,154,134,227,197,66,13,
                                           204,169,152,191>>,
                                         <<0,122,218,88,133,243,228,11,191,188,
                                           24,106,74,213,136,24,242,41,172,56,
                                           220,24,59,236,149,26,141,119,81,208,
                                           193,237,106>>,
                                         50000,32577,
                                         [{blockchain_state_channel_summary_v1_pb,
                                           <<0,203,188,223,81,21,99,185,200,86,
                                             227,157,251,232,15,134,210,206,
                                             127,43,79,74,134,231,97,221,244,
                                             247,231,2,56,93,87>>,
                                           1,2},
                                          {blockchain_state_channel_summary_v1_pb,
                                           <<0,221,150,138,221,174,134,220,105,
                                             229,79,189,163,44,111,123,209,101,
                                             196,190,78,181,17,70,222,52,193,
                                             56,73,32,141,254,108>>,
                                           3,4},
                                          {blockchain_state_channel_summary_v1_pb,
                                           <<0,203,112,21,82,68,79,17,41,64,
                                             136,135,227,211,182,150,98,174,
                                             146,66,50,237,136,169,144,179,100,
                                             63,197,20,217,14,153>>,
                                           1,2},
                                          {blockchain_state_channel_summary_v1_pb,
                                           <<0,220,204,63,119,64,216,121,68,
                                             137,190,41,42,197,145,157,235,133,
                                             98,102,174,104,179,155,216,134,
                                             156,211,54,96,220,166,65>>,
                                           3,3},
                                          {blockchain_state_channel_summary_v1_pb,
                                           <<0,255,245,156,213,118,233,122,43,
                                             186,188,231,201,227,219,86,190,
                                             203,250,22,13,17,125,60,57,149,86,
                                             232,168,144,6,89,15>>,
                                           2,4},
                                          {blockchain_state_channel_summary_v1_pb,
                                           <<0,186,227,176,32,99,67,64,88,238,
                                             233,5,104,192,209,168,81,57,122,
                                             116,49,214,193,122,45,241,169,242,
                                             102,114,121,174,137>>,
                                           13,13},
                                          {blockchain_state_channel_summary_v1_pb,
                                           <<0,212,224,173,53,172,214,156,247,
                                             153,246,254,71,170,12,208,221,160,
                                             159,208,153,34,166,92,143,195,130,
                                             157,61,43,199,168,177>>,
                                           3,3},
                                          {blockchain_state_channel_summary_v1_pb,
                                           <<0,208,94,218,167,70,145,192,21,38,
                                             56,61,65,1...
2021-08-09 03:26:59.627 [warning] <0.5358.0>@blockchain_txn:validate:331 invalid txn blockchain_txn_validator_heartbeat_v1 : {error,{bad_height,prev,957689,height,957699,got,957689}} / type=validator_heartbeat, validator="bouncy-mahogany-owl", height=957689, version=10010004
2021-08-09 03:26:59.627 [warning] <0.5358.0>@blockchain_txn:validate:331 invalid txn blockchain_txn_validator_heartbeat_v1 : {error,{bad_height,prev,957696,height,957699,got,957696}} / type=validator_heartbeat, validator="keen-iron-ape", height=957696, version=10010004
2021-08-09 03:26:59.627 [warning] <0.5358.0>@blockchain_txn:validate:331 invalid txn blockchain_txn_validator_heartbeat_v1 : {error,{bad_height,prev,957697,height,957699,got,957697}} / type=validator_heartbeat, validator="perfect-vinyl-mammoth", height=957697, version=10010004
2021-08-09 03:26:59.628 [warning] <0.5358.0>@blockchain_txn:validate:331 invalid txn blockchain_txn_validator_heartbeat_v1 : {error,{bad_height,prev,957697,height,957699,got,957697}} / type=validator_heartbeat, validator="furry-coal-chimpanzee", height=957697, version=10010000
2021-08-09 03:26:59.628 [warning] <0.5358.0>@blockchain_txn:validate:331 invalid txn blockchain_txn_validator_heartbeat_v1 : {error,{bad_height,prev,957697,height,957699,got,957697}} / type=validator_heartbeat, validator="polished-steel-salamander", height=957697, version=10010000
2021-08-09 03:26:59.629 [warning] <0.5358.0>@blockchain_txn:validate:331 invalid txn blockchain_txn_validator_heartbeat_v1 : {error,{bad_height,prev,957698,height,957699,got,957698}} / type=validator_heartbeat, validator="acrobatic-glass-tortoise", height=957698, version=10010004
2021-08-09 03:26:59.629 [warning] <0.5358.0>@blockchain_txn:validate:331 invalid txn blockchain_txn_validator_heartbeat_v1 : {error,{bad_height,prev,957697,height,957699,got,957697}} / type=validator_heartbeat, validator="powerful-slate-locust", height=957697, version=10010004
2021-08-09 03:26:59.630 [warning] <0.5358.0>@blockchain_txn:validate:331 invalid txn blockchain_txn_validator_heartbeat_v1 : {error,{bad_height,prev,957696,height,957699,got,957696}} / type=validator_heartbeat, validator="happy-citron-buffalo", height=957696, version=10010004
2021-08-09 03:26:59.631 [warning] <0.5358.0>@blockchain_txn:validate:331 invalid txn blockchain_txn_validator_heartbeat_v1 : {error,{bad_height,prev,957698,height,957699,got,957698}} / type=validator_heartbeat, validator="jovial-mahogany-rattlesnake", height=957698, version=10010004
2021-08-09 03:26:59.632 [warning] <0.5358.0>@blockchain_txn:validate:331 invalid txn blockchain_txn_validator_heartbeat_v1 : {error,{bad_height,prev,957698,height,957699,got,957698}} / type=validator_heartbeat, validator="magnificent-magenta-wolverine", height=957698, version=10010002
2021-08-09 03:26:59.632 [warning] <0.5358.0>@blockchain_txn:validate:331 invalid txn blockchain_txn_validator_heartbeat_v1 : {error,{bad_height,prev,957695,height,957699,got,957695}} / type=validator_heartbeat, validator="tiny-chocolate-hamster", height=957695, version=10010004
2021-08-09 03:26:59.633 [warning] <0.5358.0>@blockchain_txn:validate:331 invalid txn blockchain_txn_validator_heartbeat_v1 : {error,{bad_height,prev,957696,height,957699,got,957696}} / type=validator_heartbeat, validator="digital-powder-koala", height=957696, version=10010004
2021-08-09 03:26:59.634 [warning] <0.5358.0>@blockchain_txn:validate:331 invalid txn blockchain_txn_validator_heartbeat_v1 : {error,{bad_height,prev,957697,height,957699,got,957697}} / type=validator_heartbeat, validator="soaring-tartan-newt", height=957697, version=10010004
2021-08-09 03:26:59.634 [warning] <0.5358.0>@blockchain_txn:validate:331 invalid txn blockchain_txn_validator_heartbeat_v1 : {error,{bad_height,prev,957698,height,957699,got,957698}} / type=validator_heartbeat, validator="nice-misty-octopus", height=957698, version=10010004
2021-08-09 03:26:59.635 [warning] <0.5358.0>@blockchain_txn:validate:331 invalid txn blockchain_txn_validator_heartbeat_v1 : {error,{bad_height,prev,957697,height,957699,got,957697}} / type=validator_heartbeat, validator="keen-aquamarine-flamingo", height=957697, version=10010004
2021-08-09 03:26:59.636 [warning] <0.5358.0>@blockchain_txn:validate:331 invalid txn blockchain_txn_validator_heartbeat_v1 : {error,{bad_height,prev,957697,height,957699,got,957697}} / type=validator_heartbeat, validator="bouncy-porcelain-fish", height=957697, version=10010004
2021-08-09 03:26:59.637 [warning] <0.5358.0>@blockchain_txn:validate:331 invalid txn blockchain_txn_validator_heartbeat_v1 : {error,{bad_height,prev,957698,height,957699,got,957698}} / type=validator_heartbeat, validator="orbiting-aquamarine-jay", height=957698, version=10010004
2021-08-09 03:26:59.638 [warning] <0.5358.0>@blockchain_txn:validate:331 invalid txn blockchain_txn_validator_heartbeat_v1 : {error,{bad_height,prev,957698,height,957699,got,957698}} / type=validator_heartbeat, validator="silly-ultraviolet-grasshopper", height=957698, version=10010004
2021-08-09 03:26:59.638 [warning] <0.5358.0>@blockchain_txn:validate:331 invalid txn blockchain_txn_validator_heartbeat_v1 : {error,{bad_height,prev,957698,height,957699,got,957698}} / type=validator_heartbeat, validator="calm-tweed-urchin", height=957698, version=10010004
2021-08-09 03:26:59.639 [warning] <0.5358.0>@blockchain_txn:validate:331 invalid txn blockchain_txn_validator_heartbeat_v1 : {error,{bad_height,prev,957698,height,957699,got,957698}} / type=validator_heartbeat, validator="active-inky-seahorse", height=957698, version=10010004
2021-08-09 03:26:59.640 [warning] <0.5358.0>@blockchain_txn:validate:331 invalid txn blockchain_txn_validator_heartbeat_v1 : {error,{bad_height,prev,957696,height,957699,got,957696}} / type=validator_heartbeat, validator="decent-grey-owl", height=957696, version=10010004
2021-08-09 03:26:59.642 [warning] <0.5358.0>@blockchain_txn:validate:331 invalid txn blockchain_txn_validator_heartbeat_v1 : {error,{bad_height,prev,957696,height,957699,got,957696}} / type=validator_heartbeat, validator="huge-satin-unicorn", height=957696, version=10010004
2021-08-09 03:26:59.643 [warning] <0.5358.0>@blockchain_txn:validate:331 invalid txn blockchain_txn_validator_heartbeat_v1 : {error,{bad_height,prev,957694,height,957699,got,957694}} / type=validator_heartbeat, validator="orbiting-shadow-unicorn", height=957694, version=10010004
2021-08-09 03:26:59.644 [warning] <0.5358.0>@blockchain_txn:validate:331 invalid txn blockchain_txn_validator_heartbeat_v1 : {error,{bad_height,prev,957698,height,957699,got,957698}} / type=validator_heartbeat, validator="boxy-pebble-perch", height=957698, version=10010004
2021-08-09 03:26:59.645 [info] <0.5358.0>@blockchain_txn:validate:290 valid: [{blockchain_txn_poc_request_v1,327}], invalid: [{blockchain_txn_payment_v1,1},{blockchain_txn_poc_receipts_v1,135},{blockchain_txn_state_channel_close_v1,1},{blockchain_txn_validator_heartbeat_v1,24}]
2021-08-09 03:26:59.645 [error] <0.5358.0>@blockchain_txn:absorb_and_commit:454 found invalid transactions: [{{blockchain_txn_validator_heartbeat_v1_pb,<<0,230,180,69,172,223,141,49,139,17,231,125,155,81,241,232,83,133,54,205,177,88,12,135,234,138,238,41,67,201,57,50,155>>,957698,10010004,<<48,70,2,33,0,184,171,242,203,51,249,225,242,14,1,82,43,178,239,196,160,246,231,109,167,121,251,138,46,58,17,22,60,129,13,235,234,2,33,0,218,102,6,41,46,185,9,9,21,193,163,143,130,37,13,194,160,41,104,0,72,147,38,92,118,121,220,98,38,129,190,86>>},validation_failed},{{blockchain_txn_validator_heartbeat_v1_pb,<<0,159,58,7,196,51,111,76,31,3,210,255,159,143,87,221,1,230,27,120,41,94,177,91,173,230,171,38,116,185,75,210,119>>,957694,10010004,<<48,70,2,33,0,165,113,230,145,79,208,243,156,160,141,163,148,11,233,163,183,223,67,70,57,108,241,5,34,247,108,11,163,183,108,25,22,2,33,0,242,33,195,148,145,53,191,89,21,54,140,66,9,20,18,208,226,96,128,119,94,243,65,68,23,223,214,153,166,117,71,66>>},validation_failed},{{blockchain_txn_validator_heartbeat_v1_pb,<<0,127,175,209,208,80,224,187,105,146,126,226,174,92,155,124,217,143,214,131,231,114,16,233,50,64,124,56,106,140,66,49,140>>,957696,10010004,<<48,70,2,33,0,141,141,105,13,71,229,175,212,155,161,174,32,221,24,132,233,78,106,39,0,42,63,192,79,72,37,194,128,187,80,108,194,2,33,0,249,241,45,100,169,155,249,93,223,170,24,201,160,95,222,75,20,57,16,169,45,209,165,154,196,212,58,57,43,245,27,177>>},validation_failed},{{blockchain_txn_validator_heartbeat_v1_pb,<<0,122,231,13,20,225,218,136,77,42,243,96,182,105,155,78,247,121,143,79,205,207,97,9,200,73,55,1,201,179,191,32,158>>,957696,10010004,<<48,70,2,33,0,212,232,220,18,106,91,9,147,121,80,7,175,183,61,5,97,3,26,36,172,129,113,127,132,95,210,153,213,29,48,175,17,2,33,0,168,102,254,253,38,160,191,10,195,25,161,173,23,20,230,185,69,52,238,100,65,67,227,86,169,53,29,161,13,139,68,150>>},validation_failed},{{blockchain_txn_validator_heartbeat_v1_pb,<<0,85,40,106,176,41,141,63,154,129,35,83,156,214,86,187,199,80,36,123,85,239,65,71,112,50,158,80,230,72,161,154,152>>,957698,10010004,<<48,70,2,33,0,241,154,141,192,202,101,236,141,169,241,83,225,102,199,237,4,19,54,205,229,61,10,53,23,37,113,17,59,79,125,45,34,2,33,0,242,99,240,173,201,210,102,238,211,60,49,98,171,165,178,4,115,248,24,79,157,248,106,234,96,19,123,240,34,248,27,154>>},validation_failed},{{blockchain_txn_validator_heartbeat_v1_pb,<<0,84,104,19,91,221,61,45,131,48,215,62,211,231,191,104,212,148,167,62,54,45,222,17,12,40,83,92,172,244,157,183,65>>,957698,10010004,<<48,70,2,33,0,202,51,13,199,75,42,54,224,122,71,45,83,46,144,157,226,207,194,143,29,252,131,81,33,173,123,98,22,183,193,106,15,2,33,0,253,48,175,166,219,168,231,121,69,235,159,208,199,52,14,4,189,83,214,82,43,106,244,72,136,153,146,207,201,32,179,146>>},validation_failed},{{blockchain_txn_validator_heartbeat_v1_pb,<<0,42,199,92,23,95,241,79,24,106,225,100,47,19,49,170,177,146,134,16,14,225,182,255,192,212,73,142,50,29,211,161,238>>,957698,10010004,<<48,70,2,33,0,198,155,186,195,111,52,30,208,252,69,81,204,165,77,17,197,109,229,230,2,119,162,245,89,0,80,154,254,52,160,2,22,2,33,0,164,206,224,242,101,101,31,45,76,76,23,173,29,158,199,207,108,106,199,131,128,211,131,92,24,57,169,124,111,10,177,23>>},validation_failed},{{blockchain_txn_validator_heartbeat_v1_pb,<<0,246,133,239,231,207,6,225,91,206,99,245,46,228,99,135,0,65,160,14,61,71,225,128,47,202,70,33,5,239,198,97,144>>,957698,10010004,<<48,69,2,32,120,75,182,74,2,217,19,244,76,64,239,125,103,244,39,253,22,219,103,89,64,54,228,251,100,217,188,79,5,220,61,148,2,33,0,128,169,90,205,138,30,98,191,114,131,224,249,174,38,167,4,211,44,209,219,199,232,39,141,54,157,10,46,118,130,156,249>>},validation_failed},{{blockchain_txn_validator_heartbeat_v1_pb,<<0,234,164,95,39,125,118,180,99,250,241,141,90,110,41,65,88,138,137,212,144,205,253,37,217,251,32,17,43,230,251,255,181>>,957697,10010004,<<48,69,2,33,0,208,213,213,40,182,58,102,202,4,32,76,114,196,243,147,107,124,40,242,143,150,43,25,116,51,31,19,41,151,94,165,38,2,32,11,192,88,181,146,201,245,84,132,41,64,78,93,167,242,172,9,40,80,180,195,108,154,94,154,184,231,80,49,230,116,73>>},validation_failed},...]
2021-08-09 03:26:59.645 [error] <0.5358.0>@blockchain:replay_blocks:1023 Error absorbing transaction, Ignoring Hash: <<35,171,83,249,21,89,25,37,7,169,41,253,198,57,156,64,194,84,179,201,126,244,183,47,229,206,7,110,114,15,126,130>>, Reason: invalid_txns
2021-08-09 03:27:02.059 [error] <0.5448.0>@libp2p_stream_relay:handle_server_data:193 fail to pass request {libp2p_relay_bridge_cr_pb,"/p2p/11KgF1FfnnVSRnhwTTSNbdsfbpWgA5y9jsZ9ZdmrRyt581nB1Qu","/ip4/152.136.129.112/tcp/44158"} Server not found
anthonyra commented 3 years ago

Just some more context, I also ran into the initial memory size being too small. However, I was able to get one validator to create a snapshot without increasing the memory setting.

I was able to load this snapshot without the above errors. Maybe expanding the heap size lead to the load issues above?

densone commented 3 years ago

@madninja , system available memory

image
evanmcc commented 3 years ago

should be resolved by the cli taking the lock. reopen on core if it's still happening.