search

algorand / indexer

searchable history and current state
MIT License
114 stars 90 forks source link

block 29015075 import failed (testnet) #1516

Closed scholtz closed 1 year ago

scholtz commented 1 year ago

Subject of the issue

I am running testnet indexer v 2.15.3 for testnet and noticed these errors in the logs

Your environment

algod -v

12885884928
3.15.0.stable [rel/stable] (commit #814bc0bd)
go-algorand is licensed with AGPLv3.0
source code available at https://github.com/algorand/go-algorand

goal node status

Last committed block: 29015158
Time since last block: 1.6s
Sync Time: 0.0s
Last consensus protocol: https://github.com/algorandfoundation/specs/tree/44fa607d6051730f5264526bf3c108d51f0eadb6
Next consensus protocol: https://github.com/algorandfoundation/specs/tree/44fa607d6051730f5264526bf3c108d51f0eadb6
Round for next consensus protocol: 29015159
Next consensus protocol supported: true
Last Catchpoint: 29010000#3ZEHUZ3BJAZNS6LZRRXT5QOIDV2VBWF7S2KY6SCXAEMNZUYHFENA
Genesis ID: testnet-v1.0
Genesis hash: SGO1GKSzyE7IEPItTxCByw9x8FmnrCDexi9/cOUJOiI=

Operating System details

K8S setup

Steps to reproduce

Run the indexer

Expected behaviour

No error logs

Actual behaviour

{"level":"info","msg":"round r=29013481 (0 txn) imported in 4.465332ms","time":"2023-04-09T07:16:57Z"}
{"level":"info","msg":"adding block 29013482","time":"2023-04-09T07:17:00Z"}
{"level":"info","msg":"round r=29013482 (1879 txn) imported in 399.929112ms","time":"2023-04-09T07:17:01Z"}
{"level":"info","msg":"adding block 29013483","time":"2023-04-09T07:17:04Z"}
{"level":"info","msg":"round r=29013483 (306 txn) imported in 66.278992ms","time":"2023-04-09T07:17:04Z"}
{"level":"info","msg":"adding block 29013484","time":"2023-04-09T07:17:07Z"}
{"level":"info","msg":"round r=29013484 (133 txn) imported in 33.357236ms","time":"2023-04-09T07:17:07Z"}
{"level":"info","msg":"adding block 29013485","time":"2023-04-09T07:17:11Z"}
{"level":"info","msg":"round r=29013485 (90 txn) imported in 26.163655ms","time":"2023-04-09T07:17:11Z"}
{"level":"info","msg":"adding block 29013486","time":"2023-04-09T07:17:15Z"}
{"level":"info","msg":"round r=29013486 (150 txn) imported in 35.790069ms","time":"2023-04-09T07:17:15Z"}
{"level":"info","msg":"adding block 29013487","time":"2023-04-09T07:17:18Z"}
{"level":"info","msg":"round r=29013487 (22 txn) imported in 8.869277ms","time":"2023-04-09T07:17:18Z"}
{"level":"info","msg":"adding block 29013488","time":"2023-04-09T07:17:22Z"}
{"level":"info","msg":"round r=29013488 (2 txn) imported in 5.979003ms","time":"2023-04-09T07:17:22Z"}
time="2023-04-09T07:17:26.056630 +0000" level=error msg="[Stack] goroutine 231088161 [running]:\nruntime/debug.Stack()\n\t/usr/local/go/src/runtime/debug/stack.go:24 +0x65\ngithub.com/algorand/go-algorand/logging.logger.Errorf({0xc000139340, 0xc000116060}, {0x16cd563, 0x1b}, {0xc017810000, 0x1, 0x1})\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/logging/log.go:229 +0x69\ngithub.com/algorand/go-algorand/data/transactions/logic.eval.func1()\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/data/transactions/logic/eval.go:818 +0x1cc\npanic({0x15203c0, 0x2480760})\n\t/usr/local/go/src/runtime/panic.go:1047 +0x266\ngithub.com/algorand/go-algorand/ledger/internal.(*roundCowState).blockHdrCached(...)\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/ledger/internal/cow.go:249\ngithub.com/algorand/go-algorand/ledger/internal.(*roundCowState).BlockHdrCached(_, _)\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/ledger/internal/applications.go:132 +0x7a\ngithub.com/algorand/go-algorand/data/transactions/logic.opBlock(0xc01733e000)\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/data/transactions/logic/eval.go:5252 +0x19a\ngithub.com/algorand/go-algorand/data/transactions/logic.(*EvalContext).step(0xc01733e000)\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/data/transactions/logic/eval.go:1101 +0x7d4\ngithub.com/algorand/go-algorand/data/transactions/logic.eval({0xc00cfb4000, 0xc01733e000, 0xc002951000}, 0xc01733e000)\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/data/transactions/logic/eval.go:868 +0x4bf\ngithub.com/algorand/go-algorand/data/transactions/logic.EvalContract({0xc00cfb4000, 0xe55, 0xe55}, 0x1, 0x99a80fd, 0xc0177a6620)\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/data/transactions/logic/eval.go:761 +0x43c\ngithub.com/algorand/go-algorand/ledger/internal.(*roundCowState).StatefulEval(0xc008b37680, 0x1, 0xc0177a6620, 0x85, {0xc00cfb4000, 0xe55, 0xe55})\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/ledger/internal/appcow.go:464 +0x13d\ngithub.com/algorand/go-algorand/ledger/apply.ApplicationCall({{}, 0x99a80fd, 0x0, {0xc017666480, 0xf, 0xf}, {0xc0176122c0, 0x2, 0x2}, {0xc021806280, ...}, ...}, ...)\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/ledger/apply/application.go:446 +0x473\ngithub.com/algorand/go-algorand/ledger/internal.(*BlockEvaluator).applyTransaction(_, {{}, {0xc021806278, 0x4}, {{}, {0x89, 0xe5, 0x6f, 0x13, 0x68, ...}, ...}, ...}, ...)\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/ledger/internal/eval.go:1171 +0x64a\ngithub.com/algorand/go-algorand/ledger/internal.(*BlockEvaluator).transaction(_, {{}, {0xe5, 0x15, 0x56, 0xbe, 0xb5, 0xaa, 0x60, 0x7d, ...}, ...}, ...)\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/ledger/internal/eval.go:1096 +0x574\ngithub.com/algorand/go-algorand/ledger/internal.(*BlockEvaluator).transactionGroup(0xc01777c000, {0xc0177a2880, 0x2, 0x2})\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/ledger/internal/eval.go:962 +0x3bc\ngithub.com/algorand/go-algorand/ledger/internal.(*BlockEvaluator).TransactionGroup(...)\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/ledger/internal/eval.go:929\ngithub.com/algorand/go-algorand/ledger/internal.(*BlockEvaluator).ProcessBlockForIndexer(0xc01777c000, 0xffffffffffffffff)\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/ledger/internal/evalindexer.go:36 +0x2ff\ngithub.com/algorand/go-algorand/ledger.EvalForIndexer({_, _}, _, {0x2710, 0x2328, 0x222e0, 0x2710, 0x249f0, 0x80, 0x500000, ...}, ...)\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/ledger/evalindexer.go:249 +0x425\ngithub.com/algorand/indexer/processor/blockprocessor.(*blockProcessor).Process(0xc005dbf8d8, 0xc0170fc000)\n\t/go/src/github.com/algorand/indexer/processor/blockprocessor/block_processor.go:107 +0x2d3\nmain.handleBlock(0xc0170fc000, {0x19720c0, 0xc005dbf8d8})\n\t/go/src/github.com/algorand/indexer/cmd/algorand-indexer/daemon.go:525 +0x7e\nmain.blockHandler.func1({0x1985298, 0xc00f979440}, 0x0)\n\t/go/src/github.com/algorand/indexer/cmd/algorand-indexer/daemon.go:506 +0x75\ngithub.com/algorand/indexer/fetcher.(*fetcherImpl).processQueue(0xc000156080, {0x1985298, 0xc00f979440})\n\t/go/src/github.com/algorand/indexer/fetcher/fetcher.go:84 +0xf4\ngithub.com/algorand/indexer/fetcher.(*fetcherImpl).Run.func1()\n\t/go/src/github.com/algorand/indexer/fetcher/fetcher.go:227 +0x2e\ncreated by github.com/algorand/indexer/fetcher.(*fetcherImpl).Run\n\t/go/src/github.com/algorand/indexer/fetcher/fetcher.go:226 +0x167\n" file=eval.go function=github.com/algorand/go-algorand/data/transactions/logic.eval.func1 line=818
time="2023-04-09T07:17:26.056818 +0000" level=error msg="recovered panic in Eval: panic in TEAL Eval: runtime error: invalid memory address or nil pointer dereference\ngoroutine 231088161 [running]:\ngithub.com/algorand/go-algorand/data/transactions/logic.eval.func1()\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/data/transactions/logic/eval.go:811 +0x89\npanic({0x15203c0, 0x2480760})\n\t/usr/local/go/src/runtime/panic.go:1047 +0x266\ngithub.com/algorand/go-algorand/ledger/internal.(*roundCowState).blockHdrCached(...)\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/ledger/internal/cow.go:249\ngithub.com/algorand/go-algorand/ledger/internal.(*roundCowState).BlockHdrCached(_, _)\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/ledger/internal/applications.go:132 +0x7a\ngithub.com/algorand/go-algorand/data/transactions/logic.opBlock(0xc01733e000)\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/data/transactions/logic/eval.go:5252 +0x19a\ngithub.com/algorand/go-algorand/data/transactions/logic.(*EvalContext).step(0xc01733e000)\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/data/transactions/logic/eval.go:1101 +0x7d4\ngithub.com/algorand/go-algorand/data/transactions/logic.eval({0xc00cfb4000, 0xc01733e000, 0xc002951000}, 0xc01733e000)\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/data/transactions/logic/eval.go:868 +0x4bf\ngithub.com/algorand/go-algorand/data/transactions/logic.EvalContract({0xc00cfb4000, 0xe55, 0xe55}, 0x1, 0x99a80fd, 0xc0177a6620)\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/data/transactions/logic/eval.go:761 +0x43c\ngithub.com/algorand/go-algorand/ledger/internal.(*roundCowState).StatefulEval(0xc008b37680, 0x1, 0xc0177a6620, 0x85, {0xc00cfb4000, 0xe55, 0xe55})\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/ledger/internal/appcow.go:464 +0x13d\ngithub.com/algorand/go-algorand/ledger/apply.ApplicationCall({{}, 0x99a80fd, 0x0, {0xc017666480, 0xf, 0xf}, {0xc0176122c0, 0x2, 0x2}, {0xc021806280, ...}, ...}, ...)\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/ledger/apply/application.go:446 +0x473\ngithub.com/algorand/go-algorand/ledger/internal.(*BlockEvaluator).applyTransaction(_, {{}, {0xc021806278, 0x4}, {{}, {0x89, 0xe5, 0x6f, 0x13, 0x68, ...}, ...}, ...}, ...)\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/ledger/internal/eval.go:1171 +0x64a\ngithub.com/algorand/go-algorand/ledger/internal.(*BlockEvaluator).transaction(_, {{}, {0xe5, 0x15, 0x56, 0xbe, 0xb5, 0xaa, 0x60, 0x7d, ...}, ...}, ...)\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/ledger/internal/eval.go:1096 +0x574\ngithub.com/algorand/go-algorand/ledger/internal.(*BlockEvaluator).transactionGroup(0xc01777c000, {0xc0177a2880, 0x2, 0x2})\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/ledger/internal/eval.go:962 +0x3bc\ngithub.com/algorand/go-algorand/ledger/internal.(*BlockEvaluator).TransactionGroup(...)\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/ledger/internal/eval.go:929\ngithub.com/algorand/go-algorand/ledger/internal.(*BlockEvaluator).ProcessBlockForIndexer(0xc01777c000, 0xffffffffffffffff)\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/ledger/internal/evalindexer.go:36 +0x2ff\ngithub.com/algorand/go-algorand/ledger.EvalForIndexer({_, _}, _, {0x2710, 0x2328, 0x222e0, 0x2710, 0x249f0, 0x80, 0x500000, ...}, ...)\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/ledger/evalindexer.go:249 +0x425\ngithub.com/algorand/indexer/processor/blockprocessor.(*blockProcessor).Process(0xc005dbf8d8, 0xc0170fc000)\n\t/go/src/github.com/algorand/indexer/processor/blockprocessor/block_processor.go:107 +0x2d3\nmain.handleBlock(0xc0170fc000, {0x19720c0, 0xc005dbf8d8})\n\t/go/src/github.com/algorand/indexer/cmd/algorand-indexer/daemon.go:525 +0x7e\nmain.blockHandler.func1({0x1985298, 0xc00f979440}, 0x0)\n\t/go/src/github.com/algorand/indexer/cmd/algorand-indexer/daemon.go:506 +0x75\ngithub.com/algorand/indexer/fetcher.(*fetcherImpl).processQueue(0xc000156080, {0x1985298, 0xc00f979440})\n\t/go/src/github.com/algorand/indexer/fetcher/fetcher.go:84 +0xf4\ngithub.com/algorand/indexer/fetcher.(*fetcherImpl).Run.func1()\n\t/go/src/github.com/algorand/indexer/fetcher/fetcher.go:227 +0x2e\ncreated by github.com/algorand/indexer/fetcher.(*fetcherImpl).Run\n\t/go/src/github.com/algorand/indexer/fetcher/fetcher.go:226 +0x167\n" file=eval.go function=github.com/algorand/go-algorand/data/transactions/logic.eval.func1 line=818
{"error":"Process() eval err: ProcessBlockForIndexer() err: transaction OS4K6O27G2SIDNP255VNWQT3KN3OS5JOS6SVSDEDQG6Y2JMWWFZQ: logic eval error: panic in TEAL Eval: runtime error: invalid memory address or nil pointer dereference\ngoroutine 231088161 [running]:\ngithub.com/algorand/go-algorand/data/transactions/logic.eval.func1()\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/data/transactions/logic/eval.go:811 +0x89\npanic({0x15203c0, 0x2480760})\n\t/usr/local/go/src/runtime/panic.go:1047 +0x266\ngithub.com/algorand/go-algorand/ledger/internal.(*roundCowState).blockHdrCached(...)\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/ledger/internal/cow.go:249\ngithub.com/algorand/go-algorand/ledger/internal.(*roundCowState).BlockHdrCached(_, _)\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/ledger/internal/applications.go:132 +0x7a\ngithub.com/algorand/go-algorand/data/transactions/logic.opBlock(0xc01733e000)\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/data/transactions/logic/eval.go:5252 +0x19a\ngithub.com/algorand/go-algorand/data/transactions/logic.(*EvalContext).step(0xc01733e000)\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/data/transactions/logic/eval.go:1101 +0x7d4\ngithub.com/algorand/go-algorand/data/transactions/logic.eval({0xc00cfb4000, 0xc01733e000, 0xc002951000}, 0xc01733e000)\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/data/transactions/logic/eval.go:868 +0x4bf\ngithub.com/algorand/go-algorand/data/transactions/logic.EvalContract({0xc00cfb4000, 0xe55, 0xe55}, 0x1, 0x99a80fd, 0xc0177a6620)\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/data/transactions/logic/eval.go:761 +0x43c\ngithub.com/algorand/go-algorand/ledger/internal.(*roundCowState).StatefulEval(0xc008b37680, 0x1, 0xc0177a6620, 0x85, {0xc00cfb4000, 0xe55, 0xe55})\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/ledger/internal/appcow.go:464 +0x13d\ngithub.com/algorand/go-algorand/ledger/apply.ApplicationCall({{}, 0x99a80fd, 0x0, {0xc017666480, 0xf, 0xf}, {0xc0176122c0, 0x2, 0x2}, {0xc021806280, ...}, ...}, ...)\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/ledger/apply/application.go:446 +0x473\ngithub.com/algorand/go-algorand/ledger/internal.(*BlockEvaluator).applyTransaction(_, {{}, {0xc021806278, 0x4}, {{}, {0x89, 0xe5, 0x6f, 0x13, 0x68, ...}, ...}, ...}, ...)\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/ledger/internal/eval.go:1171 +0x64a\ngithub.com/algorand/go-algorand/ledger/internal.(*BlockEvaluator).transaction(_, {{}, {0xe5, 0x15, 0x56, 0xbe, 0xb5, 0xaa, 0x60, 0x7d, ...}, ...}, ...)\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/ledger/internal/eval.go:1096 +0x574\ngithub.com/algorand/go-algorand/ledger/internal.(*BlockEvaluator).transactionGroup(0xc01777c000, {0xc0177a2880, 0x2, 0x2})\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/ledger/internal/eval.go:962 +0x3bc\ngithub.com/algorand/go-algorand/ledger/internal.(*BlockEvaluator).TransactionGroup(...)\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/ledger/internal/eval.go:929\ngithub.com/algorand/go-algorand/ledger/internal.(*BlockEvaluator).ProcessBlockForIndexer(0xc01777c000, 0xffffffffffffffff)\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/ledger/internal/evalindexer.go:36 +0x2ff\ngithub.com/algorand/go-algorand/ledger.EvalForIndexer({_, _}, _, {0x2710, 0x2328, 0x222e0, 0x2710, 0x249f0, 0x80, 0x500000, ...}, ...)\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/ledger/evalindexer.go:249 +0x425\ngithub.com/algorand/indexer/processor/blockprocessor.(*blockProcessor).Process(0xc005dbf8d8, 0xc0170fc000)\n\t/go/src/github.com/algorand/indexer/processor/blockprocessor/block_processor.go:107 +0x2d3\nmain.handleBlock(0xc0170fc000, {0x19720c0, 0xc005dbf8d8})\n\t/go/src/github.com/algorand/indexer/cmd/algorand-indexer/daemon.go:525 +0x7e\nmain.blockHandler.func1({0x1985298, 0xc00f979440}, 0x0)\n\t/go/src/github.com/algorand/indexer/cmd/algorand-indexer/daemon.go:506 +0x75\ngithub.com/algorand/indexer/fetcher.(*fetcherImpl).processQueue(0xc000156080, {0x1985298, 0xc00f979440})\n\t/go/src/github.com/algorand/indexer/fetcher/fetcher.go:84 +0xf4\ngithub.com/algorand/indexer/fetcher.(*fetcherImpl).Run.func1()\n\t/go/src/github.com/algorand/indexer/fetcher/fetcher.go:227 +0x2e\ncreated by github.com/algorand/indexer/fetcher.(*fetcherImpl).Run\n\t/go/src/github.com/algorand/indexer/fetcher/fetcher.go:226 +0x167\n. Details: pc=2123, opcodes=gtxnsa ApplicationArgs 3\nframe_dig -1\nblock BlkSeed\n","level":"error","msg":"block 29013489 import failed","time":"2023-04-09T07:17:26Z"}
{"level":"info","msg":"adding block 29013489","time":"2023-04-09T07:17:27Z"}
{"level":"info","msg":"round r=29013489 (8 txn) imported in 8.838589ms","time":"2023-04-09T07:17:27Z"}
{"level":"info","msg":"adding block 29013490","time":"2023-04-09T07:17:29Z"}
{"level":"info","msg":"round r=29013490 (9 txn) imported in 6.41808ms","time":"2023-04-09T07:17:29Z"}
{"level":"info","msg":"adding block 29013491","time":"2023-04-09T07:17:33Z"}


{"level":"info","msg":"round r=29015072 (288 txn) imported in 61.664683ms","time":"2023-04-09T08:53:03Z"}
{"level":"info","msg":"adding block 29015073","time":"2023-04-09T08:53:07Z"}
{"level":"info","msg":"round r=29015073 (132 txn) imported in 35.489664ms","time":"2023-04-09T08:53:07Z"}
{"level":"info","msg":"adding block 29015074","time":"2023-04-09T08:53:10Z"}
{"level":"info","msg":"round r=29015074 (121 txn) imported in 36.198389ms","time":"2023-04-09T08:53:10Z"}
time="2023-04-09T08:53:14.539617 +0000" level=error msg="[Stack] goroutine 231088161 [running]:\nruntime/debug.Stack()\n\t/usr/local/go/src/runtime/debug/stack.go:24 +0x65\ngithub.com/algorand/go-algorand/logging.logger.Errorf({0xc000139340, 0xc000116060}, {0x16cd563, 0x1b}, {0xc0185cc000, 0x1, 0x1})\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/logging/log.go:229 +0x69\ngithub.com/algorand/go-algorand/data/transactions/logic.eval.func1()\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/data/transactions/logic/eval.go:818 +0x1cc\npanic({0x15203c0, 0x2480760})\n\t/usr/local/go/src/runtime/panic.go:1047 +0x266\ngithub.com/algorand/go-algorand/ledger/internal.(*roundCowState).blockHdrCached(...)\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/ledger/internal/cow.go:249\ngithub.com/algorand/go-algorand/ledger/internal.(*roundCowState).BlockHdrCached(_, _)\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/ledger/internal/applications.go:132 +0x7a\ngithub.com/algorand/go-algorand/data/transactions/logic.opBlock(0xc018580000)\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/data/transactions/logic/eval.go:5252 +0x19a\ngithub.com/algorand/go-algorand/data/transactions/logic.(*EvalContext).step(0xc018580000)\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/data/transactions/logic/eval.go:1101 +0x7d4\ngithub.com/algorand/go-algorand/data/transactions/logic.eval({0xc00946d000, 0xc018580000, 0xc0054a3000}, 0xc018580000)\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/data/transactions/logic/eval.go:868 +0x4bf\ngithub.com/algorand/go-algorand/data/transactions/logic.EvalContract({0xc00946d000, 0xe55, 0xe55}, 0x1, 0x99a80fd, 0xc0181c2e00)\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/data/transactions/logic/eval.go:761 +0x43c\ngithub.com/algorand/go-algorand/ledger/internal.(*roundCowState).StatefulEval(0xc020f6b200, 0x1, 0xc0181c2e00, 0x85, {0xc00946d000, 0xe55, 0xe55})\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/ledger/internal/appcow.go:464 +0x13d\ngithub.com/algorand/go-algorand/ledger/apply.ApplicationCall({{}, 0x99a80fd, 0x0, {0xc017c12a80, 0xf, 0xf}, {0xc0179ee5c0, 0x2, 0x2}, {0xc0130444d0, ...}, ...}, ...)\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/ledger/apply/application.go:446 +0x473\ngithub.com/algorand/go-algorand/ledger/internal.(*BlockEvaluator).applyTransaction(_, {{}, {0xc0130444e0, 0x4}, {{}, {0x6f, 0x4d, 0x75, 0x6a, 0x99, ...}, ...}, ...}, ...)\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/ledger/internal/eval.go:1171 +0x64a\ngithub.com/algorand/go-algorand/ledger/internal.(*BlockEvaluator).transaction(_, {{}, {0xd9, 0x74, 0xb4, 0xdf, 0x2f, 0xab, 0x82, 0x14, ...}, ...}, ...)\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/ledger/internal/eval.go:1096 +0x574\ngithub.com/algorand/go-algorand/ledger/internal.(*BlockEvaluator).transactionGroup(0xc01810a000, {0xc018142000, 0x2, 0x2})\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/ledger/internal/eval.go:962 +0x3bc\ngithub.com/algorand/go-algorand/ledger/internal.(*BlockEvaluator).TransactionGroup(...)\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/ledger/internal/eval.go:929\ngithub.com/algorand/go-algorand/ledger/internal.(*BlockEvaluator).ProcessBlockForIndexer(0xc01810a000, 0xffffffffffffffff)\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/ledger/internal/evalindexer.go:36 +0x2ff\ngithub.com/algorand/go-algorand/ledger.EvalForIndexer({_, _}, _, {0x2710, 0x2328, 0x222e0, 0x2710, 0x249f0, 0x80, 0x500000, ...}, ...)\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/ledger/evalindexer.go:249 +0x425\ngithub.com/algorand/indexer/processor/blockprocessor.(*blockProcessor).Process(0xc005dbf8d8, 0xc0179ca000)\n\t/go/src/github.com/algorand/indexer/processor/blockprocessor/block_processor.go:107 +0x2d3\nmain.handleBlock(0xc0179ca000, {0x19720c0, 0xc005dbf8d8})\n\t/go/src/github.com/algorand/indexer/cmd/algorand-indexer/daemon.go:525 +0x7e\nmain.blockHandler.func1({0x1985298, 0xc00f979440}, 0x0)\n\t/go/src/github.com/algorand/indexer/cmd/algorand-indexer/daemon.go:506 +0x75\ngithub.com/algorand/indexer/fetcher.(*fetcherImpl).processQueue(0xc000156080, {0x1985298, 0xc00f979440})\n\t/go/src/github.com/algorand/indexer/fetcher/fetcher.go:84 +0xf4\ngithub.com/algorand/indexer/fetcher.(*fetcherImpl).Run.func1()\n\t/go/src/github.com/algorand/indexer/fetcher/fetcher.go:227 +0x2e\ncreated by github.com/algorand/indexer/fetcher.(*fetcherImpl).Run\n\t/go/src/github.com/algorand/indexer/fetcher/fetcher.go:226 +0x167\n" file=eval.go function=github.com/algorand/go-algorand/data/transactions/logic.eval.func1 line=818
time="2023-04-09T08:53:14.539755 +0000" level=error msg="recovered panic in Eval: panic in TEAL Eval: runtime error: invalid memory address or nil pointer dereference\ngoroutine 231088161 [running]:\ngithub.com/algorand/go-algorand/data/transactions/logic.eval.func1()\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/data/transactions/logic/eval.go:811 +0x89\npanic({0x15203c0, 0x2480760})\n\t/usr/local/go/src/runtime/panic.go:1047 +0x266\ngithub.com/algorand/go-algorand/ledger/internal.(*roundCowState).blockHdrCached(...)\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/ledger/internal/cow.go:249\ngithub.com/algorand/go-algorand/ledger/internal.(*roundCowState).BlockHdrCached(_, _)\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/ledger/internal/applications.go:132 +0x7a\ngithub.com/algorand/go-algorand/data/transactions/logic.opBlock(0xc018580000)\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/data/transactions/logic/eval.go:5252 +0x19a\ngithub.com/algorand/go-algorand/data/transactions/logic.(*EvalContext).step(0xc018580000)\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/data/transactions/logic/eval.go:1101 +0x7d4\ngithub.com/algorand/go-algorand/data/transactions/logic.eval({0xc00946d000, 0xc018580000, 0xc0054a3000}, 0xc018580000)\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/data/transactions/logic/eval.go:868 +0x4bf\ngithub.com/algorand/go-algorand/data/transactions/logic.EvalContract({0xc00946d000, 0xe55, 0xe55}, 0x1, 0x99a80fd, 0xc0181c2e00)\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/data/transactions/logic/eval.go:761 +0x43c\ngithub.com/algorand/go-algorand/ledger/internal.(*roundCowState).StatefulEval(0xc020f6b200, 0x1, 0xc0181c2e00, 0x85, {0xc00946d000, 0xe55, 0xe55})\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/ledger/internal/appcow.go:464 +0x13d\ngithub.com/algorand/go-algorand/ledger/apply.ApplicationCall({{}, 0x99a80fd, 0x0, {0xc017c12a80, 0xf, 0xf}, {0xc0179ee5c0, 0x2, 0x2}, {0xc0130444d0, ...}, ...}, ...)\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/ledger/apply/application.go:446 +0x473\ngithub.com/algorand/go-algorand/ledger/internal.(*BlockEvaluator).applyTransaction(_, {{}, {0xc0130444e0, 0x4}, {{}, {0x6f, 0x4d, 0x75, 0x6a, 0x99, ...}, ...}, ...}, ...)\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/ledger/internal/eval.go:1171 +0x64a\ngithub.com/algorand/go-algorand/ledger/internal.(*BlockEvaluator).transaction(_, {{}, {0xd9, 0x74, 0xb4, 0xdf, 0x2f, 0xab, 0x82, 0x14, ...}, ...}, ...)\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/ledger/internal/eval.go:1096 +0x574\ngithub.com/algorand/go-algorand/ledger/internal.(*BlockEvaluator).transactionGroup(0xc01810a000, {0xc018142000, 0x2, 0x2})\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/ledger/internal/eval.go:962 +0x3bc\ngithub.com/algorand/go-algorand/ledger/internal.(*BlockEvaluator).TransactionGroup(...)\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/ledger/internal/eval.go:929\ngithub.com/algorand/go-algorand/ledger/internal.(*BlockEvaluator).ProcessBlockForIndexer(0xc01810a000, 0xffffffffffffffff)\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/ledger/internal/evalindexer.go:36 +0x2ff\ngithub.com/algorand/go-algorand/ledger.EvalForIndexer({_, _}, _, {0x2710, 0x2328, 0x222e0, 0x2710, 0x249f0, 0x80, 0x500000, ...}, ...)\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/ledger/evalindexer.go:249 +0x425\ngithub.com/algorand/indexer/processor/blockprocessor.(*blockProcessor).Process(0xc005dbf8d8, 0xc0179ca000)\n\t/go/src/github.com/algorand/indexer/processor/blockprocessor/block_processor.go:107 +0x2d3\nmain.handleBlock(0xc0179ca000, {0x19720c0, 0xc005dbf8d8})\n\t/go/src/github.com/algorand/indexer/cmd/algorand-indexer/daemon.go:525 +0x7e\nmain.blockHandler.func1({0x1985298, 0xc00f979440}, 0x0)\n\t/go/src/github.com/algorand/indexer/cmd/algorand-indexer/daemon.go:506 +0x75\ngithub.com/algorand/indexer/fetcher.(*fetcherImpl).processQueue(0xc000156080, {0x1985298, 0xc00f979440})\n\t/go/src/github.com/algorand/indexer/fetcher/fetcher.go:84 +0xf4\ngithub.com/algorand/indexer/fetcher.(*fetcherImpl).Run.func1()\n\t/go/src/github.com/algorand/indexer/fetcher/fetcher.go:227 +0x2e\ncreated by github.com/algorand/indexer/fetcher.(*fetcherImpl).Run\n\t/go/src/github.com/algorand/indexer/fetcher/fetcher.go:226 +0x167\n" file=eval.go function=github.com/algorand/go-algorand/data/transactions/logic.eval.func1 line=818
{"error":"Process() eval err: ProcessBlockForIndexer() err: transaction HQZIRAP57GUHINFPAPPO6NCCRR5X567BSNGFNFMCNKXSP2IC5VCA: logic eval error: panic in TEAL Eval: runtime error: invalid memory address or nil pointer dereference\ngoroutine 231088161 [running]:\ngithub.com/algorand/go-algorand/data/transactions/logic.eval.func1()\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/data/transactions/logic/eval.go:811 +0x89\npanic({0x15203c0, 0x2480760})\n\t/usr/local/go/src/runtime/panic.go:1047 +0x266\ngithub.com/algorand/go-algorand/ledger/internal.(*roundCowState).blockHdrCached(...)\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/ledger/internal/cow.go:249\ngithub.com/algorand/go-algorand/ledger/internal.(*roundCowState).BlockHdrCached(_, _)\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/ledger/internal/applications.go:132 +0x7a\ngithub.com/algorand/go-algorand/data/transactions/logic.opBlock(0xc018580000)\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/data/transactions/logic/eval.go:5252 +0x19a\ngithub.com/algorand/go-algorand/data/transactions/logic.(*EvalContext).step(0xc018580000)\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/data/transactions/logic/eval.go:1101 +0x7d4\ngithub.com/algorand/go-algorand/data/transactions/logic.eval({0xc00946d000, 0xc018580000, 0xc0054a3000}, 0xc018580000)\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/data/transactions/logic/eval.go:868 +0x4bf\ngithub.com/algorand/go-algorand/data/transactions/logic.EvalContract({0xc00946d000, 0xe55, 0xe55}, 0x1, 0x99a80fd, 0xc0181c2e00)\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/data/transactions/logic/eval.go:761 +0x43c\ngithub.com/algorand/go-algorand/ledger/internal.(*roundCowState).StatefulEval(0xc020f6b200, 0x1, 0xc0181c2e00, 0x85, {0xc00946d000, 0xe55, 0xe55})\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/ledger/internal/appcow.go:464 +0x13d\ngithub.com/algorand/go-algorand/ledger/apply.ApplicationCall({{}, 0x99a80fd, 0x0, {0xc017c12a80, 0xf, 0xf}, {0xc0179ee5c0, 0x2, 0x2}, {0xc0130444d0, ...}, ...}, ...)\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/ledger/apply/application.go:446 +0x473\ngithub.com/algorand/go-algorand/ledger/internal.(*BlockEvaluator).applyTransaction(_, {{}, {0xc0130444e0, 0x4}, {{}, {0x6f, 0x4d, 0x75, 0x6a, 0x99, ...}, ...}, ...}, ...)\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/ledger/internal/eval.go:1171 +0x64a\ngithub.com/algorand/go-algorand/ledger/internal.(*BlockEvaluator).transaction(_, {{}, {0xd9, 0x74, 0xb4, 0xdf, 0x2f, 0xab, 0x82, 0x14, ...}, ...}, ...)\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/ledger/internal/eval.go:1096 +0x574\ngithub.com/algorand/go-algorand/ledger/internal.(*BlockEvaluator).transactionGroup(0xc01810a000, {0xc018142000, 0x2, 0x2})\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/ledger/internal/eval.go:962 +0x3bc\ngithub.com/algorand/go-algorand/ledger/internal.(*BlockEvaluator).TransactionGroup(...)\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/ledger/internal/eval.go:929\ngithub.com/algorand/go-algorand/ledger/internal.(*BlockEvaluator).ProcessBlockForIndexer(0xc01810a000, 0xffffffffffffffff)\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/ledger/internal/evalindexer.go:36 +0x2ff\ngithub.com/algorand/go-algorand/ledger.EvalForIndexer({_, _}, _, {0x2710, 0x2328, 0x222e0, 0x2710, 0x249f0, 0x80, 0x500000, ...}, ...)\n\t/go/src/github.com/algorand/indexer/third_party/go-algorand/ledger/evalindexer.go:249 +0x425\ngithub.com/algorand/indexer/processor/blockprocessor.(*blockProcessor).Process(0xc005dbf8d8, 0xc0179ca000)\n\t/go/src/github.com/algorand/indexer/processor/blockprocessor/block_processor.go:107 +0x2d3\nmain.handleBlock(0xc0179ca000, {0x19720c0, 0xc005dbf8d8})\n\t/go/src/github.com/algorand/indexer/cmd/algorand-indexer/daemon.go:525 +0x7e\nmain.blockHandler.func1({0x1985298, 0xc00f979440}, 0x0)\n\t/go/src/github.com/algorand/indexer/cmd/algorand-indexer/daemon.go:506 +0x75\ngithub.com/algorand/indexer/fetcher.(*fetcherImpl).processQueue(0xc000156080, {0x1985298, 0xc00f979440})\n\t/go/src/github.com/algorand/indexer/fetcher/fetcher.go:84 +0xf4\ngithub.com/algorand/indexer/fetcher.(*fetcherImpl).Run.func1()\n\t/go/src/github.com/algorand/indexer/fetcher/fetcher.go:227 +0x2e\ncreated by github.com/algorand/indexer/fetcher.(*fetcherImpl).Run\n\t/go/src/github.com/algorand/indexer/fetcher/fetcher.go:226 +0x167\n. Details: pc=2123, opcodes=gtxnsa ApplicationArgs 3\nframe_dig -1\nblock BlkSeed\n","level":"error","msg":"block 29015075 import failed","time":"2023-04-09T08:53:14Z"}
{"level":"info","msg":"adding block 29015075","time":"2023-04-09T08:53:15Z"}
{"level":"info","msg":"round r=29015075 (133 txn) imported in 32.573721ms","time":"2023-04-09T08:53:15Z"}
{"level":"info","msg":"adding block 29015076","time":"2023-04-09T08:53:18Z"}
{"level":"info","msg":"round r=29015076 (18 txn) imported in 11.557642ms","time":"2023-04-09T08:53:18Z"}
{"level":"info","msg":"adding block 29015077","time":"2023-04-09T08:53:21Z"}
shiqizng commented 1 year ago

did your indexer instance exit with an error code? the logs seems to show it recovered from the error after retry and continue to import rounds after 29015075. ... BlkSeed\n","level":"error","msg":"block 29015075 import failed","time":"2023-04-09T08:53:14Z"} {"level":"info","msg":"adding block 29015075","time":"2023-04-09T08:53:15Z"} {"level":"info","msg":"round r=29015075 (133 txn) imported in 32.573721ms","time":"2023-04-09T08:53:15Z"} {"level":"info","msg":"adding block 29015076","time":"2023-04-09T08:53:18Z"} {"level":"info","msg":"round r=29015076 (18 txn) imported in 11.557642ms","time":"2023-04-09T08:53:18Z"} {"level":"info","msg":"adding block 29015077","time":"2023-04-09T08:53:21Z"}

shiqizng commented 1 year ago

this error should be fixed in the next go-algorand release.