indexer sync halt

[dnjef9duo]

i setup ton-indexer v3 for mainnet.

indexer halt everyday recently.

ton-indexer version commit 72e5760db8b5ea4ce3371f2f583024cebf92c235 (HEAD -> master, origin/master, origin/HEAD)

my config for ton-indexer

# TON Indexer config
POSTGRES_HOST=postgres
POSTGRES_PORT=5432
POSTGRES_USER=postgres
POSTGRES_PASSWORD_FILE=private/postgres_password
POSTGRES_DBNAME=ton_index
POSTGRES_PUBLISH_PORT=5432

TON_INDEXER_API_ROOT_PATH=/api/v3
TON_INDEXER_API_PORT=81
TON_INDEXER_API_TITLE=TON Indexer
TON_INDEXER_WORKERS=4

TON_INDEXER_TON_HTTP_API_ENDPOINT=localhost

TON_WORKER_DBROOT=/var/ton-work/db/
TON_WORKER_FROM=38354000
TON_WORKER_MAX_PARALLEL_TASKS=1024
TON_WORKER_INSERT_BATCH_SIZE=512
TON_WORKER_INSERT_PARALLEL_ACTORS=3

and here's the logs i grep by command cat docker.20240620-12h.log grep -v 'cell not found' | grep -v 'Rescheduling' | grep -v 'block not found'

ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 5][2024-06-19 17:55:14.064028833][IndexScheduler.cpp:172][!indexscheduler]        Last: 38469273 / 38469305       Blk/s: 0.253808 ETA: 2m 6s      Q[0M, 0b, 0t, 0m]ESC[0m
ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 4][2024-06-19 17:55:24.075977842][IndexScheduler.cpp:172][!indexscheduler]        Last: 38469273 / 38469307       Blk/s: 0.088497 ETA: 6m 24s     Q[0M, 0b, 0t, 0m]ESC[0m
ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 3][2024-06-19 17:55:34.088068994][IndexScheduler.cpp:172][!indexscheduler]        Last: 38469275 / 38469311       Blk/s: 0.116951 ETA: 5m 7s      Q[0M, 0b, 0t, 0m]ESC[0m
ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 2][2024-06-19 17:55:44.100348436][IndexScheduler.cpp:172][!indexscheduler]        Last: 38469279 / 38469312       Blk/s: 0.268367 ETA: 2m 2s      Q[0M, 0b, 0t, 0m]ESC[0m
ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 6][2024-06-19 17:55:54.110987234][IndexScheduler.cpp:172][!indexscheduler]        Last: 38469280 / 38469314       Blk/s: 0.174574 ETA: 3m 14s     Q[0M, 0b, 0t, 0m]ESC[0m
ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 6][2024-06-19 17:56:04.124228634][IndexScheduler.cpp:172][!indexscheduler]        Last: 38469285 / 38469319       Blk/s: 0.440814 ETA: 1m 17s     Q[1M, 2b, 231t, 400m]ESC[0m
ton-indexer-index-worker-1  | ESC[1;31m[ 0][t 6][2024-06-19 17:56:07.133652766][ActorOwn.h:80][!slice]  Check `!empty()` failedESC[0m
ton-indexer-index-worker-1  | [pid 11] [time 1718819767] Signal: 6
ton-indexer-index-worker-1  | ------- Stack Backtrace -------
ton-indexer-index-worker-1  | ton-index-postgres(+0x35b162)[0x558c8f07f162]
ton-indexer-index-worker-1  | ton-index-postgres(+0x35b475)[0x558c8f07f475]
ton-indexer-index-worker-1  | ton-index-postgres(+0x35afbf)[0x558c8f07efbf]
ton-indexer-index-worker-1  | /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7f77644f2520]
ton-indexer-index-worker-1  | /lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x12c)[0x7f77645469fc]
ton-indexer-index-worker-1  | /lib/x86_64-linux-gnu/libc.so.6(raise+0x16)[0x7f77644f2476]
ton-indexer-index-worker-1  | /lib/x86_64-linux-gnu/libc.so.6(abort+0xd3)[0x7f77644d87f3]
ton-indexer-index-worker-1  | ton-index-postgres(+0x101e44)[0x558c8ee25e44]
ton-indexer-index-worker-1  | ton-index-postgres(+0x380ef9)[0x558c8f0a4ef9]
ton-indexer-index-worker-1  | ton-index-postgres(+0x378996)[0x558c8f09c996]
ton-indexer-index-worker-1  | ton-index-postgres(+0x366aba)[0x558c8f08aaba]
ton-indexer-index-worker-1  | ton-index-postgres(+0x4b2e39)[0x558c8f1d6e39]
ton-indexer-index-worker-1  | ton-index-postgres(+0x4b355a)[0x558c8f1d755a]
ton-indexer-index-worker-1  | ton-index-postgres(+0x398bd6)[0x558c8f0bcbd6]
ton-indexer-index-worker-1  | ton-index-postgres(+0x398e1d)[0x558c8f0bce1d]
ton-indexer-index-worker-1  | ton-index-postgres(+0x3a99e3)[0x558c8f0cd9e3]
ton-indexer-index-worker-1  | ton-index-postgres(+0x39c7df)[0x558c8f0c07df]
ton-indexer-index-worker-1  | ton-index-postgres(+0x2a1b5a)[0x558c8efc5b5a]
ton-indexer-index-worker-1  | /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3)[0x7f7764544ac3]
ton-indexer-index-worker-1  | /lib/x86_64-linux-gnu/libc.so.6(clone+0x44)[0x7f77645d5a04]
ton-indexer-index-worker-1  | -------------------------------
ton-indexer-index-worker-1  | Postgres password file: /run/secrets/postgres_password
ton-indexer-index-worker-1  | Postgres host: postgres (ip: 192.168.64.2)
ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 1][2024-06-19 17:56:09.900240451][InsertManagerBase.cpp:18][!insertmanager]       InsertManagerBase::start_up calledESC[0m
ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 1][2024-06-19 17:56:09.900333713][InsertManagerBase.cpp:29][!insertmanager]       Insert manager(parallel=12, max_batch_mc_blocks=2000, max_batch_blocks=2000, max_batch_txs=10000, max_batch_ms
gs=10000)ESC[0m
ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 2][2024-06-19 17:56:09.901292155][InsertManagerPostgres.cpp:1139][!insertmanager] Reading existing seqnosESC[0m
ton-indexer-index-worker-1  | ESC[1;31m[ 1][t 1][2024-06-19 17:56:10.013268917][archive-manager.cpp:876][!archive]      checking file /tondb/archive/states/state_38299753_0_4000000000000000_925603F00C0780DB90289DFB78E1A3431C2F270BAC46CBFF
F4BAEF1AAE710A58ESC[0m
ton-indexer-index-worker-1  | ESC[1;31m[ 1][t 1][2024-06-19 17:56:10.013371332][archive-manager.cpp:876][!archive]      checking file /tondb/archive/states/state_38210813_0_e000000000000000_DA24F9363055A874BA0EA1B6472264A94840FB70921994A1
D86B7DE1EE72FF52ESC[0m
ton-indexer-index-worker-1  | ESC[1;31m[ 1][t 1][2024-06-19 17:56:10.013384187][archive-manager.cpp:876][!archive]      checking file /tondb/archive/states/state_38210813_0_3000000000000000_4C0D11D42315B93C3640E14EF411DB1B7F5951F1FE70D21D
5822A73097DA5AB3ESC[0m
ton-indexer-index-worker-1  | ESC[1;31m[ 1][t 1][2024-06-19 17:56:10.013395483][archive-manager.cpp:876][!archive]      checking file /tondb/archive/states/state_38210813_0_1000000000000000_BD49D3F87D87302E68D076605F7B20F3B1EBE54BF0C213BC
B535D7E2D968BED4ESC[0m
ton-indexer-index-worker-1  | ESC[1;31m[ 1][t 1][2024-06-19 17:56:10.013412217][archive-manager.cpp:876][!archive]      checking file /tondb/archive/states/state_38299753_-1_8000000000000000_2B66D44A24ECD5BB77E5E4615BA0E606F501B24DB1C303A
2D5D45A603392BF49ESC[0m
ton-indexer-index-worker-1  | ESC[1;31m[ 1][t 1][2024-06-19 17:56:10.013423607][archive-manager.cpp:876][!archive]      checking file /tondb/archive/states/state_38210813_-1_8000000000000000_D0E749AE414202B2E84F12366BA9D08FD974546CBF5A54F
6469C9E04B86E128DESC[0m
ton-indexer-index-worker-1  | ESC[1;31m[ 1][t 1][2024-06-19 17:56:10.013433569][archive-manager.cpp:876][!archive]      checking file /tondb/archive/states/state_38210813_0_9000000000000000_9D9A652D40AFAD09DD75F4F887EBE415230BDC9CBAD8E929
5BE83C3BA55D8E89ESC[0m
ton-indexer-index-worker-1  | ESC[1;31m[ 1][t 1][2024-06-19 17:56:10.013444152][archive-manager.cpp:876][!archive]      checking file /tondb/archive/states/state_38210813_0_b000000000000000_37FC5B27D643323304670205D967852E35231B291BDDF1C7
8702E8B457DA4D67ESC[0m
ton-indexer-index-worker-1  | ESC[1;31m[ 1][t 1][2024-06-19 17:56:10.013453834][archive-manager.cpp:876][!archive]      checking file /tondb/archive/states/state_38210813_0_6000000000000000_199B4DE9A88F3741D972F2DB3FB8EE70E07CF8748BC49563
5DC91C27C999DE69ESC[0m
ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 2][2024-06-19 17:56:10.035842339][IndexScheduler.cpp:80][!indexscheduler] Found 115285 existing seqnosESC[0m
ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 7][2024-06-19 17:56:11.036855540][IndexScheduler.cpp:172][!indexscheduler]        Last: 0 / 38354000      Blk/s: 0.000000 ETA: -8s        Q[0M, 0b, 0t, 0m]ESC[0m
ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 7][2024-06-19 17:56:11.047042450][IndexScheduler.cpp:95][!indexscheduler] Skipped 115285 existing seqnosESC[0m
ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 7][2024-06-19 17:56:11.047115112][IndexScheduler.cpp:211][!indexscheduler]        Syncronization complete!ESC[0m
ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 1][2024-06-19 17:56:21.041446861][IndexScheduler.cpp:172][!indexscheduler]        Last: 0 / 38469325      Blk/s: 0.000000 ETA: -8s        Q[0M, 0b, 0t, 0m]ESC[0m
ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 3][2024-06-19 17:56:31.046156802][IndexScheduler.cpp:172][!indexscheduler]        Last: 38469288 / 38469329       Blk/s: 0.252000 ETA: 2m 42s     Q[0M, 0b, 0t, 0m]ESC[0m
ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 6][2024-06-19 17:56:41.052334993][IndexScheduler.cpp:172][!indexscheduler]        Last: 38469289 / 38469332       Blk/s: 0.135697 ETA: 5m 16s     Q[0M, 0b, 0t, 0m]ESC[0m

[dnjef9duo]

here's another case in same day.

ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 6][2024-06-19 19:18:07.311817206][IndexScheduler.cpp:172][!indexscheduler]        Last: 38470413 / 38470568       Blk/s: 0.132504 ETA: 19m 29s    Q[0M, 0b, 0t, 0m]ESC[0m
ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 7][2024-06-19 19:18:17.323430654][IndexScheduler.cpp:172][!indexscheduler]        Last: 38470415 / 38470572       Blk/s: 0.184031 ETA: 14m 13s    Q[0M, 0b, 0t, 0m]ESC[0m
ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 7][2024-06-19 19:18:27.335369615][IndexScheduler.cpp:172][!indexscheduler]        Last: 38470416 / 38470573       Blk/s: 0.117312 ETA: 22m 18s    Q[1M, 3b, 342t, 591m]ESC[0m
ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 2][2024-06-19 19:18:37.347217737][IndexScheduler.cpp:172][!indexscheduler]        Last: 38470421 / 38470574       Blk/s: 0.396144 ETA: 6m 26s     Q[2M, 4b, 288t, 507m]ESC[0m
ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 6][2024-06-19 19:18:47.359880756][IndexScheduler.cpp:172][!indexscheduler]        Last: 38470421 / 38470576       Blk/s: 0.138127 ETA: 18m 42s    Q[0M, 0b, 0t, 0m]ESC[0m
ton-indexer-index-worker-1  | [pid 11] [time 1718824735] Signal: 11
ton-indexer-index-worker-1  | ------- Stack Backtrace -------
ton-indexer-index-worker-1  | ton-index-postgres(+0x35b162)[0x55659bfde162]
ton-indexer-index-worker-1  | ton-index-postgres(+0x35b475)[0x55659bfde475]
ton-indexer-index-worker-1  | ton-index-postgres(+0x35afbf)[0x55659bfddfbf]
ton-indexer-index-worker-1  | /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7fc306d5f520]
ton-indexer-index-worker-1  | ton-index-postgres(+0x6fdd46)[0x55659c380d46]
ton-indexer-index-worker-1  | ton-index-postgres(+0x4dc180)[0x55659c15f180]
ton-indexer-index-worker-1  | ton-index-postgres(+0x4b3456)[0x55659c136456]
ton-indexer-index-worker-1  | ton-index-postgres(+0x398bd6)[0x55659c01bbd6]
ton-indexer-index-worker-1  | ton-index-postgres(+0x398e1d)[0x55659c01be1d]
ton-indexer-index-worker-1  | ton-index-postgres(+0x3a99e3)[0x55659c02c9e3]
ton-indexer-index-worker-1  | ton-index-postgres(+0x39c7df)[0x55659c01f7df]
ton-indexer-index-worker-1  | ton-index-postgres(+0x2a1b5a)[0x55659bf24b5a]
ton-indexer-index-worker-1  | /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3)[0x7fc306db1ac3]
ton-indexer-index-worker-1  | /lib/x86_64-linux-gnu/libc.so.6(clone+0x44)[0x7fc306e42a04]
ton-indexer-index-worker-1  | -------------------------------
ton-indexer-index-worker-1  | Postgres password file: /run/secrets/postgres_password
ton-indexer-index-worker-1  | Postgres host: postgres (ip: 192.168.64.2)
ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 1][2024-06-19 19:18:56.961869763][InsertManagerBase.cpp:18][!insertmanager]       InsertManagerBase::start_up calledESC[0m
ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 1][2024-06-19 19:18:56.961971428][InsertManagerBase.cpp:29][!insertmanager]       Insert manager(parallel=12, max_batch_mc_blocks=2000, max_batch_blocks=2000, max_batch_txs=10000, max_batch_ms
gs=10000)ESC[0m
ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 2][2024-06-19 19:18:56.964037480][InsertManagerPostgres.cpp:1139][!insertmanager] Reading existing seqnosESC[0m
ton-indexer-index-worker-1  | ESC[1;31m[ 1][t 1][2024-06-19 19:18:57.025943299][archive-manager.cpp:876][!archive]      checking file /tondb/archive/states/state_38299753_0_4000000000000000_925603F00C0780DB90289DFB78E1A3431C2F270BAC46CBFF
F4BAEF1AAE710A58ESC[0m
ton-indexer-index-worker-1  | ESC[1;31m[ 1][t 1][2024-06-19 19:18:57.026020115][archive-manager.cpp:876][!archive]      checking file /tondb/archive/states/state_38210813_0_e000000000000000_DA24F9363055A874BA0EA1B6472264A94840FB70921994A1
D86B7DE1EE72FF52ESC[0m
ton-indexer-index-worker-1  | ESC[1;31m[ 1][t 1][2024-06-19 19:18:57.026034345][archive-manager.cpp:876][!archive]      checking file /tondb/archive/states/state_38210813_0_3000000000000000_4C0D11D42315B93C3640E14EF411DB1B7F5951F1FE70D21D
5822A73097DA5AB3ESC[0m
ton-indexer-index-worker-1  | ESC[1;31m[ 1][t 1][2024-06-19 19:18:57.026044682][archive-manager.cpp:876][!archive]      checking file /tondb/archive/states/state_38210813_0_1000000000000000_BD49D3F87D87302E68D076605F7B20F3B1EBE54BF0C213BC
B535D7E2D968BED4ESC[0m
ton-indexer-index-worker-1  | ESC[1;31m[ 1][t 1][2024-06-19 19:18:57.026053299][archive-manager.cpp:876][!archive]      checking file /tondb/archive/states/state_38299753_-1_8000000000000000_2B66D44A24ECD5BB77E5E4615BA0E606F501B24DB1C303A
2D5D45A603392BF49ESC[0m
ton-indexer-index-worker-1  | ESC[1;31m[ 1][t 1][2024-06-19 19:18:57.026061551][archive-manager.cpp:876][!archive]      checking file /tondb/archive/states/state_38210813_-1_8000000000000000_D0E749AE414202B2E84F12366BA9D08FD974546CBF5A54F
6469C9E04B86E128DESC[0m
ton-indexer-index-worker-1  | ESC[1;31m[ 1][t 1][2024-06-19 19:18:57.026074456][archive-manager.cpp:876][!archive]      checking file /tondb/archive/states/state_38210813_0_9000000000000000_9D9A652D40AFAD09DD75F4F887EBE415230BDC9CBAD8E929
5BE83C3BA55D8E89ESC[0m
ton-indexer-index-worker-1  | ESC[1;31m[ 1][t 1][2024-06-19 19:18:57.026087148][archive-manager.cpp:876][!archive]      checking file /tondb/archive/states/state_38210813_0_b000000000000000_37FC5B27D643323304670205D967852E35231B291BDDF1C7
8702E8B457DA4D67ESC[0m
ton-indexer-index-worker-1  | ESC[1;31m[ 1][t 1][2024-06-19 19:18:57.026105759][archive-manager.cpp:876][!archive]      checking file /tondb/archive/states/state_38210813_0_6000000000000000_199B4DE9A88F3741D972F2DB3FB8EE70E07CF8748BC49563
5DC91C27C999DE69ESC[0m
ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 2][2024-06-19 19:18:57.123912698][IndexScheduler.cpp:80][!indexscheduler] Found 116423 existing seqnosESC[0m
ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 2][2024-06-19 19:18:58.124357089][IndexScheduler.cpp:172][!indexscheduler]        Last: 0 / 38354000      Blk/s: 0.000000 ETA: -8s        Q[0M, 0b, 0t, 0m]ESC[0m
ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 2][2024-06-19 19:18:58.136064095][IndexScheduler.cpp:95][!indexscheduler] Skipped 116423 existing seqnosESC[0m
ton-indexer-index-worker-1  | ESC[1;33m[ 2][t 3][2024-06-19 19:19:06.648128784][InterfaceDetectors.hpp:717][!nft_collection_detector]   Failed to parse collection content for 0:FD8C0CF95B84F3A6816D624E41840AA62C84CA85C57F9A290869E7FA12BC5
2AD: [Error : 0 : Unknown token data type]ESC[0m
ton-indexer-index-worker-1  | ESC[1;33m[ 2][t 3][2024-06-19 19:19:06.648193741][InterfaceDetectors.hpp:718][!nft_collection_detector]   te6cckEBAQEANwAAamlwZnM6Ly9RbWU2dU1MZHZYbVN1TUFQb3ZrY3E3bW83TFZvUFV3bmhrQ21MQXhhYThSRlVx3rPonA==ESC[0m
ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 4][2024-06-19 19:19:08.130576114][IndexScheduler.cpp:172][!indexscheduler]        Last: 0 / 38470579      Blk/s: 0.000000 ETA: -8s        Q[0M, 0b, 0t, 0m]ESC[0m
ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 2][2024-06-19 19:19:18.135292318][IndexScheduler.cpp:172][!indexscheduler]        Last: 38470425 / 38470583       Blk/s: 0.077484 ETA: 33m 59s    Q[0M, 0b, 0t, 0m]ESC[0m
ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 5][2024-06-19 19:19:28.140136987][IndexScheduler.cpp:172][!indexscheduler]        Last: 38470425 / 38470586       Blk/s: 0.027017 ETA: 1h 39m 19s Q[0M, 0b, 0t, 0m]ESC[0m
ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 5][2024-06-19 19:19:38.146084122][IndexScheduler.cpp:172][!indexscheduler]        Last: 38470425 / 38470588       Blk/s: 0.009420 ETA: 4h 48m 23s Q[0M, 0b, 0t, 0m]ESC[0m
ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 5][2024-06-19 19:19:48.150923460][IndexScheduler.cpp:172][!indexscheduler]        Last: 38470425 / 38470591       Blk/s: 0.003285 ETA: 14h 2m 18s Q[0M, 0b, 0t, 0m]ESC[0m
ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 6][2024-06-19 19:19:58.155169442][IndexScheduler.cpp:172][!indexscheduler]        Last: 38470425 / 38470592       Blk/s: 0.001145 ETA: 1d 16h 30m 15s     Q[0M, 0b, 0t, 0m]ESC[0m
ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 7][2024-06-19 19:20:08.160238788][IndexScheduler.cpp:172][!indexscheduler]        Last: 38470425 / 38470595       Blk/s: 0.000399 ETA: 4d 22h 15m 7s      Q[0M, 0b, 0t, 0m]ESC[0m
ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 1][2024-06-19 19:20:18.164655788][IndexScheduler.cpp:172][!indexscheduler]        Last: 38470425 / 38470597       Blk/s: 0.000139 ETA: 14d 7h 7m 59s      Q[0M, 0b, 0t, 0m]ESC[0m
ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 2][2024-06-19 19:20:28.170366672][IndexScheduler.cpp:172][!indexscheduler]        Last: 38470425 / 38470598       Blk/s: 0.000049 ETA: 41d 5h 49m 4s      Q[0M, 0b, 0t, 0m]ESC[0m

[kdimentionaltree]

Hello, from your logs I see a successful restart of the worker. Could you please check if restart occurs due to Out Of Memory?

[dnjef9duo]

i think there is no OOM. there is no kind of message shows OOM or kill or something like that in system log and here is the monitor chart. and we have 128g for ton node.

[dnjef9duo]

unfortunately it happened again, quite similar logs. can provide a full log (38MB, 6hours before docker restart) if needed.

8 times in 6 hours, after indexer auto restart, it syncing very low or even stop syncing.

2 cases of index-worker auto restart. case1: signal 6

ton-indexer-index-worker-1  | ESC[1;31m[ 0][t 2][2024-06-25 19:54:54.409955543][ActorOwn.h:80][!slice]  Check `!empty()` failedESC[0m
ton-indexer-index-worker-1  | [pid 11] [time 1719345294] Signal: 6
ton-indexer-index-worker-1  | ------- Stack Backtrace -------
ton-indexer-index-worker-1  | ton-index-postgres(+0x35b162)[0x56398193e162]
ton-indexer-index-worker-1  | ton-index-postgres(+0x35b475)[0x56398193e475]
ton-indexer-index-worker-1  | ton-index-postgres(+0x35afbf)[0x56398193dfbf]
ton-indexer-index-worker-1  | /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7f114888e520]
ton-indexer-index-worker-1  | /lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x12c)[0x7f11488e29fc]
ton-indexer-index-worker-1  | /lib/x86_64-linux-gnu/libc.so.6(raise+0x16)[0x7f114888e476]
ton-indexer-index-worker-1  | /lib/x86_64-linux-gnu/libc.so.6(abort+0xd3)[0x7f11488747f3]
ton-indexer-index-worker-1  | ton-index-postgres(+0x101e44)[0x5639816e4e44]
ton-indexer-index-worker-1  | ton-index-postgres(+0x380ef9)[0x563981963ef9]
ton-indexer-index-worker-1  | ton-index-postgres(+0x378996)[0x56398195b996]
ton-indexer-index-worker-1  | ton-index-postgres(+0x366aba)[0x563981949aba]
ton-indexer-index-worker-1  | ton-index-postgres(+0x4b2e39)[0x563981a95e39]
ton-indexer-index-worker-1  | ton-index-postgres(+0x4b355a)[0x563981a9655a]
ton-indexer-index-worker-1  | ton-index-postgres(+0x398bd6)[0x56398197bbd6]
ton-indexer-index-worker-1  | ton-index-postgres(+0x398e1d)[0x56398197be1d]
ton-indexer-index-worker-1  | ton-index-postgres(+0x3a99e3)[0x56398198c9e3]
ton-indexer-index-worker-1  | ton-index-postgres(+0x39c7df)[0x56398197f7df]
ton-indexer-index-worker-1  | ton-index-postgres(+0x2a1b5a)[0x563981884b5a]
ton-indexer-index-worker-1  | /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3)[0x7f11488e0ac3]
ton-indexer-index-worker-1  | /lib/x86_64-linux-gnu/libc.so.6(clone+0x44)[0x7f1148971a04]
ton-indexer-index-worker-1  | -------------------------------
ton-indexer-index-worker-1  | Postgres password file: /run/secrets/postgres_password
ton-indexer-index-worker-1  | Postgres host: postgres (ip: 192.168.96.2)
ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 1][2024-06-25 19:54:55.781090741][InsertManagerBase.cpp:18][!insertmanager]       InsertManagerBase::start_up calledESC[0m
ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 1][2024-06-25 19:54:55.781210067][InsertManagerBase.cpp:29][!insertmanager]       Insert manager(parallel=12, max_batch_mc_blocks=2000, max_batch_blocks=2000, max_batch_txs=10000, max_batch_msgs=10000)ESC[0m
ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 2][2024-06-25 19:54:55.782887243][InsertManagerPostgres.cpp:1139][!insertmanager] Reading existing seqnosESC[0m

case 2: signal 11

ton-indexer-index-worker-1  | ESC[1;31m[ 1][t 3][2024-06-25 19:34:56.391172836][IndexScheduler.cpp:106][!indexquery]    Failed to fetch seqno 38599997: [Error : 651 : (0,2000000000000000,442240
85): ltdb: block not found]ESC[0m
ton-indexer-index-worker-1  | ESC[1;31m[ 1][t 1][2024-06-25 19:34:56.391177588][IndexScheduler.cpp:106][!indexquery]    Failed to fetch seqno 38599998: [Error : 651 : (0,1000000000000000,442240
86): ltdb: block not found]ESC[0m
ton-indexer-index-worker-1  | ESC[1;33m[ 2][t 3][2024-06-25 19:34:56.391225825][IndexScheduler.cpp:117][!indexscheduler]        Rescheduling seqno 38599997ESC[0m
ton-indexer-index-worker-1  | ESC[1;33m[ 2][t 3][2024-06-25 19:34:56.391234462][IndexScheduler.cpp:117][!indexscheduler]        Rescheduling seqno 38599998ESC[0m
ton-indexer-index-worker-1  | ESC[1;31m[ 1][t 7][2024-06-25 19:34:56.391699328][IndexScheduler.cpp:106][!indexquery]    Failed to fetch seqno 38599995: [Error : 651 : (0,2000000000000000,442240
85): ltdb: block not found]ESC[0m
ton-indexer-index-worker-1  | ESC[1;33m[ 2][t 7][2024-06-25 19:34:56.391741690][IndexScheduler.cpp:117][!indexscheduler]        Rescheduling seqno 38599995ESC[0m
ton-indexer-index-worker-1  | [pid 11] [time 1719344096] Signal: 11
ton-indexer-index-worker-1  | ------- Stack Backtrace -------
ton-indexer-index-worker-1  | ton-index-postgres(+0x35b162)[0x560e5a75f162]
ton-indexer-index-worker-1  | ton-index-postgres(+0x35b475)[0x560e5a75f475]
ton-indexer-index-worker-1  | ton-index-postgres(+0x35afbf)[0x560e5a75efbf]
ton-indexer-index-worker-1  | /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7f14c0c4a520]
ton-indexer-index-worker-1  | ton-index-postgres(+0x6fdd46)[0x560e5ab01d46]
ton-indexer-index-worker-1  | ton-index-postgres(+0x4dc180)[0x560e5a8e0180]
ton-indexer-index-worker-1  | ton-index-postgres(+0x4b3456)[0x560e5a8b7456]
ton-indexer-index-worker-1  | ton-index-postgres(+0x398bd6)[0x560e5a79cbd6]
ton-indexer-index-worker-1  | ton-index-postgres(+0x398e1d)[0x560e5a79ce1d]
ton-indexer-index-worker-1  | ton-index-postgres(+0x3a99e3)[0x560e5a7ad9e3]
ton-indexer-index-worker-1  | ton-index-postgres(+0x39c7df)[0x560e5a7a07df]
ton-indexer-index-worker-1  | ton-index-postgres(+0x2a1b5a)[0x560e5a6a5b5a]
ton-indexer-index-worker-1  | /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3)[0x7f14c0c9cac3]
ton-indexer-index-worker-1  | /lib/x86_64-linux-gnu/libc.so.6(clone+0x44)[0x7f14c0d2da04]
ton-indexer-index-worker-1  | -------------------------------
ton-indexer-index-worker-1  | Postgres password file: /run/secrets/postgres_password
ton-indexer-index-worker-1  | Postgres host: postgres (ip: 192.168.96.2)
ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 1][2024-06-25 19:34:58.491999860][InsertManagerBase.cpp:18][!insertmanager]       InsertManagerBase::start_up calledESC[0m
ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 1][2024-06-25 19:34:58.492225239][InsertManagerBase.cpp:29][!insertmanager]       Insert manager(parallel=12, max_batch_mc_blocks=2000, max_batch_b
locks=2000, max_batch_txs=10000, max_batch_msgs=10000)ESC[0m
ton-indexer-index-worker-1  | ESC[1;36m[ 3][t 1][2024-06-25 19:34:58.492252719][InsertManagerPostgres.cpp:1139][!insertmanager] Reading existing seqnosESC[0m

[dungeon-master-666]

Hello, thanks for reporting. We've made several significant improvements to the stability of the index worker. Updating the repo to the latest master (incl submodules) should help. Feel free to reopen if you encounter a crash after that.

[dnjef9duo]

Great! new commit! let me try later, thanks.

[dnjef9duo]

unfortunately, seems still same issue after upgrade.

it happened on testnet this time. 4 times restart in 12hours, and the final one it stop syncing. and mainent looks good so far. will keep monitoring.

i have upgrade by following command

git pull.
git submodule update --recursive --init

and now on this commit commit 15888e88dc571c0efbdedd73be2123b8be8de872 (HEAD -> master, origin/master, origin/HEAD)

and here is the dump logs looks sometimes it can sync up automatically after restart, sometimes not.

index-worker-1  | ESC[1;33m[ 2][t 3][2024-07-08 14:16:08.619524722][InterfaceDetectors.hpp:255][!jetton_master_detector]        Failed to parse jetton content for 0:3064595B098F76870997FFF7D0FA49F6F8192F86856CB03BEF60B9077D27CB5C: [Error
: 0 : Unknown token data type]ESC[0m
index-worker-1  | ESC[1;33m[ 2][t 3][2024-07-08 14:16:08.619569527][InterfaceDetectors.hpp:256][!jetton_master_detector]        te6cckEBAgEAqAAB/lRFU1QtVEFYLTU1dGVzdCBkZSBqZXR0b24gY29uIHRheFRFU1RGNTVodHRwczovL3Jhdy5naXRodWJ1c2VyY29udGVudC
5jb20vcG9sYXJ5c2ZvdW5kYXRpb24vbG9nb19jb2luX3Rlc3QvZmI0NDMzNDFiZWY1ODFkNWZkY2YBAEhkNmI0YWUyYmZhNmFlNTQ0YjExNS9idXlib3QtbG9nby5wbmfBVJSJESC[0m
index-worker-1  | ESC[1;36m[ 3][t 7][2024-07-08 14:16:11.543344235][IndexScheduler.cpp:172][!indexscheduler]    Last: 20850897 / 20850897       Blk/s: 0.402578 ETA: 0s Q[0M, 0b, 0t, 0m]ESC[0m
index-worker-1  | ESC[1;36m[ 3][t 6][2024-07-08 14:16:21.555240185][IndexScheduler.cpp:172][!indexscheduler]    Last: 20850900 / 20850901       Blk/s: 0.343810 ETA: 2s Q[1M, 5b, 3t, 1m]ESC[0m
index-worker-1  | ESC[1;33m[ 2][t 1][2024-07-08 14:16:23.571451350][InterfaceDetectors.hpp:255][!jetton_master_detector]        Failed to parse jetton content for 0:FC39ED78AFEF008478E5637B2563F10425C8D9F5E16C21B3F09D7033E89AFBA5: [Error
: 0 : Unknown token data type]ESC[0m
index-worker-1  | ESC[1;33m[ 2][t 1][2024-07-08 14:16:23.571490550][InterfaceDetectors.hpp:256][!jetton_master_detector]        te6cckEBAQEAAgAAAEysuc0=ESC[0m
index-worker-1  | ESC[1;36m[ 3][t 6][2024-07-08 14:16:31.567076644][IndexScheduler.cpp:172][!indexscheduler]    Last: 20850904 / 20850905       Blk/s: 0.365665 ETA: 2s Q[0M, 0b, 0t, 0m]ESC[0m
index-worker-1  | ESC[1;36m[ 3][t 7][2024-07-08 14:16:41.578981728][IndexScheduler.cpp:172][!indexscheduler]    Last: 20850908 / 20850908       Blk/s: 0.364986 ETA: 0s Q[0M, 0b, 0t, 0m]ESC[0m
index-worker-1  | ESC[1;36m[ 3][t 3][2024-07-08 14:16:51.590673248][IndexScheduler.cpp:172][!indexscheduler]    Last: 20850912 / 20850912       Blk/s: 0.407041 ETA: 0s Q[1M, 5b, 4t, 3m]ESC[0m
index-worker-1  | ESC[1;36m[ 3][t 4][2024-07-08 14:17:01.601429450][IndexScheduler.cpp:172][!indexscheduler]    Last: 20850916 / 20850916       Blk/s: 0.409117 ETA: 0s Q[0M, 0b, 0t, 0m]ESC[0m
index-worker-1  | [pid 11] [time 1720448223] Signal: 11
index-worker-1  | ------- Stack Backtrace -------
index-worker-1  | ton-index-postgres(+0x35b162)[0x558ac4899162]
index-worker-1  | ton-index-postgres(+0x35b475)[0x558ac4899475]
index-worker-1  | ton-index-postgres(+0x35afbf)[0x558ac4898fbf]
index-worker-1  | /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7fcecb30f520]
index-worker-1  | ton-index-postgres(+0x6fdd46)[0x558ac4c3bd46]
index-worker-1  | ton-index-postgres(+0x4dc180)[0x558ac4a1a180]
index-worker-1  | ton-index-postgres(+0x4b3456)[0x558ac49f1456]
index-worker-1  | ton-index-postgres(+0x398bd6)[0x558ac48d6bd6]
index-worker-1  | ton-index-postgres(+0x398e1d)[0x558ac48d6e1d]
index-worker-1  | ton-index-postgres(+0x3a99e3)[0x558ac48e79e3]
index-worker-1  | ton-index-postgres(+0x39c7df)[0x558ac48da7df]
index-worker-1  | ton-index-postgres(+0x2a1b5a)[0x558ac47dfb5a]
index-worker-1  | /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3)[0x7fcecb361ac3]
index-worker-1  | /lib/x86_64-linux-gnu/libc.so.6(clone+0x44)[0x7fcecb3f2a04]
index-worker-1  | -------------------------------
index-worker-1  | Postgres password file: /run/secrets/postgres_password
index-worker-1  | Postgres host: postgres (ip: 172.26.0.2)
index-worker-1  | ESC[1;36m[ 3][t 1][2024-07-08 14:17:05.711713935][InsertManagerBase.cpp:18][!insertmanager]   InsertManagerBase::start_up calledESC[0m
index-worker-1  | ESC[1;36m[ 3][t 1][2024-07-08 14:17:05.711788310][InsertManagerBase.cpp:29][!insertmanager]   Insert manager(parallel=12, max_batch_mc_blocks=2000, max_batch_blocks=2000, max_batch_txs=10000, max_batch_msgs=10000)ESC[0m
index-worker-1  | ESC[1;36m[ 3][t 3][2024-07-08 14:17:05.712624035][InsertManagerPostgres.cpp:1139][!insertmanager]     Reading existing seqnosESC[0m
index-worker-1  | ESC[1;36m[ 3][t 3][2024-07-08 14:17:05.744343829][IndexScheduler.cpp:80][!indexscheduler]     Found 5730 existing seqnosESC[0m
index-worker-1  | ESC[1;36m[ 3][t 4][2024-07-08 14:17:06.745314942][IndexScheduler.cpp:172][!indexscheduler]    Last: 0 / 20845187      Blk/s: 0.000000 ETA: -8s        Q[0M, 0b, 0t, 0m]ESC[0m
index-worker-1  | ESC[1;36m[ 3][t 4][2024-07-08 14:17:06.745817508][IndexScheduler.cpp:95][!indexscheduler]     Skipped 5729 existing seqnosESC[0m
index-worker-1  | ESC[1;36m[ 3][t 4][2024-07-08 14:17:06.745856652][IndexScheduler.cpp:211][!indexscheduler]    Syncronization complete!ESC[0m
index-worker-1  | ESC[1;36m[ 3][t 2][2024-07-08 14:17:16.747803050][IndexScheduler.cpp:172][!indexscheduler]    Last: 20850920 / 20850920       Blk/s: 0.281974 ETA: 0s Q[0M, 0b, 0t, 0m]ESC[0m
index-worker-1  | ESC[1;36m[ 3][t 7][2024-07-08 14:17:26.751430171][IndexScheduler.cpp:172][!indexscheduler]    Last: 20850924 / 20850924       Blk/s: 0.387975 ETA: 0s Q[0M, 0b, 0t, 0m]ESC[0m
index-worker-1  | ESC[1;36m[ 3][t 2][2024-07-08 14:17:36.754820711][IndexScheduler.cpp:172][!indexscheduler]    Last: 20850926 / 20850927       Blk/s: 0.275327 ETA: 3s Q[1M, 5b, 5t, 8m]ESC[0m

index-worker-1  | ESC[1;36m[ 3][t 3][2024-07-08 18:15:42.265280935][IndexScheduler.cpp:172][!indexscheduler]    Last: 20855531 / 20856172       Blk/s: 0.286455 ETA: 37m 17s    Q[0M, 0b, 0t, 0m]ESC[0m
index-worker-1  | ESC[1;36m[ 3][t 7][2024-07-08 18:15:52.276927661][IndexScheduler.cpp:172][!indexscheduler]    Last: 20855532 / 20856176       Blk/s: 0.172781 ETA: 1h 2m 7s   Q[0M, 0b, 0t, 0m]ESC[0m
index-worker-1  | ESC[1;36m[ 3][t 1][2024-07-08 18:16:02.288799279][IndexScheduler.cpp:172][!indexscheduler]    Last: 20855533 / 20856179       Blk/s: 0.160245 ETA: 1h 7m 11s  Q[0M, 0b, 0t, 0m]ESC[0m
index-worker-1  | ESC[1;36m[ 3][t 6][2024-07-08 18:16:12.300821813][IndexScheduler.cpp:172][!indexscheduler]    Last: 20855536 / 20856183       Blk/s: 0.242065 ETA: 44m 32s    Q[0M, 0b, 0t, 0m]ESC[0m
index-worker-1  | ESC[1;36m[ 3][t 2][2024-07-08 18:16:22.313089972][IndexScheduler.cpp:172][!indexscheduler]    Last: 20855538 / 20856188       Blk/s: 0.238303 ETA: 45m 27s    Q[0M, 0b, 0t, 0m]ESC[0m
index-worker-1  | ESC[1;36m[ 3][t 7][2024-07-08 18:16:32.325003581][IndexScheduler.cpp:172][!indexscheduler]    Last: 20855544 / 20856192       Blk/s: 0.414984 ETA: 26m 1s     Q[0M, 0b, 0t, 0m]ESC[0m
index-worker-1  | ESC[1;36m[ 3][t 2][2024-07-08 18:16:42.336201429][IndexScheduler.cpp:172][!indexscheduler]    Last: 20855546 / 20856196       Blk/s: 0.327792 ETA: 33m 2s     Q[0M, 0b, 0t, 0m]ESC[0m
index-worker-1  | ESC[1;36m[ 3][t 6][2024-07-08 18:16:52.347707469][IndexScheduler.cpp:172][!indexscheduler]    Last: 20855549 / 20856198       Blk/s: 0.367438 ETA: 29m 26s    Q[0M, 0b, 0t, 0m]ESC[0m
index-worker-1  | ESC[1;36m[ 3][t 6][2024-07-08 18:17:02.359024286][IndexScheduler.cpp:172][!indexscheduler]    Last: 20855550 / 20856202       Blk/s: 0.201018 ETA: 54m 3s     Q[1M, 5b, 6t, 7m]ESC[0m
index-worker-1  | [pid 11] [time 1720462626] Signal: 11
index-worker-1  | ------- Stack Backtrace -------
index-worker-1  | ton-index-postgres(+0x35b162)[0x556df34fb162]
index-worker-1  | ton-index-postgres(+0x35b475)[0x556df34fb475]
index-worker-1  | ton-index-postgres(+0x35afbf)[0x556df34fafbf]
index-worker-1  | /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7f06997b8520]
index-worker-1  | ton-index-postgres(+0x6fdd46)[0x556df389dd46]
index-worker-1  | ton-index-postgres(+0x4dc180)[0x556df367c180]
index-worker-1  | ton-index-postgres(+0x4b3456)[0x556df3653456]
index-worker-1  | ton-index-postgres(+0x398bd6)[0x556df3538bd6]
index-worker-1  | ton-index-postgres(+0x398e1d)[0x556df3538e1d]
index-worker-1  | ton-index-postgres(+0x3a99e3)[0x556df35499e3]
index-worker-1  | ton-index-postgres(+0x39c7df)[0x556df353c7df]
index-worker-1  | ton-index-postgres(+0x2a1b5a)[0x556df3441b5a]
index-worker-1  | /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3)[0x7f069980aac3]
index-worker-1  | /lib/x86_64-linux-gnu/libc.so.6(clone+0x44)[0x7f069989ba04]
index-worker-1  | -------------------------------
index-worker-1  | Postgres password file: /run/secrets/postgres_password
index-worker-1  | Postgres host: postgres (ip: 172.26.0.2)
index-worker-1  | ESC[1;36m[ 3][t 1][2024-07-08 18:17:08.249014976][InsertManagerBase.cpp:18][!insertmanager]   InsertManagerBase::start_up calledESC[0m
index-worker-1  | ESC[1;36m[ 3][t 1][2024-07-08 18:17:08.249094377][InsertManagerBase.cpp:29][!insertmanager]   Insert manager(parallel=12, max_batch_mc_blocks=2000, max_batch_blocks=2000, max_batch_txs=10000, max_batch_msgs=10000)ESC[0m
index-worker-1  | ESC[1;36m[ 3][t 1][2024-07-08 18:17:08.249126664][InsertManagerPostgres.cpp:1139][!insertmanager]     Reading existing seqnosESC[0m
index-worker-1  | ESC[1;36m[ 3][t 1][2024-07-08 18:17:08.291694249][IndexScheduler.cpp:80][!indexscheduler]     Found 10366 existing seqnosESC[0m
index-worker-1  | ESC[1;36m[ 3][t 5][2024-07-08 18:17:09.292483794][IndexScheduler.cpp:172][!indexscheduler]    Last: 0 / 20845187      Blk/s: 0.000000 ETA: -8s        Q[0M, 0b, 0t, 0m]ESC[0m
index-worker-1  | ESC[1;36m[ 3][t 5][2024-07-08 18:17:09.293177207][IndexScheduler.cpp:95][!indexscheduler]     Skipped 10365 existing seqnosESC[0m
index-worker-1  | ESC[1;36m[ 3][t 7][2024-07-08 18:17:19.296936863][IndexScheduler.cpp:172][!indexscheduler]    Last: 20855565 / 20856208       Blk/s: 0.602654 ETA: 17m 46s    Q[0M, 0b, 0t, 0m]ESC[0m
index-worker-1  | ESC[1;36m[ 3][t 6][2024-07-08 18:17:29.299921635][IndexScheduler.cpp:172][!indexscheduler]    Last: 20855565 / 20856212       Blk/s: 0.210132 ETA: 51m 19s    Q[0M, 0b, 0t, 0m]ESC[0m
index-worker-1  | ESC[1;36m[ 3][t 1][2024-07-08 18:17:39.305268041][IndexScheduler.cpp:172][!indexscheduler]    Last: 20855565 / 20856215       Blk/s: 0.073269 ETA: 2h 27m 51s Q[0M, 0b, 0t, 0m]ESC[0m
index-worker-1  | ESC[1;36m[ 3][t 6][2024-07-08 18:17:49.314950353][IndexScheduler.cpp:172][!indexscheduler]    Last: 20855565 / 20856218       Blk/s: 0.025547 ETA: 7h 6m 0s   Q[0M, 0b, 0t, 0m]ESC[0m
index-worker-1  | ESC[1;36m[ 3][t 5][2024-07-08 18:17:59.326237798][IndexScheduler.cpp:172][!indexscheduler]    Last: 20855565 / 20856221       Blk/s: 0.008908 ETA: 20h 27m 23s        Q[0M, 0b, 0t, 0m]ESC[0m
index-worker-1  | ESC[1;36m[ 3][t 6][2024-07-08 18:18:09.337115944][IndexScheduler.cpp:172][!indexscheduler]    Last: 20855565 / 20856226       Blk/s: 0.003106 ETA: 2d 11h 6m 57s      Q[0M, 0b, 0t, 0m]ESC[0m

[lcgogo]

same issue

[ 1][t 4][2024-08-12 04:13:48.390177153][IndexScheduler.cpp:104][!indexquery]   Failed to fetch seqno 39634454: [Error : 651 : (0,8000000000000000,45056665): ltdb: block not found]
[ 2][t 5][2024-08-12 04:13:48.390307797][IndexScheduler.cpp:115][!indexscheduler]   Rescheduling seqno 39634476
[ 2][t 5][2024-08-12 04:13:48.390317497][IndexScheduler.cpp:115][!indexscheduler]   Rescheduling seqno 39634374
[ 2][t 5][2024-08-12 04:13:48.390321677][IndexScheduler.cpp:115][!indexscheduler]   Rescheduling seqno 39634454
[ 1][t 6][2024-08-12 04:13:48.390546074][IndexScheduler.cpp:104][!indexquery]   Failed to fetch seqno 39634398: [Error : 651 : (0,8000000000000000,45056626): ltdb: block not found]
[ 2][t 6][2024-08-12 04:13:48.390571114][IndexScheduler.cpp:115][!indexscheduler]   Rescheduling seqno 39634398
[ 1][t 3][2024-08-12 04:13:49.384366298][IndexScheduler.cpp:104][!indexquery]   Failed to fetch seqno 39634478: [Error : 651 : (0,8000000000000000,45056677): ltdb: block not found]
[ 2][t 3][2024-08-12 04:13:49.384408779][IndexScheduler.cpp:115][!indexscheduler]   Rescheduling seqno 39634478
[ 1][t 7][2024-08-12 04:13:49.384455890][IndexScheduler.cpp:104][!indexquery]   Failed to fetch seqno 39634385: [Error : 651 : (0,8000000000000000,45056621): ltdb: block not found]
[ 2][t 7][2024-08-12 04:13:49.384494642][IndexScheduler.cpp:115][!indexscheduler]   Rescheduling seqno 39634385