search

toncenter / ton-indexer

TON Indexer system to store and serve blockchain data using SQL-database
https://toncenter.com/api/v3/
GNU General Public License v3.0
69 stars 34 forks source link

index sync halt #68

Closed dnjef9duo closed 4 months ago

dnjef9duo commented 4 months ago

unfortunately, seems still same issue after upgrade. and i cannot reopen the previous one. https://github.com/toncenter/ton-indexer/issues/62 so i create an new one.

it happened on testnet this time. 4 times restart in 12hours, and the final one it stop syncing. and mainent looks good so far. will keep monitoring.

i have upgrade by following command 

git pull.
git submodule update --recursive --init

and now on this commit commit 15888e88dc571c0efbdedd73be2123b8be8de872 (HEAD -> master, origin/master, origin/HEAD)

and here is the dump logs looks sometimes it can sync up automatically after restart, sometimes not.

index-worker-1  | ESC[1;33m[ 2][t 3][2024-07-08 14:16:08.619524722][InterfaceDetectors.hpp:255][!jetton_master_detector]        Failed to parse jetton content for 0:3064595B098F76870997FFF7D0FA49F6F8192F86856CB03BEF60B9077D27CB5C: [Error
: 0 : Unknown token data type]ESC[0m
index-worker-1  | ESC[1;33m[ 2][t 3][2024-07-08 14:16:08.619569527][InterfaceDetectors.hpp:256][!jetton_master_detector]        te6cckEBAgEAqAAB/lRFU1QtVEFYLTU1dGVzdCBkZSBqZXR0b24gY29uIHRheFRFU1RGNTVodHRwczovL3Jhdy5naXRodWJ1c2VyY29udGVudC
5jb20vcG9sYXJ5c2ZvdW5kYXRpb24vbG9nb19jb2luX3Rlc3QvZmI0NDMzNDFiZWY1ODFkNWZkY2YBAEhkNmI0YWUyYmZhNmFlNTQ0YjExNS9idXlib3QtbG9nby5wbmfBVJSJESC[0m
index-worker-1  | ESC[1;36m[ 3][t 7][2024-07-08 14:16:11.543344235][IndexScheduler.cpp:172][!indexscheduler]    Last: 20850897 / 20850897       Blk/s: 0.402578 ETA: 0s Q[0M, 0b, 0t, 0m]ESC[0m
index-worker-1  | ESC[1;36m[ 3][t 6][2024-07-08 14:16:21.555240185][IndexScheduler.cpp:172][!indexscheduler]    Last: 20850900 / 20850901       Blk/s: 0.343810 ETA: 2s Q[1M, 5b, 3t, 1m]ESC[0m
index-worker-1  | ESC[1;33m[ 2][t 1][2024-07-08 14:16:23.571451350][InterfaceDetectors.hpp:255][!jetton_master_detector]        Failed to parse jetton content for 0:FC39ED78AFEF008478E5637B2563F10425C8D9F5E16C21B3F09D7033E89AFBA5: [Error
: 0 : Unknown token data type]ESC[0m
index-worker-1  | ESC[1;33m[ 2][t 1][2024-07-08 14:16:23.571490550][InterfaceDetectors.hpp:256][!jetton_master_detector]        te6cckEBAQEAAgAAAEysuc0=ESC[0m
index-worker-1  | ESC[1;36m[ 3][t 6][2024-07-08 14:16:31.567076644][IndexScheduler.cpp:172][!indexscheduler]    Last: 20850904 / 20850905       Blk/s: 0.365665 ETA: 2s Q[0M, 0b, 0t, 0m]ESC[0m
index-worker-1  | ESC[1;36m[ 3][t 7][2024-07-08 14:16:41.578981728][IndexScheduler.cpp:172][!indexscheduler]    Last: 20850908 / 20850908       Blk/s: 0.364986 ETA: 0s Q[0M, 0b, 0t, 0m]ESC[0m
index-worker-1  | ESC[1;36m[ 3][t 3][2024-07-08 14:16:51.590673248][IndexScheduler.cpp:172][!indexscheduler]    Last: 20850912 / 20850912       Blk/s: 0.407041 ETA: 0s Q[1M, 5b, 4t, 3m]ESC[0m
index-worker-1  | ESC[1;36m[ 3][t 4][2024-07-08 14:17:01.601429450][IndexScheduler.cpp:172][!indexscheduler]    Last: 20850916 / 20850916       Blk/s: 0.409117 ETA: 0s Q[0M, 0b, 0t, 0m]ESC[0m
index-worker-1  | [pid 11] [time 1720448223] Signal: 11
index-worker-1  | ------- Stack Backtrace -------
index-worker-1  | ton-index-postgres(+0x35b162)[0x558ac4899162]
index-worker-1  | ton-index-postgres(+0x35b475)[0x558ac4899475]
index-worker-1  | ton-index-postgres(+0x35afbf)[0x558ac4898fbf]
index-worker-1  | /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7fcecb30f520]
index-worker-1  | ton-index-postgres(+0x6fdd46)[0x558ac4c3bd46]
index-worker-1  | ton-index-postgres(+0x4dc180)[0x558ac4a1a180]
index-worker-1  | ton-index-postgres(+0x4b3456)[0x558ac49f1456]
index-worker-1  | ton-index-postgres(+0x398bd6)[0x558ac48d6bd6]
index-worker-1  | ton-index-postgres(+0x398e1d)[0x558ac48d6e1d]
index-worker-1  | ton-index-postgres(+0x3a99e3)[0x558ac48e79e3]
index-worker-1  | ton-index-postgres(+0x39c7df)[0x558ac48da7df]
index-worker-1  | ton-index-postgres(+0x2a1b5a)[0x558ac47dfb5a]
index-worker-1  | /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3)[0x7fcecb361ac3]
index-worker-1  | /lib/x86_64-linux-gnu/libc.so.6(clone+0x44)[0x7fcecb3f2a04]
index-worker-1  | -------------------------------
index-worker-1  | Postgres password file: /run/secrets/postgres_password
index-worker-1  | Postgres host: postgres (ip: 172.26.0.2)
index-worker-1  | ESC[1;36m[ 3][t 1][2024-07-08 14:17:05.711713935][InsertManagerBase.cpp:18][!insertmanager]   InsertManagerBase::start_up calledESC[0m
index-worker-1  | ESC[1;36m[ 3][t 1][2024-07-08 14:17:05.711788310][InsertManagerBase.cpp:29][!insertmanager]   Insert manager(parallel=12, max_batch_mc_blocks=2000, max_batch_blocks=2000, max_batch_txs=10000, max_batch_msgs=10000)ESC[0m
index-worker-1  | ESC[1;36m[ 3][t 3][2024-07-08 14:17:05.712624035][InsertManagerPostgres.cpp:1139][!insertmanager]     Reading existing seqnosESC[0m
index-worker-1  | ESC[1;36m[ 3][t 3][2024-07-08 14:17:05.744343829][IndexScheduler.cpp:80][!indexscheduler]     Found 5730 existing seqnosESC[0m
index-worker-1  | ESC[1;36m[ 3][t 4][2024-07-08 14:17:06.745314942][IndexScheduler.cpp:172][!indexscheduler]    Last: 0 / 20845187      Blk/s: 0.000000 ETA: -8s        Q[0M, 0b, 0t, 0m]ESC[0m
index-worker-1  | ESC[1;36m[ 3][t 4][2024-07-08 14:17:06.745817508][IndexScheduler.cpp:95][!indexscheduler]     Skipped 5729 existing seqnosESC[0m
index-worker-1  | ESC[1;36m[ 3][t 4][2024-07-08 14:17:06.745856652][IndexScheduler.cpp:211][!indexscheduler]    Syncronization complete!ESC[0m
index-worker-1  | ESC[1;36m[ 3][t 2][2024-07-08 14:17:16.747803050][IndexScheduler.cpp:172][!indexscheduler]    Last: 20850920 / 20850920       Blk/s: 0.281974 ETA: 0s Q[0M, 0b, 0t, 0m]ESC[0m
index-worker-1  | ESC[1;36m[ 3][t 7][2024-07-08 14:17:26.751430171][IndexScheduler.cpp:172][!indexscheduler]    Last: 20850924 / 20850924       Blk/s: 0.387975 ETA: 0s Q[0M, 0b, 0t, 0m]ESC[0m
index-worker-1  | ESC[1;36m[ 3][t 2][2024-07-08 14:17:36.754820711][IndexScheduler.cpp:172][!indexscheduler]    Last: 20850926 / 20850927       Blk/s: 0.275327 ETA: 3s Q[1M, 5b, 5t, 8m]ESC[0m
index-worker-1  | ESC[1;36m[ 3][t 3][2024-07-08 18:15:42.265280935][IndexScheduler.cpp:172][!indexscheduler]    Last: 20855531 / 20856172       Blk/s: 0.286455 ETA: 37m 17s    Q[0M, 0b, 0t, 0m]ESC[0m
index-worker-1  | ESC[1;36m[ 3][t 7][2024-07-08 18:15:52.276927661][IndexScheduler.cpp:172][!indexscheduler]    Last: 20855532 / 20856176       Blk/s: 0.172781 ETA: 1h 2m 7s   Q[0M, 0b, 0t, 0m]ESC[0m
index-worker-1  | ESC[1;36m[ 3][t 1][2024-07-08 18:16:02.288799279][IndexScheduler.cpp:172][!indexscheduler]    Last: 20855533 / 20856179       Blk/s: 0.160245 ETA: 1h 7m 11s  Q[0M, 0b, 0t, 0m]ESC[0m
index-worker-1  | ESC[1;36m[ 3][t 6][2024-07-08 18:16:12.300821813][IndexScheduler.cpp:172][!indexscheduler]    Last: 20855536 / 20856183       Blk/s: 0.242065 ETA: 44m 32s    Q[0M, 0b, 0t, 0m]ESC[0m
index-worker-1  | ESC[1;36m[ 3][t 2][2024-07-08 18:16:22.313089972][IndexScheduler.cpp:172][!indexscheduler]    Last: 20855538 / 20856188       Blk/s: 0.238303 ETA: 45m 27s    Q[0M, 0b, 0t, 0m]ESC[0m
index-worker-1  | ESC[1;36m[ 3][t 7][2024-07-08 18:16:32.325003581][IndexScheduler.cpp:172][!indexscheduler]    Last: 20855544 / 20856192       Blk/s: 0.414984 ETA: 26m 1s     Q[0M, 0b, 0t, 0m]ESC[0m
index-worker-1  | ESC[1;36m[ 3][t 2][2024-07-08 18:16:42.336201429][IndexScheduler.cpp:172][!indexscheduler]    Last: 20855546 / 20856196       Blk/s: 0.327792 ETA: 33m 2s     Q[0M, 0b, 0t, 0m]ESC[0m
index-worker-1  | ESC[1;36m[ 3][t 6][2024-07-08 18:16:52.347707469][IndexScheduler.cpp:172][!indexscheduler]    Last: 20855549 / 20856198       Blk/s: 0.367438 ETA: 29m 26s    Q[0M, 0b, 0t, 0m]ESC[0m
index-worker-1  | ESC[1;36m[ 3][t 6][2024-07-08 18:17:02.359024286][IndexScheduler.cpp:172][!indexscheduler]    Last: 20855550 / 20856202       Blk/s: 0.201018 ETA: 54m 3s     Q[1M, 5b, 6t, 7m]ESC[0m
index-worker-1  | [pid 11] [time 1720462626] Signal: 11
index-worker-1  | ------- Stack Backtrace -------
index-worker-1  | ton-index-postgres(+0x35b162)[0x556df34fb162]
index-worker-1  | ton-index-postgres(+0x35b475)[0x556df34fb475]
index-worker-1  | ton-index-postgres(+0x35afbf)[0x556df34fafbf]
index-worker-1  | /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7f06997b8520]
index-worker-1  | ton-index-postgres(+0x6fdd46)[0x556df389dd46]
index-worker-1  | ton-index-postgres(+0x4dc180)[0x556df367c180]
index-worker-1  | ton-index-postgres(+0x4b3456)[0x556df3653456]
index-worker-1  | ton-index-postgres(+0x398bd6)[0x556df3538bd6]
index-worker-1  | ton-index-postgres(+0x398e1d)[0x556df3538e1d]
index-worker-1  | ton-index-postgres(+0x3a99e3)[0x556df35499e3]
index-worker-1  | ton-index-postgres(+0x39c7df)[0x556df353c7df]
index-worker-1  | ton-index-postgres(+0x2a1b5a)[0x556df3441b5a]
index-worker-1  | /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3)[0x7f069980aac3]
index-worker-1  | /lib/x86_64-linux-gnu/libc.so.6(clone+0x44)[0x7f069989ba04]
index-worker-1  | -------------------------------
index-worker-1  | Postgres password file: /run/secrets/postgres_password
index-worker-1  | Postgres host: postgres (ip: 172.26.0.2)
index-worker-1  | ESC[1;36m[ 3][t 1][2024-07-08 18:17:08.249014976][InsertManagerBase.cpp:18][!insertmanager]   InsertManagerBase::start_up calledESC[0m
index-worker-1  | ESC[1;36m[ 3][t 1][2024-07-08 18:17:08.249094377][InsertManagerBase.cpp:29][!insertmanager]   Insert manager(parallel=12, max_batch_mc_blocks=2000, max_batch_blocks=2000, max_batch_txs=10000, max_batch_msgs=10000)ESC[0m
index-worker-1  | ESC[1;36m[ 3][t 1][2024-07-08 18:17:08.249126664][InsertManagerPostgres.cpp:1139][!insertmanager]     Reading existing seqnosESC[0m
index-worker-1  | ESC[1;36m[ 3][t 1][2024-07-08 18:17:08.291694249][IndexScheduler.cpp:80][!indexscheduler]     Found 10366 existing seqnosESC[0m
index-worker-1  | ESC[1;36m[ 3][t 5][2024-07-08 18:17:09.292483794][IndexScheduler.cpp:172][!indexscheduler]    Last: 0 / 20845187      Blk/s: 0.000000 ETA: -8s        Q[0M, 0b, 0t, 0m]ESC[0m
index-worker-1  | ESC[1;36m[ 3][t 5][2024-07-08 18:17:09.293177207][IndexScheduler.cpp:95][!indexscheduler]     Skipped 10365 existing seqnosESC[0m
index-worker-1  | ESC[1;36m[ 3][t 7][2024-07-08 18:17:19.296936863][IndexScheduler.cpp:172][!indexscheduler]    Last: 20855565 / 20856208       Blk/s: 0.602654 ETA: 17m 46s    Q[0M, 0b, 0t, 0m]ESC[0m
index-worker-1  | ESC[1;36m[ 3][t 6][2024-07-08 18:17:29.299921635][IndexScheduler.cpp:172][!indexscheduler]    Last: 20855565 / 20856212       Blk/s: 0.210132 ETA: 51m 19s    Q[0M, 0b, 0t, 0m]ESC[0m
index-worker-1  | ESC[1;36m[ 3][t 1][2024-07-08 18:17:39.305268041][IndexScheduler.cpp:172][!indexscheduler]    Last: 20855565 / 20856215       Blk/s: 0.073269 ETA: 2h 27m 51s Q[0M, 0b, 0t, 0m]ESC[0m
index-worker-1  | ESC[1;36m[ 3][t 6][2024-07-08 18:17:49.314950353][IndexScheduler.cpp:172][!indexscheduler]    Last: 20855565 / 20856218       Blk/s: 0.025547 ETA: 7h 6m 0s   Q[0M, 0b, 0t, 0m]ESC[0m
index-worker-1  | ESC[1;36m[ 3][t 5][2024-07-08 18:17:59.326237798][IndexScheduler.cpp:172][!indexscheduler]    Last: 20855565 / 20856221       Blk/s: 0.008908 ETA: 20h 27m 23s        Q[0M, 0b, 0t, 0m]ESC[0m
index-worker-1  | ESC[1;36m[ 3][t 6][2024-07-08 18:18:09.337115944][IndexScheduler.cpp:172][!indexscheduler]    Last: 20855565 / 20856226       Blk/s: 0.003106 ETA: 2d 11h 6m 57s      Q[0M, 0b, 0t, 0m]ESC[0m
dungeon-master-666 commented 4 months ago

Based on logs I see that it is still not the latest version of the index worker. Make sure that you rebuild docker image after pulling changes:

git pull
git submodule update --recursive --init
docker compose build --no-cache
docker compose up -d
dnjef9duo commented 4 months ago

ahhhh sorry, my bad, i did not rebuild after pull..

thanks for reminding.

will close this ticket.