search

toncenter / ton-indexer

GNU General Public License v3.0
63 stars 28 forks source link

index-worker can not up #61

Closed zhp1254 closed 2 months ago

zhp1254 commented 3 months ago

command: docker compose up -d index-worker

.env: TON_WORKER_DBROOT=/data/coin/ton/ton-node-docker/db TON_WORKER_FROM=1 TON_WORKER_MAX_PARALLEL_TASKS=1024 TON_WORKER_INSERT_BATCH_SIZE=512 TON_WORKER_INSERT_PARALLEL_ACTORS=3

log: [ 3][t 1][2024-06-19 11:50:22.376103991][InsertManagerBase.cpp:18][!insertmanager] InsertManagerBase::start_up called [ 3][t 1][2024-06-19 11:50:22.376157522][InsertManagerBase.cpp:29][!insertmanager] Insert manager(parallel=12, max_batch_mc_blocks=2000, max_batch_blocks=2000, max_batch_txs=10000, max_batch_msgs=10000) [ 3][t 2][2024-06-19 11:50:22.377335222][InsertManagerPostgres.cpp:1139][!insertmanager] Reading existing seqnos [pid 11] [time 1718797822] Signal: 11 ------- Stack Backtrace ------- ton-index-postgres(+0x35b162)[0x558522e4e162] ton-index-postgres(+0x35b475)[0x558522e4e475] ton-index-postgres(+0x35afbf)[0x558522e4dfbf] /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7fe6d6af5520] ton-index-postgres(+0x4b3110)[0x558522fa6110] ton-index-postgres(+0x4719cc)[0x558522f649cc] ton-index-postgres(+0x398bd6)[0x558522e8bbd6] ton-index-postgres(+0x398e1d)[0x558522e8be1d] ton-index-postgres(+0x3a99e3)[0x558522e9c9e3] ton-index-postgres(+0x39c7df)[0x558522e8f7df] ton-index-postgres(+0x2a1b5a)[0x558522d94b5a] /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3)[0x7fe6d6b47ac3] /lib/x86_64-linux-gnu/libc.so.6(clone+0x44)[0x7fe6d6bd8a04]

zhp1254 commented 3 months ago

final it up; but we got an other err and no data in pgsql table:

[ 1][t 4][2024-06-19 13:44:57.416268942][IndexScheduler.cpp:106][!indexquery] Failed to fetch seqno 38407693: [Error : 651 : seqno not in db] [ 2][t 5][2024-06-19 13:44:57.416272609][IndexScheduler.cpp:117][!indexscheduler] Rescheduling seqno 38407696 [ 2][t 5][2024-06-19 13:44:57.416276354][IndexScheduler.cpp:117][!indexscheduler] Rescheduling seqno 38407693 [ 1][t 4][2024-06-19 13:44:57.416276148][IndexScheduler.cpp:106][!indexquery] Failed to fetch seqno 38407698: [Error : 651 : seqno not in db] [ 2][t 4][2024-06-19 13:44:57.416282429][IndexScheduler.cpp:117][!indexscheduler] Rescheduling seqno 38407698 [ 1][t 3][2024-06-19 13:44:57.416289511][IndexScheduler.cpp:106][!indexquery] Failed to fetch seqno 38407695: [Error : 651 : seqno not in db] [ 2][t 3][2024-06-19 13:44:57.416305092][IndexScheduler.cpp:117][!indexscheduler] Rescheduling seqno 38407695 [ 3][t 3][2024-06-19 13:44:58.417293029][IndexScheduler.cpp:172][!indexscheduler] Last: 0 / 38421348 Blk/s: 0.000000 ETA: -8s Q[0M, 0b, 0t, 0m] [ 1][t 3][2024-06-19 13:44:58.417376143][IndexScheduler.cpp:106][!indexquery] Failed to fetch seqno 38407720: [Error : 651 : seqno not in db] [ 2][t 3][2024-06-19 13:44:58.417383370][IndexScheduler.cpp:117][!indexscheduler] Rescheduling seqno 38407720 [ 1][t 3][2024-06-19 13:44:58.417390951][IndexScheduler.cpp:106][!indexquery] Failed to fetch seqno 38407703: [Error : 651 : seqno not in db] [ 2][t 3][2024-06-19 13:44:58.417398897][IndexScheduler.cpp:117][!indexscheduler] Rescheduling seqno 38407703 [ 1][t 3][2024-06-19 13:44:58.417403636][IndexScheduler.cpp:106][!indexquery] Failed to fetch seqno 38407702: [Error : 651 : seqno not in db] [ 1][t 7][2024-06-19 13:44:58.417396084][IndexScheduler.cpp:106][!indexquery] Failed to fetch seqno 38407700: [Error : 651 : seqno not in db] [ 2][t 3][2024-06-19 13:44:58.417408018][IndexScheduler.cpp:117][!indexscheduler] Rescheduling seqno 38407702 [ 1][t 5][2024-06-19 13:44:58.417407448][IndexScheduler.cpp:106][!indexquery] Failed to fetch seqno 38407701: [Error : 651 : seqno not in db] [ 1][t 7][2024-06-19 13:44:58.417416666][IndexScheduler.cpp:106][!indexquery] Failed to fetch seqno 38407704: [Error : 651 : seqno not in db] [ 2][t 3][2024-06-19 13:44:58.417416251][IndexScheduler.cpp:117][!indexscheduler] Rescheduling seqno 38407700 [ 2][t 3][2024-06-19 13:44:58.417427689][IndexScheduler.cpp:117][!indexscheduler] Rescheduling seqno 38407701 [ 2][t 3][2024-06-19 13:44:58.417432395][IndexScheduler.cpp:117][!indexscheduler] Rescheduling seqno 38407704 [ 1][t 4][2024-06-19 13:44:58.417435756][IndexScheduler.cpp:106][!indexquery] Failed to fetch seqno 38407713: [Error : 651 : seqno not in db] [ 2][t 4][2024-06-19 13:44:58.417459100][IndexScheduler.cpp:117][!indexscheduler] Rescheduling seqno 38407713 [ 1][t 7][2024-06-19 13:44:59.418570283][IndexScheduler.cpp:106][!indexquery] Failed to fetch seqno 38407708: [Error : 651 : seqno not in db] [ 1][t 4][2024-06-19 13:44:59.418578739][IndexScheduler.cpp:106][!indexquery] Failed to fetch seqno 38407727: [Error : 651 : seqno not in db] [ 2][t 7][2024-06-19 13:44:59.418604767][IndexScheduler.cpp:117][!indexscheduler] Rescheduling seqno 38407708

zhp1254 commented 3 months ago

command: docker compose up -d index-worker

.env: TON_WORKER_DBROOT=/data/coin/ton/ton-node-docker/db TON_WORKER_FROM=1 TON_WORKER_MAX_PARALLEL_TASKS=1024 TON_WORKER_INSERT_BATCH_SIZE=512 TON_WORKER_INSERT_PARALLEL_ACTORS=3

log: [ 3][t 1][2024-06-19 11:50:22.376103991][InsertManagerBase.cpp:18][!insertmanager] InsertManagerBase::start_up called [ 3][t 1][2024-06-19 11:50:22.376157522][InsertManagerBase.cpp:29][!insertmanager] Insert manager(parallel=12, max_batch_mc_blocks=2000, max_batch_blocks=2000, max_batch_txs=10000, max_batch_msgs=10000) [ 3][t 2][2024-06-19 11:50:22.377335222][InsertManagerPostgres.cpp:1139][!insertmanager] Reading existing seqnos [pid 11] [time 1718797822] Signal: 11 ------- Stack Backtrace ------- ton-index-postgres(+0x35b162)[0x558522e4e162] ton-index-postgres(+0x35b475)[0x558522e4e475] ton-index-postgres(+0x35afbf)[0x558522e4dfbf] /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7fe6d6af5520] ton-index-postgres(+0x4b3110)[0x558522fa6110] ton-index-postgres(+0x4719cc)[0x558522f649cc] ton-index-postgres(+0x398bd6)[0x558522e8bbd6] ton-index-postgres(+0x398e1d)[0x558522e8be1d] ton-index-postgres(+0x3a99e3)[0x558522e9c9e3] ton-index-postgres(+0x39c7df)[0x558522e8f7df] ton-index-postgres(+0x2a1b5a)[0x558522d94b5a] /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3)[0x7fe6d6b47ac3] /lib/x86_64-linux-gnu/libc.so.6(clone+0x44)[0x7fe6d6bd8a04]

edit file://ton-index-cpp/scripts/entrypoint.sh & set password fixed, ton-index-postgres --host $POSTGRES_HOST_IP \ --port $POSTGRES_PORT \ --user $POSTGRES_USER \ --password 123456\ --dbname $POSTGRES_DBNAME \ --db /tondb $@

kdimentionaltree commented 3 months ago

Hello, @zhp1254. 1) First of all, I ask you to check if you have the main-branch latest commit. 2) Second, I recommend you to change FROM seqno to 3. 3) Third, please decrease TON_WORKER_MAX_PARALLEL_TASKS to number of the order of your CPU number (2 CPU, or 1 CPU count). 3) Fourth, please check if you have \r or \n symbol in the end of the file with the password. This broke the indexer.

Please, let me know if this helped you.

jun0tpyrc commented 3 months ago

hitting same problem using latest branch , seems above not work 

Postgres password file: /run/secrets/postgres_password
Postgres host: postgres (ip: 172.21.0.2)
[ 3][t 1][2024-06-24 07:33:40.249460275][InsertManagerBase.cpp:18][!insertmanager]      InsertManagerBase::start_up called
[ 3][t 1][2024-06-24 07:33:40.249508777][InsertManagerBase.cpp:29][!insertmanager]      Insert manager(parallel=12, max_batch_mc_blocks=2000, max_batch_blocks=2000, max_batch_txs=10000, m
ax_batch_msgs=10000)
[ 3][t 3][2024-06-24 07:33:40.250848361][InsertManagerPostgres.cpp:1139][!insertmanager]        Reading existing seqnos
[pid 12] [time 1719214420] Signal: 11
------- Stack Backtrace -------                                                                                                                                                            ton-index-postgres(+0x35b162)[0x5c5f42c72162]
ton-index-postgres(+0x35b475)[0x5c5f42c72475]                                                                                                                                              ton-index-postgres(+0x35afbf)[0x5c5f42c71fbf]
/lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7e4737b22520]                                                                                                                                  ton-index-postgres(+0x4b3110)[0x5c5f42dca110]
ton-index-postgres(+0x4719cc)[0x5c5f42d889cc]
ton-index-postgres(+0x398bd6)[0x5c5f42cafbd6]
ton-index-postgres(+0x398e1d)[0x5c5f42cafe1d]
ton-index-postgres(+0x3a99e3)[0x5c5f42cc09e3]
ton-index-postgres(+0x39c7df)[0x5c5f42cb37df]
ton-index-postgres(+0x2a1b5a)[0x5c5f42bb8b5a]
/lib/x86_64-linux-gnu/libc.so.6(+0x94ac3)[0x7e4737b74ac3]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x44)[0x7e4737c05a04]

checked postgres already initiated with some initial schema though..

jun0tpyrc commented 3 months ago

^ for latest ton-index-cpp main branch, seem can't build

4.501 -- Found PostgreSQL: /usr/lib/x86_64-linux-gnu/libpq.so (found version "14.12")                                                                                              [0/1121]
4.502 -- Looking for poll
4.554 -- Looking for poll - found
4.554 -- Detecting CXX compile features
4.555 -- Detecting CXX compile features - done
7.714 -- Generating config.h
7.720 -- Generating config.h - done
7.739 absl::int128 properties:
7.739   INCLUDE_DIRECTORIES : $<BUILD_INTERFACE:/app/external/ton/third-party/abseil-cpp>;$<INSTALL_INTERFACE:include>
7.739   BINARY_DIR : /app/build/external/ton/third-party/abseil-cpp/absl/numeric
7.739   INTERFACE_INCLUDE_DIRECTORIES : $<BUILD_INTERFACE:/app/external/ton/third-party/abseil-cpp>;$<INSTALL_INTERFACE:include>
7.739   INTERFACE_LINK_LIBRARIES : absl::config;absl::core_headers;absl::bits
7.739   LINK_LIBRARIES : absl::config;absl::core_headers;absl::bits
7.739   LINK_LIBRARIES_ONLY_TARGETS : PROPERTY-NOTFOUND
7.739   IMPORTED_LOCATION : PROPERTY-NOTFOUND
7.739 cityhash::cityhash properties:
7.739   INCLUDE_DIRECTORIES : /app/external/clickhouse-cpp/contrib/lz4/lz4;/app/external/clickhouse-cpp/contrib/cityhash/cityhash
7.739   BINARY_DIR : /app/build/external/clickhouse-cpp/contrib/cityhash/cityhash
7.739   INTERFACE_INCLUDE_DIRECTORIES : PROPERTY-NOTFOUND
7.739   INTERFACE_LINK_LIBRARIES : PROPERTY-NOTFOUND
7.739   LINK_LIBRARIES : PROPERTY-NOTFOUND
7.740   LINK_LIBRARIES_ONLY_TARGETS : PROPERTY-NOTFOUND
7.740   IMPORTED_LOCATION : PROPERTY-NOTFOUND
7.740 lz4::lz4 properties:
7.740   INCLUDE_DIRECTORIES : /app/external/clickhouse-cpp/contrib/lz4/lz4
7.740   BINARY_DIR : /app/build/external/clickhouse-cpp/contrib/lz4/lz4
7.740   INTERFACE_INCLUDE_DIRECTORIES : PROPERTY-NOTFOUND
7.740   INTERFACE_LINK_LIBRARIES : PROPERTY-NOTFOUND
7.740   LINK_LIBRARIES : PROPERTY-NOTFOUND
7.740   LINK_LIBRARIES_ONLY_TARGETS : PROPERTY-NOTFOUND
7.740   IMPORTED_LOCATION : PROPERTY-NOTFOUND
7.741 CMake Error at CMakeLists.txt:11 (add_subdirectory):
7.741   add_subdirectory given source "ton-index-postgres-v2" which is not an
7.741   existing directory.
7.741
7.741
7.743 CMake Error at CMakeLists.txt:14 (add_subdirectory):
7.743   add_subdirectory given source "sandbox-cpp" which is not an existing
7.743   directory.
7.743
7.743
7.747 -- Configuring incomplete, errors occurred!
7.747 See also "/app/build/CMakeFiles/CMakeOutput.log".
7.747 See also "/app/build/CMakeFiles/CMakeError.log".
------
failed to solve: process "/bin/sh -c cmake -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=off .. -GNinja" did not complete successfully: exit code: 1
jun0tpyrc commented 3 months ago

if we lock to current 

commit d1a11850b896d0f53366dcc4d0a2b5455a702e36 (HEAD)
Author: k-dimentional tree <98183693+kdimentionaltree@users.noreply.github.com>
Date:   Thu May 16 18:37:57 2024 +0300

as master branch of this repo pointing to -> can build but the topic's error

jun0tpyrc commented 3 months ago

missing this one on latest branch that side

diff --git a/Dockerfile b/Dockerfile
index 69185dc..2aa9284 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -10,8 +10,10 @@ RUN apt update -y \
 COPY external/ /app/external/
 COPY tondb-scanner/ /app/tondb-scanner/
 COPY ton-index-postgres/ /app/ton-index-postgres/
+COPY ton-index-postgres-v2/ /app/ton-index-postgres-v2/
 COPY ton-index-clickhouse/ /app/ton-index-clickhouse/
 COPY ton-integrity-checker/ /app/ton-integrity-checker/
+COPY sandbox-cpp/ /app/sandbox-cpp/
 COPY CMakeLists.txt /app/

https://github.com/toncenter/ton-index-worker

jun0tpyrc commented 3 months ago

seem same even i could build that branch with the Dockerfile fix

jun0tpyrc commented 3 months ago

figured out wrong mapping of ton db directory

jun0tpyrc commented 3 months ago

wonder another 2 questsions of 🙏
1) https://github.com/toncenter/ton-indexer/blob/master/docker-compose.yaml#L48C15-L49C36 why alembic HEAD-1

2)

[ 4][t 5][2024-06-24 14:14:32.239678123][IndexScheduler.cpp:101][!indexscheduler]       Scheduled seqno 103
[ 4][t 5][2024-06-24 14:14:32.239695864][IndexScheduler.cpp:101][!indexscheduler]       Scheduled seqno 104
[ 4][t 5][2024-06-24 14:14:32.239702760][IndexScheduler.cpp:101][!indexscheduler]       Scheduled seqno 105
[ 1][t 7][2024-06-24 14:14:32.278594068][IndexScheduler.cpp:106][!indexquery]   Failed to fetch seqno 105: [Error : 602 : state already gc'd]
[ 2][t 7][2024-06-24 14:14:32.278625153][IndexScheduler.cpp:117][!indexscheduler]       Rescheduling seqno 105
[ 1][t 7][2024-06-24 14:14:32.278703633][IndexScheduler.cpp:106][!indexquery]   Failed to fetch seqno 102: [Error : 602 : state already gc'd]
[ 2][t 7][2024-06-24 14:14:32.278730393][IndexScheduler.cpp:117][!indexscheduler]       Rescheduling seqno 102
[ 1][t 7][2024-06-24 14:14:32.278781355][IndexScheduler.cpp:106][!indexquery]   Failed to fetch seqno 103: [Error : 602 : state already gc'd]
[ 2][t 7][2024-06-24 14:14:32.278793747][IndexScheduler.cpp:117][!indexscheduler]       Rescheduling seqno 103
[ 1][t 5][2024-06-24 14:14:32.278867377][IndexScheduler.cpp:106][!indexquery]   Failed to fetch seqno 104: [Error : 602 : state already gc'd]
[ 2][t 5][2024-06-24 14:14:32.278992315][IndexScheduler.cpp:117][!indexscheduler]       Rescheduling seqno 104
[ 1][t 5][2024-06-24 14:14:32.279235661][IndexScheduler.cpp:106][!indexquery]   Failed to fetch seqno 99: [Error : 602 : state already gc'd]
[ 2][t 5][2024-06-24 14:14:32.279254422][IndexScheduler.cpp:117][!indexscheduler]       Rescheduling seqno 99
[ 1][t 7][2024-06-24 14:14:32.279347429][IndexScheduler.cpp:106][!indexquery]   Failed to fetch seqno 101: [Error : 602 : state already gc'd]
[ 1][t 5][2024-06-24 14:14:32.279341509][IndexScheduler.cpp:106][!indexquery]   Failed to fetch seqno 100: [Error : 602 : state already gc'd]
[ 2][t 7][2024-06-24 14:14:32.279363799][IndexScheduler.cpp:117][!indexscheduler]       Rescheduling seqno 101
[ 2][t 7][2024-06-24 14:14:32.279371316][IndexScheduler.cpp:117][!indexscheduler]       Rescheduling seqno 100

seem TON_WORKER_FROM not respected , if i manually editting entrypoint.sh seems work

[ 1][t 1][2024-06-24 14:16:23.259652424][archive-manager.cpp:876][!archive]     checking file /tondb/archive/states/state_38210813_-1_8000000000000000_D0E749AE414202B2E84F12366BA9D08FD974
546CBF5A54F6469C9E04B86E128D
[ 1][t 1][2024-06-24 14:16:23.259657608][archive-manager.cpp:876][!archive]     checking file /tondb/archive/states/state_8930706_-1_8000000000000000_C892261931A55FB33B0F063C32153076C67F8
A17668895EC8E3E41E0587F4AB5
[ 1][t 1][2024-06-24 14:16:23.259665400][archive-manager.cpp:876][!archive]     checking file /tondb/archive/states/state_38210813_0_b000000000000000_37FC5B27D643323304670205D967852E35231
B291BDDF1C78702E8B457DA4D67
[ 1][t 1][2024-06-24 14:16:23.259676541][archive-manager.cpp:876][!archive]     checking file /tondb/archive/states/state_36907647_-1_8000000000000000_99DAA6D2D0996441C6C58CD545903C793636
C2F0A11E819D800A60A7E8A1D1B1
[ 4][t 1][2024-06-24 14:16:23.259711068][archive-slice.cpp:478][!slice] Opening archive slice /tondb/archive/packages/arch0385/archive.38572056.index
[ 4][t 1][2024-06-24 14:16:23.446085922][DbScanner.cpp:295][!scanner]   New masterchain seqno: 38574998
[ 3][t 4][2024-06-24 14:16:24.069964412][IndexScheduler.cpp:172][!indexscheduler]       Last: 0 / 38500000      Blk/s: 0.000000 ETA: -8s        Q[0M, 0b, 0t, 0m]
[ 4][t 4][2024-06-24 14:16:24.070461768][IndexScheduler.cpp:204][!indexscheduler]       Scheduling next seqnos. Current tasks: 0
[ 4][t 4][2024-06-24 14:16:24.070472092][IndexScheduler.cpp:101][!indexscheduler]       Scheduled seqno 38500001
[ 4][t 4][2024-06-24 14:16:24.070478309][IndexScheduler.cpp:101][!indexscheduler]       Scheduled seqno 38500002
[ 4][t 4][2024-06-24 14:16:24.070484132][IndexScheduler.cpp:101][!indexscheduler]       Scheduled seqno 38500003
[ 4][t 4][2024-06-24 14:16:24.070489268][IndexScheduler.cpp:101][!indexscheduler]       Scheduled seqno 38500004
[ 4][t 4][2024-06-24 14:16:24.070495137][IndexScheduler.cpp:101][!indexscheduler]       Scheduled seqno 38500005
dungeon-master-666 commented 2 months ago

Thanks for reporting these issues. We've updated the index worker with fixed Dockerfile and confusing logging. Please update to latest commit of master branch and give it a try.

Answering your question about alembic revision - currently we have 2 revisions (HEAD and HEAD-1). The only difference is that HEAD removes all SQL indexes. It is useful to use it if you are indexing a whole blockchain, and then migrating back to HEAD-1 since large indexes significantly slow down inserting to postgres.

dungeon-master-666 commented 2 months ago

Closing due to no activity.