search

ton-blockchain / ton

Main TON monorepo
Other
2.61k stars 722 forks source link

mainnet fullnode not sync, validator-engine keep crashing #985

Open dnjef9duo opened 1 month ago

dnjef9duo commented 1 month ago

mainnet fullnode cannot sync validator-engine keep crashing.

it runs smooth in the past year. but it crashed almost every mins recently and stop syncing. status shows local validator working time always very short.

what can we do ?

Machine

12 Cores
128G ram
2T disk

OS info

Distributor ID: Ubuntu
Description:    Ubuntu 20.04.3 LTS
Release:    20.04
mytonctrl > status

Mytoncore status: working, 5 hours
Local validator status: working, 1 minutes
Local validator out of sync: 465052 s
Local validator database size: 1505.89 Gb, 74.9%
Version mytonctrl: b97d502 (master)
Version validator: 4cfe1d1 (master)
tail -f /var/ton-work/log

[ 1][t 0][2024-05-09 08:04:38.289379708][TcpListener.cpp:64][!TcpListener[port:18886]]  Accept
[ 1][t 0][2024-05-09 08:05:05.001387141][TcpListener.cpp:64][!TcpListener[port:18886]]  Accept
[ 1][t 0][2024-05-09 08:05:16.805744396][TcpListener.cpp:64][!TcpListener[port:18886]]  Accept
[ 1][t 0][2024-05-09 08:05:22.401140903][TcpListener.cpp:64][!TcpListener[port:18886]]  Accept
[ 1][t 0][2024-05-09 08:05:24.038765442][TcpListener.cpp:64][!TcpListener[port:18886]]  Accept
[ 1][t 0][2024-05-09 08:05:29.238843791][TcpListener.cpp:64][!TcpListener[port:18886]]  Accept
[ 1][t 0][2024-05-09 08:05:41.629979098][TcpListener.cpp:64][!TcpListener[port:18886]]  Accept
[ 1][t 0][2024-05-09 08:05:50.597547245][TcpListener.cpp:64][!TcpListener[port:18886]]  Accept
[pid 3123580] [time 1715241967] Signal: 6
------- Stack Backtrace -------
/usr/bin/ton/validator-engine/validator-engine[0x53ec79]
/usr/bin/ton/validator-engine/validator-engine[0x53d216]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x14420)[0x7f518dd83420]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb)[0x7f518d86a00b]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x12b)[0x7f518d849859]
/usr/bin/ton/validator-engine/validator-engine[0x548c67]
/usr/bin/ton/validator-engine/validator-engine[0x55b50a]
/usr/bin/ton/validator-engine/validator-engine[0x548996]
/usr/bin/ton/validator-engine/validator-engine[0x46d972]
/usr/bin/ton/validator-engine/validator-engine[0xabf4fa]
/usr/bin/ton/validator-engine/validator-engine[0xac6c91]
/usr/bin/ton/validator-engine/validator-engine[0xadbb17]
/usr/bin/ton/validator-engine/validator-engine[0x55c4af]
/usr/bin/ton/validator-engine/validator-engine[0x562c42]
/usr/bin/ton/validator-engine/validator-engine[0x561100]
/usr/bin/ton/validator-engine/validator-engine[0x56115e]
/usr/bin/ton/validator-engine/validator-engine[0x4b8890]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8609)[0x7f518dd77609]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x43)[0x7f518d946353]
-------------------------------
[ 1][t 0][2024-05-09 08:06:40.733403801][TcpListener.cpp:64][!TcpListener[port:18886]]  Accept
[ 1][t 0][2024-05-09 08:06:51.352765713][TcpListener.cpp:64][!TcpListener[port:18886]]  Accept
[ 1][t 0][2024-05-09 08:07:01.208000893][TcpListener.cpp:64][!TcpListener[port:18886]]  Accept
[ 1][t 0][2024-05-09 08:07:02.184830228][TcpListener.cpp:64][!TcpListener[port:18886]]  Accept
[ 1][t 0][2024-05-09 08:07:10.572986009][TcpListener.cpp:64][!TcpListener[port:18886]]  Accept
[ 1][t 0][2024-05-09 08:07:11.720746323][TcpListener.cpp:64][!TcpListener[port:18886]]  Accept
[ 1][t 0][2024-05-09 08:07:14.118491353][TcpListener.cpp:64][!TcpListener[port:18886]]  Accept
[ 1][t 0][2024-05-09 08:07:32.105529929][TcpListener.cpp:64][!TcpListener[port:18886]]  Accept
[ 1][t 0][2024-05-09 08:07:33.285767072][TcpListener.cpp:64][!TcpListener[port:18886]]  Accept
[ 1][t 0][2024-05-09 08:07:37.010698391][TcpListener.cpp:64][!TcpListener[port:18886]]  Accept
[ 1][t 0][2024-05-09 08:07:40.812644431][TcpListener.cpp:64][!TcpListener[port:18886]]  Accept
[ 1][t 0][2024-05-09 08:07:45.200289302][TcpListener.cpp:64][!TcpListener[port:18886]]  Accept
[ 1][t 0][2024-05-09 08:08:02.006068970][TcpListener.cpp:64][!TcpListener[port:18886]]  Accept
[pid 3125029] [time 1715242085] Signal: 6
------- Stack Backtrace -------
/usr/bin/ton/validator-engine/validator-engine[0x53ec79]
/usr/bin/ton/validator-engine/validator-engine[0x53d216]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x14420)[0x7fabdd5a7420]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb)[0x7fabdd08e00b]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x12b)[0x7fabdd06d859]
/usr/bin/ton/validator-engine/validator-engine[0x548c67]
/usr/bin/ton/validator-engine/validator-engine[0x55b50a]
/usr/bin/ton/validator-engine/validator-engine[0x548996]
/usr/bin/ton/validator-engine/validator-engine[0x46d972]
/usr/bin/ton/validator-engine/validator-engine[0xabf4fa]
/usr/bin/ton/validator-engine/validator-engine[0xac6c91]
/usr/bin/ton/validator-engine/validator-engine[0xadbb17]
/usr/bin/ton/validator-engine/validator-engine[0x55c4af]
/usr/bin/ton/validator-engine/validator-engine[0x562c42]
/usr/bin/ton/validator-engine/validator-engine[0x561100]
/usr/bin/ton/validator-engine/validator-engine[0x56115e]
/usr/bin/ton/validator-engine/validator-engine[0x4b8890]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8609)[0x7fabdd59b609]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x43)[0x7fabdd16a353]
-------------------------------
[ 1][t 0][2024-05-09 08:08:43.892442663][TcpListener.cpp:64][!TcpListener[port:18886]]  Accept
[ 1][t 0][2024-05-09 08:08:43.904639881][TcpListener.cpp:64][!TcpListener[port:18886]]  Accept
[ 1][t 0][2024-05-09 08:08:47.518034250][TcpListener.cpp:64][!TcpListener[port:18886]]  Accept
[ 1][t 0][2024-05-09 08:08:54.577694618][TcpListener.cpp:64][!TcpListener[port:18886]]  Accept
[ 1][t 0][2024-05-09 08:09:12.925566974][TcpListener.cpp:64][!TcpListener[port:18886]]  Accept
[ 1][t 0][2024-05-09 08:09:21.804740038][TcpListener.cpp:64][!TcpListener[port:18886]]  Accept
[ 1][t 0][2024-05-09 08:09:46.532957447][TcpListener.cpp:64][!TcpListener[port:18886]]  Accept
[ 1][t 0][2024-05-09 08:09:50.562824176][TcpListener.cpp:64][!TcpListener[port:18886]]  Accept
[ 1][t 0][2024-05-09 08:09:53.892134064][TcpListener.cpp:64][!TcpListener[port:18886]]  Accept
[ 1][t 0][2024-05-09 08:09:57.961116471][TcpListener.cpp:64][!TcpListener[port:18886]]  Accept
[pid 3127186] [time 1715242203] Signal: 6
------- Stack Backtrace -------
/usr/bin/ton/validator-engine/validator-engine[0x53ec79]
/usr/bin/ton/validator-engine/validator-engine[0x53d216]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x14420)[0x7f08d4f3f420]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb)[0x7f08d4a2600b]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x12b)[0x7f08d4a05859]
/usr/bin/ton/validator-engine/validator-engine[0x548c67]
/usr/bin/ton/validator-engine/validator-engine[0x55b50a]
/usr/bin/ton/validator-engine/validator-engine[0x548996]
/usr/bin/ton/validator-engine/validator-engine[0x46d972]
/usr/bin/ton/validator-engine/validator-engine[0xabf4fa]
/usr/bin/ton/validator-engine/validator-engine[0xac6c91]
/usr/bin/ton/validator-engine/validator-engine[0xadbb17]
/usr/bin/ton/validator-engine/validator-engine[0x55c4af]
/usr/bin/ton/validator-engine/validator-engine[0x562c42]
/usr/bin/ton/validator-engine/validator-engine[0x561100]
/usr/bin/ton/validator-engine/validator-engine[0x56115e]
/usr/bin/ton/validator-engine/validator-engine[0x4b8890]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8609)[0x7f08d4f33609]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x43)[0x7f08d4b02353]
-------------------------------