Genesis out of sync

[dkatzan]

It seems like at some point MyLocalTon stops working

specifically, at some point the logs indicate that the validator-engine-genesis stopped (it gets restarted) but from that point on, genesis block gets out of sync

2024-08-06 22:50:28 19:50:28.056 [MyLocalTon - Dump Block 767] INFO  org.ton.actions.MyLocalTon - (-1,8000000000000000,768)
2024-08-06 22:50:29 19:50:29.036 [MyLocalTon - Dump Block 768] INFO  org.ton.actions.MyLocalTon - (-1,8000000000000000,769)
2024-08-06 22:50:29 19:50:29.785 [validator-engine-genesis] INFO  o.t.e.v.ValidatorEngineExecutor - validator-engine-genesis stopped
2024-08-06 22:50:29 19:50:29.788 [ForkJoinPool.commonPool-worker-2] INFO  o.t.e.v.ValidatorEngineExecutor - re-starting validator genesis...
2024-08-06 22:50:29 19:50:29.793 [ForkJoinPool.commonPool-worker-2] INFO  o.t.e.v.ValidatorEngine - starting validator-engine genesis
2024-08-06 22:50:29 19:50:29.796 [ForkJoinPool.commonPool-worker-2] INFO  o.t.e.v.ValidatorEngine - genesis validator-engine started at 127.0.0.1:4442
2024-08-06 22:50:29 19:50:29.802 [ForkJoinPool.commonPool-worker-2] INFO  o.t.e.v.ValidatorEngineExecutor - re-started validator genesis with pid 7242
2024-08-06 22:50:36 19:50:36.295 [MyLocalTon - genesis Status Monitor] INFO  org.ton.actions.MyLocalTon - genesis is ready
2024-08-06 22:50:45 19:50:45.557 [MyLocalTon - Accounts Monitor] WARN  org.ton.utils.MyLocalTonUtils - cannot identify wallet version by the code E74D
2024-08-06 22:50:45 19:50:45.870 [MyLocalTon - Validation Monitor] INFO  org.ton.utils.MyLocalTonUtils - active election id 0, 1970.01.01 00:00:00, current time 2024.08.06 19:50:45
2024-08-06 22:50:46 19:50:46.130 [MyLocalTon - Validation Monitor] INFO  org.ton.utils.MyLocalTonUtils - participants []
2024-08-06 22:50:46 19:50:46.130 [MyLocalTon - Validation Monitor] INFO  org.ton.actions.MyLocalTon - ELECTIONS CLOSED, REAPING...
2024-08-06 22:50:46 19:50:46.162 [MyLocalTon - Reaping rewards by genesis] INFO  org.ton.actions.MyLocalTon - Reaping rewards. genesis reward size is 0, nothing to reap.
2024-08-06 22:50:51 19:50:51.305 [MyLocalTon - genesis Status Monitor] INFO  org.ton.actions.MyLocalTon - genesis out of sync by 23 seconds
2024-08-06 22:51:06 19:51:06.277 [MyLocalTon - genesis Status Monitor] INFO  org.ton.actions.MyLocalTon - genesis out of sync by 38 seconds
2024-08-06 22:51:21 19:51:21.338 [MyLocalTon - genesis Status Monitor] INFO  org.ton.actions.MyLocalTon - genesis out of sync by 53 seconds
2024-08-06 22:51:36 19:51:36.375 [MyLocalTon - genesis Status Monitor] INFO  org.ton.actions.MyLocalTon - genesis out of sync by 68 seconds
2024-08-06 22:51:46 19:51:46.157 [MyLocalTon - Validation Monitor] INFO  org.ton.utils.MyLocalTonUtils - active election id 0, 1970.01.01 00:00:00, current time 2024.08.06 19:51:46
2024-08-06 22:51:46 19:51:46.253 [MyLocalTon - Validation Monitor] INFO  org.ton.utils.MyLocalTonUtils - participants []
2024-08-06 22:51:46 19:51:46.253 [MyLocalTon - Validation Monitor] INFO  org.ton.actions.MyLocalTon - ELECTIONS CLOSED, REAPING...
2024-08-06 22:51:51 19:51:51.296 [MyLocalTon - genesis Status Monitor] INFO  org.ton.actions.MyLocalTon - genesis out of sync by 83 seconds

[neodix42]

Could you please provide OS version and architecture, as well as MyLocalTon version.

[neodix42]

In the meantime, please try adding additional nodes and see if the blockchain also stops randomly.

[dkatzan]

I'm running it in a docker, base image is ubuntu:24.04 uname -a gives me Linux 4093e4612af9 5.15.49-linuxkit-pr #1 SMP PREEMPT Thu May 25 07:27:39 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux

It happened to me both with v110 and v111

WDYM by adding additional nodes? how do I do that?

[neodix42]

What is a host machine? mac on arm64 I guess? Via MyLocalTon GUI you can add up to 6 additional validators (see the screenshot). I will think maybe to add this option to a command line as well.

[dkatzan]

yes I'm using a mac with arm, my host machine details: Darwin Daniels-MacBook-Pro-2.local 22.3.0 Darwin Kernel Version 22.3.0: Mon Jan 30 20:38:37 PST 2023; root:xnu-8792.81.3~2/RELEASE_ARM64_T6000 arm64 arm Darwin

so If I'm running it with headless java inside my docker, I assume I currently don't have a way to add validators?

[neodix42]

Thank you for the info. Right, I need to add this option. Leave it for now.

[neodix42]

Could you please try out the new version v112, where a new parameter [with-validators-N] has been introduced. Just set N to 1..6 and you will get genesis plus N additional validators. Let's if it improves stability.

[dkatzan]

Thx, will try that and report back

[undron]

v113 Run command "java", "-jar", "MyLocalTon-x86-64.jar", "nogui", "with-validators-3", "ton-http-api" Dead after 1013 blocks

17:31:36.398 [MyLocalTon - Validation Monitor] INFO  org.ton.utils.MyLocalTonUtils - active election id 0, 1970.01.01 00:00:00, current time 2024.08.09 17:31:36
17:31:36.491 [MyLocalTon - Validation Monitor] INFO  org.ton.utils.MyLocalTonUtils - participants []
17:31:36.491 [MyLocalTon - Validation Monitor] INFO  org.ton.actions.MyLocalTon - ELECTIONS CLOSED, REAPING...
17:31:36.491 [MyLocalTon - Reaping rewards by node4] ERROR org.ton.actions.MyLocalTon - Reaping rewards. node4 wallet is not present.
17:31:36.515 [MyLocalTon - Reaping rewards by node3] INFO  org.ton.actions.MyLocalTon - Reaping rewards. node3 reward size is 0, nothing to reap.
17:31:36.515 [MyLocalTon - Reaping rewards by node2] INFO  org.ton.actions.MyLocalTon - Reaping rewards. node2 reward size is 0, nothing to reap.
17:31:36.515 [MyLocalTon - Reaping rewards by genesis] INFO  org.ton.actions.MyLocalTon - Reaping rewards. genesis reward size is 0, nothing to reap.
17:31:37.313 [MyLocalTon - Dump Block 1001] INFO  org.ton.actions.MyLocalTon - (-1,8000000000000000,1002)
17:31:38.310 [MyLocalTon - Dump Block 1002] INFO  org.ton.actions.MyLocalTon - (-1,8000000000000000,1003)
17:31:39.310 [MyLocalTon - Dump Block 1003] INFO  org.ton.actions.MyLocalTon - (-1,8000000000000000,1004)
17:31:40.310 [MyLocalTon - Dump Block 1004] INFO  org.ton.actions.MyLocalTon - (-1,8000000000000000,1005)
17:31:41.310 [MyLocalTon - Dump Block 1005] INFO  org.ton.actions.MyLocalTon - (-1,8000000000000000,1006)
17:31:42.311 [MyLocalTon - Dump Block 1006] INFO  org.ton.actions.MyLocalTon - (-1,8000000000000000,1007)
17:31:43.299 [MyLocalTon - node3 Status Monitor] INFO  org.ton.actions.MyLocalTon - node3 is ready
17:31:43.299 [MyLocalTon - node2 Status Monitor] INFO  org.ton.actions.MyLocalTon - node2 is ready
17:31:43.300 [MyLocalTon - node4 Status Monitor] INFO  org.ton.actions.MyLocalTon - node4 is ready
17:31:43.300 [MyLocalTon - genesis Status Monitor] INFO  org.ton.actions.MyLocalTon - genesis is ready
17:31:44.312 [MyLocalTon - Dump Block 1007] INFO  org.ton.actions.MyLocalTon - (-1,8000000000000000,1008)
17:31:45.312 [MyLocalTon - Dump Block 1008] INFO  org.ton.actions.MyLocalTon - (-1,8000000000000000,1009)
17:31:46.312 [MyLocalTon - Dump Block 1009] INFO  org.ton.actions.MyLocalTon - (-1,8000000000000000,1010)
17:31:47.312 [MyLocalTon - Dump Block 1010] INFO  org.ton.actions.MyLocalTon - (-1,8000000000000000,1011)
17:31:48.313 [MyLocalTon - Dump Block 1011] INFO  org.ton.actions.MyLocalTon - (-1,8000000000000000,1012)
17:31:50.314 [MyLocalTon - Dump Block 1012] INFO  org.ton.actions.MyLocalTon - (-1,8000000000000000,1013)
17:31:53.300 [MyLocalTon - node3 Status Monitor] INFO  org.ton.actions.MyLocalTon - node3 is ready
17:31:53.300 [MyLocalTon - node4 Status Monitor] INFO  org.ton.actions.MyLocalTon - node4 is ready
17:31:53.300 [MyLocalTon - genesis Status Monitor] INFO  org.ton.actions.MyLocalTon - genesis is ready
17:31:53.300 [MyLocalTon - node2 Status Monitor] INFO  org.ton.actions.MyLocalTon - node2 is ready
17:32:13.302 [MyLocalTon - node4 Status Monitor] INFO  org.ton.actions.MyLocalTon - node4 is not ready
17:32:13.304 [MyLocalTon - genesis Status Monitor] INFO  org.ton.actions.MyLocalTon - genesis is not ready
17:32:13.307 [MyLocalTon - node2 Status Monitor] INFO  org.ton.actions.MyLocalTon - node2 is not ready
17:32:13.309 [MyLocalTon - node3 Status Monitor] INFO  org.ton.actions.MyLocalTon - node3 is not ready
17:32:19.532 [MyLocalTon - Accounts Monitor] WARN  org.ton.utils.MyLocalTonUtils - cannot identify wallet version by the code E74D
17:32:23.303 [MyLocalTon - node4 Status Monitor] INFO  org.ton.actions.MyLocalTon - node4 is not ready
17:32:23.304 [MyLocalTon - node2 Status Monitor] INFO  org.ton.actions.MyLocalTon - node2 is not ready
17:32:23.304 [MyLocalTon - genesis Status Monitor] INFO  org.ton.actions.MyLocalTon - genesis is not ready
17:32:23.308 [MyLocalTon - node3 Status Monitor] INFO  org.ton.actions.MyLocalTon - node3 is not ready
17:32:33.301 [MyLocalTon - node2 Status Monitor] INFO  org.ton.actions.MyLocalTon - node2 is not ready
17:32:33.301 [MyLocalTon - node3 Status Monitor] INFO  org.ton.actions.MyLocalTon - node3 is not ready
17:32:33.302 [MyLocalTon - node4 Status Monitor] INFO  org.ton.actions.MyLocalTon - node4 is not ready
17:32:33.308 [MyLocalTon - genesis Status Monitor] INFO  org.ton.actions.MyLocalTon - genesis is not ready
17:32:39.539 [MyLocalTon - Accounts Monitor] WARN  org.ton.utils.MyLocalTonUtils - cannot identify wallet version by the code E74D
17:32:43.300 [MyLocalTon - genesis Status Monitor] INFO  org.ton.actions.MyLocalTon - genesis is not ready
17:32:43.300 [MyLocalTon - node3 Status Monitor] INFO  org.ton.actions.MyLocalTon - node3 is not ready
17:32:43.307 [MyLocalTon - node2 Status Monitor] INFO  org.ton.actions.MyLocalTon - node2 is not ready
17:32:43.309 [MyLocalTon - node4 Status Monitor] INFO  org.ton.actions.MyLocalTon - node4 is not ready
17:32:46.502 [MyLocalTon - Validation Monitor] ERROR org.ton.actions.MyLocalTon - Error getting blockchain configuration! Error Cannot parse null string
17:32:46.502 [MyLocalTon - Validation Monitor] ERROR org.ton.actions.MyLocalTon - java.lang.NumberFormatException: Cannot parse null string
        at java.base/java.lang.Long.parseLong(Long.java:672)
        at java.base/java.lang.Long.parseLong(Long.java:832)
        at org.ton.executors.liteclient.LiteClientParser.parseConfig12(LiteClientParser.java:224)
        at org.ton.utils.MyLocalTonUtils.getConfig(MyLocalTonUtils.java:636)
        at org.ton.actions.MyLocalTon.lambda$runValidationMonitor$10(MyLocalTon.java:574)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
        at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:358)
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1583)

17:32:53.301 [MyLocalTon - genesis Status Monitor] INFO  org.ton.actions.MyLocalTon - genesis is not ready
17:32:53.301 [MyLocalTon - node3 Status Monitor] INFO  org.ton.actions.MyLocalTon - node3 is not ready
17:32:53.304 [MyLocalTon - node2 Status Monitor] INFO  org.ton.actions.MyLocalTon - node2 is not ready
17:32:53.309 [MyLocalTon - node4 Status Monitor] INFO  org.ton.actions.MyLocalTon - node4 is not ready
17:32:59.549 [MyLocalTon - Accounts Monitor] WARN  org.ton.utils.MyLocalTonUtils - cannot identify wallet version by the code E74D
17:33:03.298 [MyLocalTon - node2 Status Monitor] INFO  org.ton.actions.MyLocalTon - node2 is not ready
17:33:03.298 [MyLocalTon - node3 Status Monitor] INFO  org.ton.actions.MyLocalTon - node3 is not ready
17:33:03.298 [MyLocalTon - node4 Status Monitor] INFO  org.ton.actions.MyLocalTon - node4 is not ready
17:33:03.298 [MyLocalTon - genesis Status Monitor] INFO  org.ton.actions.MyLocalTon - genesis is not ready
17:33:13.298 [MyLocalTon - genesis Status Monitor] INFO  org.ton.actions.MyLocalTon - genesis is not ready
17:33:13.298 [MyLocalTon - node4 Status Monitor] INFO  org.ton.actions.MyLocalTon - node4 is not ready
17:33:13.298 [MyLocalTon - node3 Status Monitor] INFO  org.ton.actions.MyLocalTon - node3 is not ready
17:33:13.298 [MyLocalTon - node2 Status Monitor] INFO  org.ton.actions.MyLocalTon - node2 is not ready
17:33:19.557 [MyLocalTon - Accounts Monitor] WARN  org.ton.utils.MyLocalTonUtils - cannot identify wallet version by the code E74D
17:33:23.298 [MyLocalTon - node3 Status Monitor] INFO  org.ton.actions.MyLocalTon - node3 is not ready
17:33:23.298 [MyLocalTon - genesis Status Monitor] INFO  org.ton.actions.MyLocalTon - genesis is not ready
17:33:23.299 [MyLocalTon - node2 Status Monitor] INFO  org.ton.actions.MyLocalTon - node2 is not ready
17:33:23.299 [MyLocalTon - node4 Status Monitor] INFO  org.ton.actions.MyLocalTon - node4 is not ready
17:33:33.298 [MyLocalTon - genesis Status Monitor] INFO  org.ton.actions.MyLocalTon - genesis is not ready
17:33:33.298 [MyLocalTon - node4 Status Monitor] INFO  org.ton.actions.MyLocalTon - node4 is not ready
17:33:33.298 [MyLocalTon - node3 Status Monitor] INFO  org.ton.actions.MyLocalTon - node3 is not ready
17:33:33.301 [MyLocalTon - node2 Status Monitor] INFO  org.ton.actions.MyLocalTon - node2 is not ready
17:33:39.561 [MyLocalTon - Accounts Monitor] WARN  org.ton.utils.MyLocalTonUtils - cannot identify wallet version by the code E74D

[dkatzan]

I also run with 3, mine died after 24 hours, 77k blocks

stck trace a bit different:

2024-08-11 16:42:00 13:42:00.141 [MyLocalTon - Dump Block 77360] INFO  org.ton.actions.MyLocalTon - (-1,8000000000000000,77361)
2024-08-11 16:42:00 13:42:00.481 [MyLocalTon - Participation in elections by node3] ERROR org.ton.wallet.MyWallet - java.lang.Error: malformed response from run_method
2024-08-11 16:42:00     at org.ton.java.tonlib.RunResultParser.lambda$new$5(RunResultParser.java:84)
2024-08-11 16:42:00     at com.google.gson.internal.bind.TreeTypeAdapter.read(TreeTypeAdapter.java:69)
2024-08-11 16:42:00     at com.google.gson.Gson.fromJson(Gson.java:991)
2024-08-11 16:42:00     at com.google.gson.Gson.fromJson(Gson.java:956)
2024-08-11 16:42:00     at com.google.gson.Gson.fromJson(Gson.java:905)
2024-08-11 16:42:00     at com.google.gson.Gson.fromJson(Gson.java:876)
2024-08-11 16:42:00     at org.ton.java.tonlib.RunResultParser.parse(RunResultParser.java:114)
2024-08-11 16:42:00     at org.ton.java.tonlib.Tonlib.runMethod(Tonlib.java:818)
2024-08-11 16:42:00     at org.ton.java.tonlib.Tonlib.runMethod(Tonlib.java:785)
2024-08-11 16:42:00     at org.ton.java.tonlib.Tonlib.getSeqno(Tonlib.java:829)
2024-08-11 16:42:00     at org.ton.wallet.MyWallet.sendTonCoins(MyWallet.java:114)
2024-08-11 16:42:00     at org.ton.utils.MyLocalTonUtils.participate(MyLocalTonUtils.java:748)
2024-08-11 16:42:00     at org.ton.actions.MyLocalTon.lambda$participateAll$11(MyLocalTon.java:638)
2024-08-11 16:42:00     at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
2024-08-11 16:42:00     at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
2024-08-11 16:42:00     at java.base/java.lang.Thread.run(Thread.java:840)
2024-08-11 16:42:00 
2024-08-11 16:42:00 13:42:00.481 [MyLocalTon - Participation in elections by node3] ERROR org.ton.utils.MyLocalTonUtils - Participation error. node3 failed to send 10001000000000 Toncoins to -1:3333333333333333333333333333333333333333333333333333333333333333
2024-08-11 16:42:00 13:42:00.481 [MyLocalTon - Participation in elections by node3] ERROR org.ton.utils.MyLocalTonUtils - Error by node3 participating in elections! Error Cannot invoke "org.ton.ui.controllers.MainController.showErrorMsg(String, double)" because "org.ton.main.App.mainController" is null
2024-08-11 16:42:00 13:42:00.886 [MyLocalTon - Participation in elections by genesis] ERROR org.ton.wallet.MyWallet - java.lang.Error: malformed response from run_method
2024-08-11 16:42:00     at org.ton.java.tonlib.RunResultParser.lambda$new$5(RunResultParser.java:84)
2024-08-11 16:42:00     at com.google.gson.internal.bind.TreeTypeAdapter.read(TreeTypeAdapter.java:69)
2024-08-11 16:42:00     at com.google.gson.Gson.fromJson(Gson.java:991)
2024-08-11 16:42:00     at com.google.gson.Gson.fromJson(Gson.java:956)
2024-08-11 16:42:00     at com.google.gson.Gson.fromJson(Gson.java:905)
2024-08-11 16:42:00     at com.google.gson.Gson.fromJson(Gson.java:876)
2024-08-11 16:42:00     at org.ton.java.tonlib.RunResultParser.parse(RunResultParser.java:114)
2024-08-11 16:42:00     at org.ton.java.tonlib.Tonlib.runMethod(Tonlib.java:818)
2024-08-11 16:42:00     at org.ton.java.tonlib.Tonlib.runMethod(Tonlib.java:785)
2024-08-11 16:42:00     at org.ton.java.tonlib.Tonlib.getSeqno(Tonlib.java:829)
2024-08-11 16:42:00     at org.ton.wallet.MyWallet.sendTonCoins(MyWallet.java:114)
2024-08-11 16:42:00     at org.ton.utils.MyLocalTonUtils.participate(MyLocalTonUtils.java:748)
2024-08-11 16:42:00     at org.ton.actions.MyLocalTon.lambda$participateAll$11(MyLocalTon.java:638)
2024-08-11 16:42:00     at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
2024-08-11 16:42:00     at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
2024-08-11 16:42:00     at java.base/java.lang.Thread.run(Thread.java:840)
2024-08-11 16:42:00 

[dkatzan]

Hi @neodix42 , any updates here? I saw u worked on some improvements last week

I'm using the latest v119 version, but running a local node is still very flaky, validators seems to be sporadically failing, most of the times coming back up, but at some point they all fail and the node is dead

also, while running, I'm seeing networking issues of no lite servers being able while trying to use for exmaple ton-http-api endpoints or pytoniq to issue some request

Help will be greatly appreciated!

[neodix42]

The problem was much bigger. It seems that ton binaries were somehow corrupted and were causing core dump failures. Please try out this version https://github.com/neodix42/MyLocalTon/releases/tag/v120

[dkatzan]

Hi, the new version seems to be much more stable for now

I'm running my node with 6 validators as per your previous suggestion, which seems to be taking up a lot of resources, should it be safe now to reduce it back to 1?

[neodix42]

Having 6 validators it's possible to remove two, after that AFAIR you can remove one, but not two, otherwise the network will lose its consensus. When you have 3 left, you won't be able to remove a single validator.