Slow Syncing (e.g Bootstrapping) broken after V23.X

[nano2dev]

Summary

Just opening up this issue for the link.

This issue is only affecting new Nodes.

FIX: Downgrade to V22-V23.0 ... wait for sync and then upgrade to V23.3

I'm still testing to see which exact version broke syncing.

Node version

23.3

Build details

N/A

OS and version

Ubuntu

Steps to reproduce the behavior

1. Setup production Node with V23.3 and let it sync.

Expected behavior

Auto synced after 1-2 days.

Actual behavior

Stuck for weeks.

Possible solution

N/A

Supporting files

No response

[qwahzi]

For more context, I believe this has been a known issue since V22:

https://github.com/nanocurrency/nano-node/issues/3294

https://github.com/nanocurrency/nano-node/issues/3578

https://github.com/nanocurrency/nano-node/issues/3580

Some of the V24 beta builds have bootstrapping changes (e.g. https://github.com/nanocurrency/nano-node/issues/3837?) that seemed to improve beta bootstrapping, so hopefully those changes get tested/deployed with similar benefits on mainnet

[vorotech]

The sync is definitely broken. I've downloaded latest available state archive 3 days ago, and since that I'm trying to catch up the latest blocks. Currently is 163858533, but my system has 162962290. With the current speed, I'm not sure I will be able to sync at all – it feels that new blocks generated at the same speed I'm catching them up. The OS resources (CPU/memory/disk IO) does't seem to be busy much. :(

[BatuAksoy]

Experiencing the same issue.

[qwahzi]

Do y'all have the same issue with V24RC2? I started bootstrapping a fresh V24RC2 node on December 22nd, and its been going pretty well so far, even without the new bootstrapping client. Here's what it looks like <6 days in:

    "count": "136908402",
    "unchecked": "35821345",
    "cemented": "91323476"
}

EDIT: Started slowing down significantly: ~12 days in now:

    "count": "147347191",
    "unchecked": "25696703",
    "cemented": "129321504"
}

[robertol]

I'm getting the same issue as quahzi related

[qwahzi]

I'm getting the same issue as quahzi related

@robertol This should be fixed with the upcoming V25 release that includes the ascended bootstrapping client & optimistic elections, otherwise you'll have to make a custom build using the optimistic elections branch (and/or manually updating rep_weights_live.bin with a more up-to-date one generated from another node)

[qwahzi]

@nano2dev This should be fixed with the current V25 tag. Seeing multiple mainnet bootstraps at >1M/hr cemented