search

ava-labs / avalanchego

Go implementation of an Avalanche node.
https://avax.network
BSD 3-Clause "New" or "Revised" License
2.12k stars 669 forks source link

Unusual disk usage when bootstrapping from scratch with v1.11.6 #3060

Closed roonodz closed 4 months ago

roonodz commented 4 months ago

Describe the bug I'm observing erratic disk usage while bootstrapping from scratch using the latest v1.11.6. The configuration file is provided below, and state-sync is enabled.

The last time I performed this operation, it completed in less than 48 hours. The attached screen captures show the disk usage differences between v1.11.5 and v1.11.6.

Notably, the configuration and server specifications are identical for both runs, with the only variable being the avalanchego version. The server specs are as follows:

Please see the screen captures and configuration file below for further details.

To Reproduce Start a new instance with an empty database with the following C-Chain configuration:

{
    "state-sync-enabled": true
}

Expected behavior

Reduced synchronization time and, more importantly, lower disk usage. As shown in the screenshot, there was a spike exceeding 80%. This issue might be related to a Core-ETH bug that I've already reported. Here is the fix: https://github.com/ava-labs/coreth/pull/565

Screenshots Here is a screenshot of the disk usage from a bootstrap I performed two weeks ago using v1.11.5: avalanchego-v1 11 5

Here is a screenshot of the disk usage from the current bootstrap with the v1.11.6: avalanchego-v1 11 6

Logs Please find attached the log file.

Metrics Please find attached the metrics.log

Operating System Ubuntu 22.04 LTS

To best protect the Avalanche community security bugs should be reported in accordance to our Security Policy metrics.log

StephenButtolph commented 4 months ago

@roonodz I see that you said Please find attached the log file. but I don't actually see a log file, did you mean to attach one?

roonodz commented 4 months ago

Please find attached the log file. Apologies for the oversight, I forgot to include it earlier. avalanchego.log

asnogordo commented 4 months ago

avalanchego_1.11.6_boostrap_logs.tar.gz

Adding logs from my validator - similar issue. You'll see in the last 12 hours, I'm bootstrapping on the older version, but the 1.11.6 logs are all there too. Ignore the wrong version name on the tar ball.

My Setup: Public Cloud 16vCPU, 32GB memory, 200GB of storage space.

StephenButtolph commented 4 months ago

The linked PR seems to have addressed the the performance regression during my testing... But if any additional issues come up with the syncing flow (in either v1.11.7 or master after the linked PR was merged) please re-open this issue or make a new issue. Thanks for all the details!