Corrupted DB after extraction

[MichaelMichaelMichaelMichaelMichael]

Hi, very often, after extraction of the utxo, I get a corrupted db. Is there any way to prevent that? I'm copying around the DB files, making backup & restore, but that's annoying and time consuming after while. Edit: Here's an example:

2019-12-29T15:21:44Z Verifying last 6 blocks at level 3
2019-12-29T15:21:44Z [0%]...[16%]...[33%]...[50%]...[66%]...[83%]...[99%]...ERROR: VerifyDB(): *** coin database inconsistencies found (last 6 blocks, 0 good transactions before that)

2019-12-29T15:21:46Z : Corrupted block database detected.
Please restart with -reindex or -reindex-chainstate to recover.
2019-12-29T15:21:46Z Aborted block database rebuild. Exiting.

[in3rsha]

I corrupted my chainstate whilst building this tool, and it wasn't a fun experience, so I'm sorry to hear you've experienced the same thing.

Does your bitcoind automatically restart itself via systemd or something like that?

I just tested the script locally and it didn't corrupt my local node. Nonetheless, LevelDB really doesn't like being accessed at the same time, which is why I've tried to prevent this script from running against an active bitcoind node. However, if you've set bitcoind to automatically start back up after shutting it down, it will end up accessing LevelDB at the same time and possibly corrupt it.

So in short, is there anything in the background causing bitcoind to run whilst your running this script?

[lamengao]

I have the same problem. Before run bitcoin-utxo-dump I make sure bitcoind is stopped.

[in3rsha]

I have the same problem. Before run bitcoin-utxo-dump I make sure bitcoind is stopped.

Just to confirm, using this tool causes your chainstate to become corrupted?

If so, have you set up bitcoind to run automatically as a service (e.g. with a file like /etc/systemd/system/bitcoind.service)?

You might have done this if you followed the Stadicus RaspiBolt guide.

[lamengao]

I have the same problem. Before run bitcoin-utxo-dump I make sure bitcoind is stopped.

Just to confirm, using this tool causes your chainstate to become corrupted?

If so, have you set up bitcoind to run automatically as a service (e.g. with a file like /etc/systemd/system/bitcoind.service)?

You might have done this if you followed the Stadicus RaspiBolt guide.

I have /etc/systemd/system/bitcoind.service file, but it's disabled.

pi@raspberrypi:~/.bitcoin $ systemctl status bitcoind
● bitcoind.service - Bitcoin daemon
   Loaded: loaded (/etc/systemd/system/bitcoind.service; disabled; vendor preset: enabled)
   Active: inactive (dead)

I have run bitcoin-utxo-dump 3 times, 2 times need reindex

[in3rsha]

Thanks, I just tested and had to -reindex-chainstate too. It's not happening all the time, and I'm not sure what is causing it.

I will look in to it to see if I can fix it.

If I can't find a reliable solution, I will make the script create a copy of the chainstate folder and access that instead (then delete it afterwards).

[MichaelMichaelMichaelMichaelMichael]

Does your bitcoind automatically restart itself via systemd or something like that?

Nope, if I say stop then it's stop. My service file is like this:

[Unit]
Description=Bitcoin daemon
After=network.target

[Service]
ExecStart=/usr/local/bin/bitcoind -daemon \
                            -pid=/run/bitcoind/bitcoind.pid \
                            -conf=/etc/bitcoin/bitcoin.conf \
                            -datadir=/var/lib/bitcoind \
                            -txindex=1 

PermissionsStartOnly=true
ExecStartPre=/bin/chgrp bitcoin /etc/bitcoin
Type=forking
PIDFile=/run/bitcoind/bitcoind.pid
Restart=on-failure
TimeoutStopSec=600
User=bitcoin
Group=bitcoin
RuntimeDirectory=bitcoind
RuntimeDirectoryMode=0710
ConfigurationDirectory=bitcoin
ConfigurationDirectoryMode=0710
StateDirectory=bitcoind
StateDirectoryMode=0710

# Hardening measures
PrivateTmp=true
ProtectSystem=full
ProtectHome=true
NoNewPrivileges=true
PrivateDevices=true
MemoryDenyWriteExecute=true

[Install]
WantedBy=multi-user.target

I just tested the script locally and it didn't corrupt my local node. Nonetheless, LevelDB really doesn't like being accessed at the same time, which is why I've tried to prevent this script from running against an active bitcoind node. However, if you've set bitcoind to automatically start back up after shutting it down, it will end up accessing LevelDB at the same time and possibly corrupt it.

I've also tested it several times since I've opened the issue - it's like waiting for a squirrel to watch it, it wouldn't happen. Right now its fine, but it happened several times before.

So in short, is there anything in the background causing bitcoind to run whilst your running this script?

That would be a good measure to implement before launching the extract: do an fuser

[in3rsha]

I've also tested it several times since I've opened the issue - it's like waiting for a squirrel to watch it, it wouldn't happen. Right now its fine, but it happened several times before.

I know what you mean.

This makes it annoyingly tricky to pinpoint, so if I can't reliably figure out what's going on I'll make the script do a backup and restore instead (so the the chainstate never gets accessed directly).

That would be a good measure to implement before launching the extract: do an fuser

Didn't think of that, will look in to it. Thanks.

[in3rsha]

I've made some changes to hopefully prevent corrupting the database.

I've basically changed the way the tool closes the database (releasing the database iterator first explictly instead of deferring it). I've also added signal catching to ensure the database gets closed even if you do a CTRL-C to interrupt the script whilst it's running.

I've tested it about 30 times and haven't been able to corrupt my chainstate. Please could you update to the latest version and try it out for me:

go get -u github.com/in3rsha/bitcoin-utxo-dump

Note: You can check you've got the latest version by doing bitcoin-utxo-dump -version. If you get 1.0.1 you are on this new version.

In all honesty I can't promise that this isn't going to corrupt your chainstate database again, but I hope it won't. Ideally I'd prefer to not copy the chainstate before working on it (to save on hard disk usage and time), so I've done what I can to try and get it working without corrupting.

If you can test it out for me I'd greatly appreciate it. If it corrupts your chainstate again, I sincerely apologise and will update the tool to copy the chainstate database instead.

[MichaelMichaelMichaelMichaelMichael]

I've made some changes to hopefully prevent corrupting the database.

Thank you very much - I appreciate your effort and your excellent support! I've just started with the Brave browser - as soon as I have something noticable accumulated in terms of BAT, I will send it over. @ all Others: Please do the same!

I've tested it once and it worked fine.

2020-01-09T18:14:00Z No coin database inconsistencies in last 6 blocks (14464 transactions)

:)

[in3rsha]

@MichaelMichaelMichaelMichaelMichael No problem at all, thanks for raising the issue.

I will keep the issue open for now in case the corruption happens again, but I'm happy to hear it's working so far.