search

iotaledger / iri

IOTA Reference Implementation
Other
1.15k stars 370 forks source link

error persisting newly received tx org.rocksdb.RocksDBException: #1813

Open damageco opened 4 years ago

damageco commented 4 years ago

Bug description

After a few hours, my node start to raise RocksDBException on new transaction persisting.

IRI version

1.8.5

Hardware Spec

Bare metal machine. 8 CPU cores i7 8 GB RAM 1To + 2To disk 1 GBit/s Traffic Unlimited

Errors

'm was using 1.8.4, I've upgraded to 1.8.5 yesterday, but the errors always appears after of few hours. I've tried to change the Linux distribution, RedHat, Ubuntu, same problem.

Got this stacktrace : ERROR ReceivedStage:57 - error persisting newly received tx org.rocksdb.RocksDBException: block checksum mismatch: expected 1900677990, got 1492066657 in /iri/data//002100.sst offset 26773563 size 1276

GalRogozinski commented 4 years ago

This seems like an underlying issue of RocksDB issue. According to this ticket it will be hard to solve...

but the errors always appears after of few hours

Are you starting from the same DB? Because once it has been corrupted it is probably hard to fix?

The only thing I can recommend atm is to try to recompile IRI with different versions of RocksDb. You can do this by recompiling IRI with mvn clean package and changing the version in the pom.xml in this section

<dependency>
            <groupId>org.rocksdb</groupId>
            <artifactId>rocksdbjni</artifactId>
            <version>6.3.6</version>
</dependency>

Perhaps upgrading to a later version can help?

damageco commented 4 years ago

Ok, let's try it. I've upgraded to 6.6.4 and start a new node. By the way, the last 6.7.3 version of RocksDb change the public API and breaks compilation.

After 5 minutes running I've got this error : 03/29 12:31:32.268 [Milestone Solidifier] ERROR MilestoneSolidifier:465 - Error while solidifying milestone #1361816 org.rocksdb.RocksDBException: block checksum mismatch: expected 1674591948, got 3341790192 in mainnetdb/000463.sst offset 46314903 size 3030

I dig the net, but I can't find way to fix the corrupted files. So, yes, I always have to start with a new snapshot database.

GalRogozinski commented 4 years ago

If it keeps happening to you after a snapshot report please :-)

dsampaolo commented 4 years ago

It should be RocksDB related. I'm using ArangoDB (which uses RocksDB) and I have exactly the same problem. This doesn't seem to be related to iotaledger...