Closed longfin closed 1 year ago
RocksDBStore
currently does quite many Pokémon exception handling (see also: #1277), so it apparently has some corner cases that data is partially written due to silent failure. I believe this specific case was caused because RocksDBStore.PutBlock<T>()
operation always commits the BlockDigest
without any exceptions even when some RocksDBStore.PutTransaction<T>()
operations it invokes silently fail.
IMHO it should be changed to throw an exception on failure rather than suppressing it, in general.
FYI, @longfin left a comment to suggest throwing exceptions instead of suppressing them, even though the rationale behind the comment differs from this context, but it apparently wasn't applied at that time.
Possibly related: https://github.com/planetarium/libplanet/issues/1539.
Apparently it's still unresolved yet, or at least we made a regression recently:
[07:40:12 ERR] An unexpected exception occurred on GetTransaction: Buffer cannot be null. (Parameter 'buffer')
System.ArgumentNullException: Buffer cannot be null. (Parameter 'buffer')
at System.IO.MemoryStream..ctor(Byte[] buffer, Boolean writable)
at Libplanet.Tx.Transaction`1.Deserialize(Byte[] bytes, Boolean validate) in /app/Lib9c/.Libplanet/Libplanet/Tx/Transaction.cs:line 279
at Libplanet.RocksDBStore.RocksDBStore.GetTransaction[T](TxId txid) in /app/Lib9c/.Libplanet/Libplanet.RocksDBStore/RocksDBStore.cs:line 492
[07:40:12 ERR] An unexpected exception occurred on GetTransaction: Buffer cannot be null. (Parameter 'buffer')
System.ArgumentNullException: Buffer cannot be null. (Parameter 'buffer')
at System.IO.MemoryStream..ctor(Byte[] buffer, Boolean writable)
at Libplanet.Tx.Transaction`1.Deserialize(Byte[] bytes, Boolean validate) in /app/Lib9c/.Libplanet/Libplanet/Tx/Transaction.cs:line 279
at Libplanet.RocksDBStore.RocksDBStore.GetTransaction[T](TxId txid) in /app/Lib9c/.Libplanet/Libplanet.RocksDBStore/RocksDBStore.cs:line 492
[07:40:12 ERR] An unexpected exception occurred on GetTransaction: Buffer cannot be null. (Parameter 'buffer')
System.ArgumentNullException: Buffer cannot be null. (Parameter 'buffer')
at System.IO.MemoryStream..ctor(Byte[] buffer, Boolean writable)
at Libplanet.Tx.Transaction`1.Deserialize(Byte[] bytes, Boolean validate) in /app/Lib9c/.Libplanet/Libplanet/Tx/Transaction.cs:line 279
at Libplanet.RocksDBStore.RocksDBStore.GetTransaction[T](TxId txid) in /app/Lib9c/.Libplanet/Libplanet.RocksDBStore/RocksDBStore.cs:line 492
[07:40:12 ERR] An unexpected exception occurred on GetTransaction: Buffer cannot be null. (Parameter 'buffer')
System.ArgumentNullException: Buffer cannot be null. (Parameter 'buffer')
at System.IO.MemoryStream..ctor(Byte[] buffer, Boolean writable)
at Libplanet.Tx.Transaction`1.Deserialize(Byte[] bytes, Boolean validate) in /app/Lib9c/.Libplanet/Libplanet/Tx/Transaction.cs:line 279
at Libplanet.RocksDBStore.RocksDBStore.GetTransaction[T](TxId txid) in /app/Lib9c/.Libplanet/Libplanet.RocksDBStore/RocksDBStore.cs:line 492
[07:40:12 ERR] An unexpected exception occurred on GetTransaction: Buffer cannot be null. (Parameter 'buffer')
System.ArgumentNullException: Buffer cannot be null. (Parameter 'buffer')
at System.IO.MemoryStream..ctor(Byte[] buffer, Boolean writable)
at Libplanet.Tx.Transaction`1.Deserialize(Byte[] bytes, Boolean validate) in /app/Lib9c/.Libplanet/Libplanet/Tx/Transaction.cs:line 279
at Libplanet.RocksDBStore.RocksDBStore.GetTransaction[T](TxId txid) in /app/Lib9c/.Libplanet/Libplanet.RocksDBStore/RocksDBStore.cs:line 492
[07:40:12 ERR] An unexpected exception occurred on GetTransaction: Buffer cannot be null. (Parameter 'buffer')
System.ArgumentNullException: Buffer cannot be null. (Parameter 'buffer')
at System.IO.MemoryStream..ctor(Byte[] buffer, Boolean writable)
at Libplanet.Tx.Transaction`1.Deserialize(Byte[] bytes, Boolean validate) in /app/Lib9c/.Libplanet/Libplanet/Tx/Transaction.cs:line 279
at Libplanet.RocksDBStore.RocksDBStore.GetTransaction[T](TxId txid) in /app/Lib9c/.Libplanet/Libplanet.RocksDBStore/RocksDBStore.cs:line 492
[07:40:12 ERR] An unexpected exception occurred on GetTransaction: Buffer cannot be null. (Parameter 'buffer')
System.ArgumentNullException: Buffer cannot be null. (Parameter 'buffer')
at System.IO.MemoryStream..ctor(Byte[] buffer, Boolean writable)
at Libplanet.Tx.Transaction`1.Deserialize(Byte[] bytes, Boolean validate) in /app/Lib9c/.Libplanet/Libplanet/Tx/Transaction.cs:line 279
at Libplanet.RocksDBStore.RocksDBStore.GetTransaction[T](TxId txid) in /app/Lib9c/.Libplanet/Libplanet.RocksDBStore/RocksDBStore.cs:line 492
[07:40:12 ERR] An unexpected exception occurred on GetTransaction: Buffer cannot be null. (Parameter 'buffer')
System.ArgumentNullException: Buffer cannot be null. (Parameter 'buffer')
at System.IO.MemoryStream..ctor(Byte[] buffer, Boolean writable)
at Libplanet.Tx.Transaction`1.Deserialize(Byte[] bytes, Boolean validate) in /app/Lib9c/.Libplanet/Libplanet/Tx/Transaction.cs:line 279
at Libplanet.RocksDBStore.RocksDBStore.GetTransaction[T](TxId txid) in /app/Lib9c/.Libplanet/Libplanet.RocksDBStore/RocksDBStore.cs:line 492
[07:40:12 ERR] An unexpected exception occurred on GetTransaction: Buffer cannot be null. (Parameter 'buffer')
System.ArgumentNullException: Buffer cannot be null. (Parameter 'buffer')
at System.IO.MemoryStream..ctor(Byte[] buffer, Boolean writable)
at Libplanet.Tx.Transaction`1.Deserialize(Byte[] bytes, Boolean validate) in /app/Lib9c/.Libplanet/Libplanet/Tx/Transaction.cs:line 279
at Libplanet.RocksDBStore.RocksDBStore.GetTransaction[T](TxId txid) in /app/Lib9c/.Libplanet/Libplanet.RocksDBStore/RocksDBStore.cs:line 492
[07:40:12 ERR] An unexpected exception occurred on GetTransaction: Buffer cannot be null. (Parameter 'buffer')
System.ArgumentNullException: Buffer cannot be null. (Parameter 'buffer')
at System.IO.MemoryStream..ctor(Byte[] buffer, Boolean writable)
at Libplanet.Tx.Transaction`1.Deserialize(Byte[] bytes, Boolean validate) in /app/Lib9c/.Libplanet/Libplanet/Tx/Transaction.cs:line 279
at Libplanet.RocksDBStore.RocksDBStore.GetTransaction[T](TxId txid) in /app/Lib9c/.Libplanet/Libplanet.RocksDBStore/RocksDBStore.cs:line 492
[07:40:12 ERR] An unexpected exception occurred on GetTransaction: Buffer cannot be null. (Parameter 'buffer')
System.ArgumentNullException: Buffer cannot be null. (Parameter 'buffer')
at System.IO.MemoryStream..ctor(Byte[] buffer, Boolean writable)
at Libplanet.Tx.Transaction`1.Deserialize(Byte[] bytes, Boolean validate) in /app/Lib9c/.Libplanet/Libplanet/Tx/Transaction.cs:line 279
at Libplanet.RocksDBStore.RocksDBStore.GetTransaction[T](TxId txid) in /app/Lib9c/.Libplanet/Libplanet.RocksDBStore/RocksDBStore.cs:line 492
[07:40:12 ERR] An unexpected exception occurred on GetTransaction: Buffer cannot be null. (Parameter 'buffer')
System.ArgumentNullException: Buffer cannot be null. (Parameter 'buffer')
at System.IO.MemoryStream..ctor(Byte[] buffer, Boolean writable)
at Libplanet.Tx.Transaction`1.Deserialize(Byte[] bytes, Boolean validate) in /app/Lib9c/.Libplanet/Libplanet/Tx/Transaction.cs:line 279
at Libplanet.RocksDBStore.RocksDBStore.GetTransaction[T](TxId txid) in /app/Lib9c/.Libplanet/Libplanet.RocksDBStore/RocksDBStore.cs:line 492
[07:40:12 ERR] An unexpected exception occurred on GetTransaction: Buffer cannot be null. (Parameter 'buffer')
System.ArgumentNullException: Buffer cannot be null. (Parameter 'buffer')
at System.IO.MemoryStream..ctor(Byte[] buffer, Boolean writable)
at Libplanet.Tx.Transaction`1.Deserialize(Byte[] bytes, Boolean validate) in /app/Lib9c/.Libplanet/Libplanet/Tx/Transaction.cs:line 279
at Libplanet.RocksDBStore.RocksDBStore.GetTransaction[T](TxId txid) in /app/Lib9c/.Libplanet/Libplanet.RocksDBStore/RocksDBStore.cs:line 492
[07:40:12 ERR] An unexpected exception occurred on GetTransaction: Buffer cannot be null. (Parameter 'buffer')
System.ArgumentNullException: Buffer cannot be null. (Parameter 'buffer')
at System.IO.MemoryStream..ctor(Byte[] buffer, Boolean writable)
at Libplanet.Tx.Transaction`1.Deserialize(Byte[] bytes, Boolean validate) in /app/Lib9c/.Libplanet/Libplanet/Tx/Transaction.cs:line 279
at Libplanet.RocksDBStore.RocksDBStore.GetTransaction[T](TxId txid) in /app/Lib9c/.Libplanet/Libplanet.RocksDBStore/RocksDBStore.cs:line 492
[07:40:12 ERR] An unexpected exception occurred on GetTransaction: Buffer cannot be null. (Parameter 'buffer')
System.ArgumentNullException: Buffer cannot be null. (Parameter 'buffer')
at System.IO.MemoryStream..ctor(Byte[] buffer, Boolean writable)
at Libplanet.Tx.Transaction`1.Deserialize(Byte[] bytes, Boolean validate) in /app/Lib9c/.Libplanet/Libplanet/Tx/Transaction.cs:line 279
at Libplanet.RocksDBStore.RocksDBStore.GetTransaction[T](TxId txid) in /app/Lib9c/.Libplanet/Libplanet.RocksDBStore/RocksDBStore.cs:line 492
[07:40:12 ERR] An unexpected exception occurred on GetTransaction: Buffer cannot be null. (Parameter 'buffer')
System.ArgumentNullException: Buffer cannot be null. (Parameter 'buffer')
at System.IO.MemoryStream..ctor(Byte[] buffer, Boolean writable)
at Libplanet.Tx.Transaction`1.Deserialize(Byte[] bytes, Boolean validate) in /app/Lib9c/.Libplanet/Libplanet/Tx/Transaction.cs:line 279
at Libplanet.RocksDBStore.RocksDBStore.GetTransaction[T](TxId txid) in /app/Lib9c/.Libplanet/Libplanet.RocksDBStore/RocksDBStore.cs:line 492
[07:40:12 ERR] An unexpected exception occurred on GetTransaction: Buffer cannot be null. (Parameter 'buffer')
System.ArgumentNullException: Buffer cannot be null. (Parameter 'buffer')
at System.IO.MemoryStream..ctor(Byte[] buffer, Boolean writable)
at Libplanet.Tx.Transaction`1.Deserialize(Byte[] bytes, Boolean validate) in /app/Lib9c/.Libplanet/Libplanet/Tx/Transaction.cs:line 279
at Libplanet.RocksDBStore.RocksDBStore.GetTransaction[T](TxId txid) in /app/Lib9c/.Libplanet/Libplanet.RocksDBStore/RocksDBStore.cs:line 492
[07:40:12 ERR] An unexpected exception occurred on GetTransaction: Buffer cannot be null. (Parameter 'buffer')
System.ArgumentNullException: Buffer cannot be null. (Parameter 'buffer')
at System.IO.MemoryStream..ctor(Byte[] buffer, Boolean writable)
at Libplanet.Tx.Transaction`1.Deserialize(Byte[] bytes, Boolean validate) in /app/Lib9c/.Libplanet/Libplanet/Tx/Transaction.cs:line 279
at Libplanet.RocksDBStore.RocksDBStore.GetTransaction[T](TxId txid) in /app/Lib9c/.Libplanet/Libplanet.RocksDBStore/RocksDBStore.cs:line 492
[07:40:12 ERR] An unexpected exception occurred on GetTransaction: Buffer cannot be null. (Parameter 'buffer')
System.ArgumentNullException: Buffer cannot be null. (Parameter 'buffer')
at System.IO.MemoryStream..ctor(Byte[] buffer, Boolean writable)
at Libplanet.Tx.Transaction`1.Deserialize(Byte[] bytes, Boolean validate) in /app/Lib9c/.Libplanet/Libplanet/Tx/Transaction.cs:line 279
at Libplanet.RocksDBStore.RocksDBStore.GetTransaction[T](TxId txid) in /app/Lib9c/.Libplanet/Libplanet.RocksDBStore/RocksDBStore.cs:line 492
[07:40:12 ERR] An unexpected exception occurred on GetTransaction: Buffer cannot be null. (Parameter 'buffer')
System.ArgumentNullException: Buffer cannot be null. (Parameter 'buffer')
at System.IO.MemoryStream..ctor(Byte[] buffer, Boolean writable)
at Libplanet.Tx.Transaction`1.Deserialize(Byte[] bytes, Boolean validate) in /app/Lib9c/.Libplanet/Libplanet/Tx/Transaction.cs:line 279
at Libplanet.RocksDBStore.RocksDBStore.GetTransaction[T](TxId txid) in /app/Lib9c/.Libplanet/Libplanet.RocksDBStore/RocksDBStore.cs:line 492
[07:40:12 ERR] An unexpected exception occurred on GetTransaction: Buffer cannot be null. (Parameter 'buffer')
System.ArgumentNullException: Buffer cannot be null. (Parameter 'buffer')
at System.IO.MemoryStream..ctor(Byte[] buffer, Boolean writable)
at Libplanet.Tx.Transaction`1.Deserialize(Byte[] bytes, Boolean validate) in /app/Lib9c/.Libplanet/Libplanet/Tx/Transaction.cs:line 279
at Libplanet.RocksDBStore.RocksDBStore.GetTransaction[T](TxId txid) in /app/Lib9c/.Libplanet/Libplanet.RocksDBStore/RocksDBStore.cs:line 492
[07:40:12 ERR] An unexpected exception occurred on GetTransaction: Buffer cannot be null. (Parameter 'buffer')
System.ArgumentNullException: Buffer cannot be null. (Parameter 'buffer')
at System.IO.MemoryStream..ctor(Byte[] buffer, Boolean writable)
at Libplanet.Tx.Transaction`1.Deserialize(Byte[] bytes, Boolean validate) in /app/Lib9c/.Libplanet/Libplanet/Tx/Transaction.cs:line 279
at Libplanet.RocksDBStore.RocksDBStore.GetTransaction[T](TxId txid) in /app/Lib9c/.Libplanet/Libplanet.RocksDBStore/RocksDBStore.cs:line 492
[07:40:12 ERR] An unexpected exception occurred on GetTransaction: Buffer cannot be null. (Parameter 'buffer')
System.ArgumentNullException: Buffer cannot be null. (Parameter 'buffer')
at System.IO.MemoryStream..ctor(Byte[] buffer, Boolean writable)
at Libplanet.Tx.Transaction`1.Deserialize(Byte[] bytes, Boolean validate) in /app/Lib9c/.Libplanet/Libplanet/Tx/Transaction.cs:line 279
at Libplanet.RocksDBStore.RocksDBStore.GetTransaction[T](TxId txid) in /app/Lib9c/.Libplanet/Libplanet.RocksDBStore/RocksDBStore.cs:line 492
[07:40:12 ERR] An unexpected exception occurred on GetTransaction: Buffer cannot be null. (Parameter 'buffer')
System.ArgumentNullException: Buffer cannot be null. (Parameter 'buffer')
at System.IO.MemoryStream..ctor(Byte[] buffer, Boolean writable)
at Libplanet.Tx.Transaction`1.Deserialize(Byte[] bytes, Boolean validate) in /app/Lib9c/.Libplanet/Libplanet/Tx/Transaction.cs:line 279
at Libplanet.RocksDBStore.RocksDBStore.GetTransaction[T](TxId txid) in /app...
AFAIK, we still can observe the corner cases where:
and in production we're using exclusively RocksDBStore
, but it might be a common bug of all subclasses of BaseStore
.
I believe there are three hypotheses to explain such situation:
PutBlock<T>()
may sometimes store only a block digest, while dropping some transactions in it for some unknown cause.PutBlock<T>()
may successfully store the entire block content and transactions in it together, but since then data could be broken for some unknown cause.GetBlock<T>()
may sometimes fail to load data even when they obviously exist, for some unknown cause.Note that such failure can happen for block digests as well as transactions, but in which case, we could see quite different situation as GetBlock<T>()
would silently return null
rather than throwing an exception.
In order to eliminate the first hypotheses, we can make RocksDBStore.PutBlock<T>()
to double-check if all transactions are properly stored before committing a block digest. If it's the first case we probably observe the failure early.
[2022-03-10 18:10:50.511] [info] [18:10:50 ERR] An unexpected exception occurred on GetBlockDigest: Buffer cannot be null. (Parameter 'buffer')
System.ArgumentNullException: Buffer cannot be null. (Parameter 'buffer')
at System.IO.MemoryStream..ctor(Byte[] buffer, Boolean writable)
at Bencodex.Codec.Decode(Byte[] bytes)
at Libplanet.Store.BlockDigest.Deserialize(Byte[] bytes) in C:\Users\circleci\project\NineChronicles.Headless\Lib9c\.Libplanet\Libplanet\Store\BlockDigest.cs:line 143
at Libplanet.RocksDBStore.RocksDBStore.GetBlockDigest(BlockHash blockHash) in C:\Users\circleci\project\NineChronicles.Headless\Lib9c\.Libplanet\Libplanet.RocksDBStore\RocksDBStore.cs:line 646
[2022-03-10 18:10:50.511] [info] [18:10:50 ERR] An unexpected exception occurred on GetBlockDigest: Buffer cannot be null. (Parameter 'buffer') System.ArgumentNullException: Buffer cannot be null. (Parameter 'buffer') at System.IO.MemoryStream..ctor(Byte[] buffer, Boolean writable) at Bencodex.Codec.Decode(Byte[] bytes) at Libplanet.Store.BlockDigest.Deserialize(Byte[] bytes) in C:\Users\circleci\project\NineChronicles.Headless\Lib9c\.Libplanet\Libplanet\Store\BlockDigest.cs:line 143 at Libplanet.RocksDBStore.RocksDBStore.GetBlockDigest(BlockHash blockHash) in C:\Users\circleci\project\NineChronicles.Headless\Lib9c\.Libplanet\Libplanet.RocksDBStore\RocksDBStore.cs:line 646
This shows the same errors also happen to block digests (not only transactions belonging to blocks). Therefore, hypotheses I previously told need to be adjusted:
RocksDB.Put()
may sometimes fail to store data for some unknown cause. It probably silently fails (unless we've missed error logs).RocksDB.Put()
may successfully store data, but since then data could be broken for some unknown cause.RocksDB.Get()
may sometimes fail to load data even when they obviously exist, for some unknown cause.[2022-03-10 18:10:50.511] [info] [18:10:50 ERR] An unexpected exception occurred on GetBlockDigest: Buffer cannot be null. (Parameter 'buffer') System.ArgumentNullException: Buffer cannot be null. (Parameter 'buffer') at System.IO.MemoryStream..ctor(Byte[] buffer, Boolean writable) at Bencodex.Codec.Decode(Byte[] bytes) at Libplanet.Store.BlockDigest.Deserialize(Byte[] bytes) in C:\Users\circleci\project\NineChronicles.Headless\Lib9c\.Libplanet\Libplanet\Store\BlockDigest.cs:line 143 at Libplanet.RocksDBStore.RocksDBStore.GetBlockDigest(BlockHash blockHash) in C:\Users\circleci\project\NineChronicles.Headless\Lib9c\.Libplanet\Libplanet.RocksDBStore\RocksDBStore.cs:line 646
This kind of logs can be recorded with expected normal scenarios and no data corruption: RocksDBStore.GetBlockDigest()
returns null
with logging such a log when it's just asked a non-existent block hash. In such cases, these logs are just garbage and even misleading.
Therefore, I'm going to prevent RocksDBStore.GetBlockDigest()
from recording such logs and let it silently returns null
for non-existent block hashes.
ArgumentNullException: Buffer cannot be null. (Parameter 'buffer')
This kind of exceptions must be no more thrown for normal cases (for instance, when it's just asked to fetch a non-existent block hash) since Libplanet 0.28.2 & 0.29.0. We need to check again if such exceptions still happen on the mainnet.
ArgumentNullException: Buffer cannot be null. (Parameter 'buffer')
This kind of exceptions must be no more thrown for normal cases (for instance, when it's just asked to fetch a non-existent block hash) since Libplanet 0.28.2 & 0.29.0. We need to check again if such exceptions still happen on the mainnet.
Since we've upgraded Libplanet to 0.29.0+, it apparently hadn't happened on the 9C mainnet for a week.
This issue has been automatically marked as stale because it has not had recent activity. Thank you for your contributions.
main (40).log