Thread 'tokio-runtime-worker' panicked at 'Critical database error with paritydb

This bug appeared during execution of a Kusama Validator node. I must add that the node stopped completely and it was impossible to restart it without a database purge. The behavior is similar to: https://github.com/paritytech/polkadot/issues/2529

There were no other errors or warnings right before this one, just regular operation. The SSD has plenty of space still there were no memory errors either. There were no special pruning flags just the --validator flag which makes it an archive node It was not started from a snapshot. The database was paritydb with the flag --db paritydb

#6470852 (0xfb0d_c040), _ 1.5MiB/s _ 1.9MiB/s                                  
2021-03-05 07:27:18  __ Imported #6470855 (0x8508_ff49)                         
2021-03-05 07:27:20  __ Idle (48 peers), best: #6470855 (0x8508_ff49), finalized
 #6470852 (0xfb0d_c040), _ 1.1MiB/s _ 989.5kiB/s                                2021-03-05 07:27:24  __ Imported #6470856 (0x65ea_81b3)                         
2021-03-05 07:27:25  __ Idle (48 peers), best: #6470856 (0x65ea_81b3), finalized
 #6470853 (0x8f77_5f34), _ 765.4kiB/s _ 900.2kiB/s                              
2021-03-05 07:27:30  __ Idle (48 peers), best: #6470856 (0x65ea_81b3), finalized
 #6470854 (0xd153_6e9e), _ 993.2kiB/s _ 1.1MiB/s                                
2021-03-05 07:27:30  __ Imported #6470857 (0x1467_d249)                         
2021-03-05 07:27:32  Background worker error: IO Error: failed to fill whole buf
fer     

====================                                                           
                                                                                Version: 0.8.29-2494dec2-x86_64-linux-gnu                                       

   0: sp_panic_handler::set::{{closure}}                                        
   1: std::panicking::rust_panic_with_hook                                      
             at rustc/cb75ad5db02783e8b0222fee363c5f63f7e2cf5b/library/std/src/p
anicking.rs:595:17                                                              
   2: std::panicking::begin_panic_handler::{{closure}}                          
             at rustc/cb75ad5db02783e8b0222fee363c5f63f7e2cf5b/library/std/src/p
anicking.rs:497:13                                                              
   3: std::sys_common::backtrace::__rust_end_short_backtrace                                 at rustc/cb75ad5db02783e8b0222fee363c5f63f7e2cf5b/library/std/src/s
ys_common/backtrace.rs:141:18                                                   
   4: rust_begin_unwind                                                         
             at rustc/cb75ad5db02783e8b0222fee363c5f63f7e2cf5b/library/std/src/p
anicking.rs:493:5                                                               
   5: std::panicking::begin_panic_fmt                                           
             at rustc/cb75ad5db02783e8b0222fee363c5f63f7e2cf5b/library/std/src/p
anicking.rs:435:5         
   6: <sc_client_db::parity_db::DbAdapter as sp_database::Database<H>>::commit  
   7: sc_client_db::Backend<Block>::try_commit_operation                       
   8: <sc_client_db::Backend<Block> as sc_client_api::backend::Backend<Block>>::commit_operation                                                                
   9: <sc_service::client::client::Client<B,E,Block,RA> as sc_client_api::backen
d::AuxStore>::insert_aux                                                        
  10: <sc_finality_grandpa::environment::Environment<B,Block,C,N,SC,VR> as final
ity_grandpa::voter::Environment<<Block as sp_runtime::traits::Block>::Hash,<<Blo
ck as sp_runtime::traits::Block>::Header as sp_runtime::traits::Header>::Number>
>::prevoted                                                                     
  11: finality_grandpa::voter::voting_round::VotingRound<H,N,E>::poll           
  12: <finality_grandpa::voter::Voter<H,N,E,GlobalIn,GlobalOut> as core::future:
:future::Future>::poll                                                            13: <sc_finality_grandpa::VoterWork<B,Block,C,N,SC,VR> as core::future::future
::Future>::poll                                                                 
  14: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Fu
ture>::poll                                                                     
  15: <futures_util::future::select::Select<A,B> as core::future::future::Future
>::poll                                                                         
  16: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Fu
ture>::poll     
  17: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Fu
ture>::poll                                                                    
  18: <sc_service::task_manager::prometheus_future::PrometheusFuture<T> as core::future::future::Future>::poll                                                  
  19: <futures_util::future::select::Select<A,B> as core::future::future::Future
>::poll                                                                         
  20: <core::future::from_generator::GenFuture<T> as core::future::future::Futur
e>::poll                                                                        
  21: <tracing_futures::Instrumented<T> as core::future::future::Future>::poll  
  22: std::thread::local::LocalKey<T>::with                                     
  23: futures_executor::local_pool::block_on                                    
  24: tokio::runtime::task::core::Core<T,S>::poll                               
  25: <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once                                                                           
  26: tokio::runtime::task::harness::Harness<T,S>::poll                         
  27: tokio::runtime::blocking::pool::Inner::run                                
  28: tokio::runtime::context::enter                                            
  29: std::sys_common::backtrace::__rust_begin_short_backtrace                  
  30: core::ops::function::FnOnce::call_once{{vtable.shim}}                     
  31: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once  
             at rustc/cb75ad5db02783e8b0222fee363c5f63f7e2cf5b/library/alloc/sr$
/boxed.rs:1328:9                                                                
      <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once 
             at rustc/cb75ad5db02783e8b0222fee363c5f63f7e2cf5b/library/alloc/src/boxed.rs:1328:9                                                                
      std::sys::unix::thread::Thread::new::thread_start                         
             at rustc/cb75ad5db02783e8b0222fee363c5f63f7e2cf5b/library/std/src/s
ys/unix/thread.rs:71:17                                                         
  32: start_thread                                                              
  33: clone                                                                     

Thread 'tokio-runtime-worker' panicked at 'Critical database error: Background(I
o(Custom { kind: UnexpectedEof, error: "failed to fill whole buffer" }))', /usr/local/cargo/git/checkouts/substrate-7e08433d4c370a21/61b1eb8/client/db/src/parit
y_db.rs:30                                                                      

This is a bug. Please report it at:                                             

        https://github.com/paritytech/polkadot/issues/new

paritytech / polkadot

Thread 'tokio-runtime-worker' panicked at 'Critical database error with paritydb #2571