Report "unable to open database file" error during making new snapshots.

[peilun-conflux]

This was triggered while setting a large additional_maintained_snapshot_count.

thread 'Consensus Execution Worker' panicked at 'No db error: Error(SqliteError(Error { code: Some(14), message: Some("unable to open database file") }), State { next_error: None, backtrace: InternalBacktrace { backtrace: Some(   0: error_chain::backtrace::imp::InternalBacktrace::new
             at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/error-chain-0.12.4/src/backtrace.rs:56:48
   1: <error_chain::State as core::default::Default>::default
             at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/error-chain-0.12.4/src/lib.rs:687:24
   2: cfx_storage::impls::errors::Error::from_kind
             at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/error-chain-0.12.4/src/error_chain.rs:269:21
      <cfx_storage::impls::errors::Error as core::convert::From<sqlite::Error>>::from
             at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/error-chain-0.12.4/src/error_chain.rs:361:21
      <T as core::convert::Into<U>>::into
             at /rustc/5c1f21c3b82297671ad3ae1e8c942d2ca92e84f2/src/libcore/convert/mod.rs:559:9
      cfx_storage::impls::storage_db::sqlite::ScopedStatement::execute
             at /home/ubuntu/conflux-rust/core/storage/src/impls/storage_db/sqlite.rs:471:17
      cfx_storage::impls::storage_db::sqlite::SqliteConnection::execute_locked
             at /home/ubuntu/conflux-rust/core/storage/src/impls/storage_db/sqlite.rs:191:22
   3: cfx_storage::impls::storage_db::sqlite::SqliteConnection::execute
             at /home/ubuntu/conflux-rust/core/storage/src/impls/storage_db/sqlite.rs:201:9
      cfx_storage::impls::storage_db::kvdb_sqlite::<impl cfx_storage::storage_db::key_value_db::SingleWriterImplByFamily<cfx_storage::impls::storage_db::kvdb_sqlite::KvdbSqlite<ValueType>> for T>::put_impl
             at /home/ubuntu/conflux-rust/core/storage/src/impls/storage_db/kvdb_sqlite.rs:766:17
   4: cfx_storage::impls::storage_db::kvdb_sqlite_sharded::<impl cfx_storage::storage_db::key_value_db::SingleWriterImplByFamily<cfx_storage::impls::storage_db::kvdb_sqlite_sharded::KvdbSqliteSharded<ValueType>> for T>::put_impl
             at /home/ubuntu/conflux-rust/core/storage/src/impls/storage_db/kvdb_sqlite_sharded.rs:422:34
      <cfx_storage::impls::storage_db::snapshot_db_sqlite::DeltaMptMergeDumperSqlite as cfx_storage::impls::merkle_patricia_trie::KVInserter<(alloc::vec::Vec<u8>,alloc::boxed::Box<[u8]>)>>::push
             at /home/ubuntu/conflux-rust/core/storage/src/impls/storage_db/snapshot_db_sqlite.rs:588:13
   5: cfx_storage::impls::delta_mpt::cow_node_ref::CowNodeRef::iterate_internal
             at /home/ubuntu/conflux-rust/core/storage/src/impls/delta_mpt/cow_node_ref.rs:596:13
   6: cfx_storage::impls::delta_mpt::cow_node_ref::CowNodeRef::iterate_internal
             at /home/ubuntu/conflux-rust/core/storage/src/impls/delta_mpt/cow_node_ref.rs:624:13
   7: cfx_storage::impls::delta_mpt::cow_node_ref::CowNodeRef::iterate_internal
             at /home/ubuntu/conflux-rust/core/storage/src/impls/delta_mpt/cow_node_ref.rs:624:13
   8: cfx_storage::impls::delta_mpt::cow_node_ref::CowNodeRef::iterate_internal
             at /home/ubuntu/conflux-rust/core/storage/src/impls/delta_mpt/cow_node_ref.rs:624:13
   9: cfx_storage::impls::delta_mpt::delta_mpt_iterator::DeltaMptIterator::iterate
             at /home/ubuntu/conflux-rust/core/storage/src/impls/delta_mpt/delta_mpt_iterator.rs:31:17
  10: cfx_storage::impls::storage_db::snapshot_db_sqlite::SnapshotDbSqlite::dump_delta_mpt
             at /home/ubuntu/conflux-rust/core/storage/src/impls/storage_db/snapshot_db_sqlite.rs:477:9
  11: <cfx_storage::impls::storage_db::snapshot_db_manager_sqlite::SnapshotDbManagerSqlite as cfx_storage::storage_db::snapshot_db_manager::SnapshotDbManagerTrait>::new_snapshot_by_merging
             at /home/ubuntu/conflux-rust/core/storage/src/impls/storage_db/snapshot_db_manager_sqlite.rs:430:25
  12: cfx_storage::impls::storage_manager::storage_manager::StorageManager::check_make_register_snapshot_background::{{closure}}::{{closure}}
             at /home/ubuntu/conflux-rust/core/storage/src/impls/storage_manager/storage_manager.rs:624:29
      cfx_storage::impls::storage_manager::storage_manager::StorageManager::check_make_register_snapshot_background::{{closure}}
             at /home/ubuntu/conflux-rust/core/storage/src/impls/storage_manager/storage_manager.rs:718:35
      std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/5c1f21c3b82297671ad3ae1e8c942d2ca92e84f2/src/libstd/sys_common/backtrace.rs:130:5
  13: std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}}
             at /rustc/5c1f21c3b82297671ad3ae1e8c942d2ca92e84f2/src/libstd/thread/mod.rs:475:17
      <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/5c1f21c3b82297671ad3ae1e8c942d2ca92e84f2/src/libstd/panic.rs:318:9
      std::panicking::try::do_call
             at /rustc/5c1f21c3b82297671ad3ae1e8c942d2ca92e84f2/src/libstd/panicking.rs:297:40
      std::panicking::try
             at /rustc/5c1f21c3b82297671ad3ae1e8c942d2ca92e84f2/src/libstd/panicking.rs:274:15
      std::panic::catch_unwind
             at /rustc/5c1f21c3b82297671ad3ae1e8c942d2ca92e84f2/src/libstd/panic.rs:394:14
      std::thread::Builder::spawn_unchecked::{{closure}}
             at /rustc/5c1f21c3b82297671ad3ae1e8c942d2ca92e84f2/src/libstd/thread/mod.rs:474:30
      core::ops::function::FnOnce::call_once{{vtable.shim}}
             at /rustc/5c1f21c3b82297671ad3ae1e8c942d2ca92e84f2/src/libcore/ops/function.rs:232:5
  14: <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once
             at /rustc/5c1f21c3b82297671ad3ae1e8c942d2ca92e84f2/src/liballoc/boxed.rs:1076:9
      <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once
             at /rustc/5c1f21c3b82297671ad3ae1e8c942d2ca92e84f2/src/liballoc/boxed.rs:1076:9
      std::sys::unix::thread::Thread::new::thread_start

[peilun-conflux]

Now try to rerun with additional_maintained_snapshot_count set to verify it's related.

Another guess is that this is a random error because a newly created sqlite3 database may not be ready even create_and_open has finished.

[peilun-conflux]

Setting additional_maintained_snapshot_count does not necessary cause this issue.

[peilun-conflux]

Might be related to #1940.