State DB overflow

[DmytroNazarenko]

Details:

• [ ] https://github.com/skalenetwork/internal-support/issues/575

[kladkogex]

Assigning to myself, since I am already working on statedb data issues

[DmytroNazarenko]

@kladkogex to investigate totalBytesUsed

[DmytroNazarenko]

Move to Release candidate, solution - https://github.com/skalenetwork/internal-support/issues/686

[DmytroNazarenko]

skaled: 3.17.0-beta.1

[oleksandrSydorenkoJ]

Actual for skalenetwork/schain:3.17.0-beta.5

Skale chain name: wan-tasty-kitalpha "contractStorageLimit": 3995998617,

/dev/mapper/schains-wan--tasty--kitalpha 21G 21G 8.0K 100% /mnt/schains-wan-tasty-kitalpha

"contractStorageUsed":3314231392","state.db_disk_usage":6627723459

Logs from skaled:

2023-07-14 15:43:46.067124   Block sealed #1403223 (#9e7d84d1…)
2023-07-14 15:43:46.067158   Block stats:BN:1403222:BTS:1689349416:TXS:3:HDRS:11:LOGS:40:SENGS:1:TXRS:2839:BLCKS:3:ACCS:150:BQS:1:BDS:3127:TSS:0:UTX:0:VTX:0:CMM:5205224
2023-07-14 15:43:46.067257   Block 1403223 DB usage is 1864. Piece DB usage is 62198396 bytes
2023-07-14 15:43:46.106580   Error writing to blocks_and_extras database: /home/ubuntu/actions-runner1/_work/skaled/skaled/libdevcore/LevelDB.cpp(60): Throw in function void dev::db::{anonymous}::checkStatus(const leveldb::Status&, const boost::filesystem::path&)
Dynamic exception type: boost::exception_detail::clone_impl<dev::db::DatabaseError>
[dev::db::tag_dbStatusCode*] = [ type: dev::db::DatabaseStatus, size: 4, dump: 05 00 00 00 ]
[dev::db::tag_dbStatusString*] = IO error: /data_dir/28e07f34/blocks_and_extras/1.db/000401.log: No space left on device

2023-07-14 15:43:46.106627   Put62657374=>9e7d84d1131cae18998af9405b2af070f96690c240b43aebce918d53cde089c5
2023-07-14 15:43:46.106637   Fail writing to blocks_and_extras database. Bombing out.
2023-07-14 15:43:46.107393   Exception in: insertBlockAndExtras /home/ubuntu/actions-runner1/_work/skaled/skaled/libethereum/BlockChain.cpp 967
[2023-07-14 15:43:46.700] [16:main] [error] 1403222:!Exception: nlohmann:Read dap proof resp:Could not read header len from:34.217.246.35
[2023-07-14 15:43:46.700] [16:main] [error] 1403222: !Caused by: IO:The peer shut down the socket, bytes to read:8
Dispatch: All threads stopped
Dispatch: All dispatch queues removed
terminate called after throwing an instance of 'boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::log::v2s_mt_posix::system_error> >'
pure virtual method called
terminate called recursively

[PolinaKiporenko]

https://github.com/skalenetwork/skaled/issues/1601

[DmytroNazarenko]

Moved to #1601 , no fixes in 3.17.0 versions anymore