Archive node signature check fails during catchup

[oleksandrSydorenkoJ]

Versions Skaled (core nodes): schain:3.14.15-stable.2 Skaled (archive node): schain:3.16.0-beta.4-historic

Environment 20 active nodes on the network 1 archive node

Preconditions Skaled is running on stable versions from mainnet (release 2.0) The archive node started from the catchup from 1st block with a historic state on version 3.16.0-beta.4-historic No load on schain

Steps to Reproduce/Description

1. Check skaled logs during catchup

Expected behavior Skaled should catchup blocks without any errors

Actual result "BlockFinalizeDownloader:State check failed" error during catchup on archive nodes before updating on core nodes to 2.1 versions and without patchTimestamps. The error is not appearing for every block.

023-01-17 11:03:05.280752   Block sealed #254513 (#11501357…)
2023-01-17 11:03:05.280797   Block stats:BN:254512:BTS:1673953369:TXS:0:HDRS:11:LOGS:40:SENGS:1:TXRS:20:BLCKS:3:ACCS:101:BQS:1:BDS:7168:TSS:0:UTX:0:VTX:0:CMM:7606642
2023-01-17 11:03:05.280971   Block 254513 DB usage is 1113. Piece DB usage is 193785437 bytes
2023-01-17 11:03:05.281347   noteChanged: {chain}
2023-01-17 11:03:05.282036   Successfully imported 0 of 0 transactions

[2023-01-17 11:03:05.282] [5:main] [info] 254513:CATCHUP_PROCESSED_BLOCKS:COUNT: 1
[2023-01-17 11:03:20.286] [5:catchup] [error] 254513:Catchupc step 3: can not read missing blocks
[2023-01-17 11:03:20.286] [5:main] [error] 254513:!Exception: CatchupClientAgent:Catchupc step 3: can not read missing blocks
[2023-01-17 11:03:20.286] [5:main] [error] 254513: !Caused by: CatchupClientAgent:Could not process block list
[2023-01-17 11:03:20.286] [5:main] [error] 254513:  !Caused by: CommittedBlockList:Could not create block list. 
LIST_SIZE:1:SERIALIZED_BLOCK_SIZE:4446:OFFSET:0:COUNTER:0:INDEX:1:END_INDEX:4445
[2023-01-17 11:03:20.286] [5:main] [error] 254513:   !Caused by: CommittedBlock Block da signature did not verify in deserialization:deserialize
[2023-01-17 11:03:20.286] [5:main] [error] 254513:    !Caused by: CryptoManager:verifyDAProofThresholdSig
[2023-01-17 11:03:20.286] [5:main] [error] 254513:     !Caused by: CryptoManager:Check failed::sig->toString() == _hash.toHex() /home/ubuntu/actions-runner/_work/skaled/skaled/libconsensus/crypto/CryptoManager.cpp:1097:Mockup threshold signature did not verify
[2023-01-17 11:03:25.288] [5:main] [info] 254513:BLOCK_COMMITED: PRPSR:2:BID: 254514:ROOT:53263243987621296363108259470846612805007831948998081626596407652284049287754:HASH:9cbde166:BLOCK_TXS:0:DMSG:0:MPRPS:0:RPRPS:0:TXS:0:TXLS:1:MGS:0:INSTS:0:BPS:0:HDRS:2:SOCK:1:FDS:313:PRT:0:BTA:41129075:BSA:0:TPS:0:LWT:0:LRT:0:LWC:122100:LRC:81407:STAMP:1673953394.74
2023-01-17 11:03:25.296605   createBlock ID = #254514

2023-01-17 11:03:25.296931   Block sealed #254514 (#df770097…)
2023-01-17 11:03:25.296968   Block stats:BN:254513:BTS:1673953381:TXS:0:HDRS:11:LOGS:40:SENGS:1:TXRS:20:BLCKS:3:ACCS:101:BQS:1:BDS:7169:TSS:0:UTX:0:VTX:0:CMM:7607482
2023-01-17 11:03:25.297098   Block 254514 DB usage is 1113. Piece DB usage is 193786550 bytes
2023-01-17 11:03:25.297358   noteChanged: {chain}
2023-01-17 11:03:25.298051   Successfully imported 0 of 0 transactions

[2023-01-17 11:03:25.298] [5:main] [info] 254514:CATCHUP_PROCESSED_BLOCKS:COUNT: 1
[2023-01-17 11:03:30.301] [5:main] [info] 254514:BLOCK_COMMITED: PRPSR:13:BID: 254515:ROOT:53263243987621296363108259470846612805007831948998081626596407652284049287754:HASH:d4e20878:BLOCK_TXS:0:DMSG:0:MPRPS:0:RPRPS:0:TXS:0:TXLS:1:MGS:0:INSTS:0:BPS:0:HDRS:2:SOCK:1:FDS:313:PRT:0:BTA:41128065:BSA:0:TPS:0:LWT:0:LRT:0:LWC:122103:LRC:81409:STAMP:1673953405.639
2023-01-17 11:03:30.309656   createBlock ID = #254515

[DmytroNazarenko]

skaled:3.16.0-beta.6

[oleksandrSydorenkoJ]

Verified on Legacy Network schain: troubled-tall-rotanev version: skalenetwork/schain:3.16.0-beta.6-historic archive node: http://52.42.213.142:10323/

[2023-01-30 16:21:11.871] [5:catchup] [info] 324468:CATCHUP_GOT_BLOCKS:COUNT:15106:STARTBLOCK:324469:FROM_NODE:IP
[2023-01-30 16:21:11.875] [5:main] [info] 324468:BLOCK_COMMITED: PRPSR:10:BID: 324469:ROOT:83258290764754301191483122051928848540448684649984403195659516337265182385036:HASH:41caf5e1:BLOCK_TXS:0:DMSG:0:MPRPS:0:R
PRPS:0:TXS:1:TXLS:15106:MGS:0:INSTS:0:BPS:0:HDRS:2:SOCK:1:FDS:326:PRT:0:BTA:0:BSA:0:TPS:0:LWT:0:LRT:0:LWC:0:LRC:7:STAMP:1674771046.275
2023-01-30 16:21:11.897343   createBlock ID = #324469

2023-01-30 16:21:11.897766   Block sealed #324469 (#2ca99c38…)