Improve chain connector storage

[ishantiw]

What was the problem?

This PR resolves #9195

How was it solved?

• [x] Improve storage to retrieve by key/val and delete efficiently instead of managing full array
• [x] Reduce number of RPC calls during new block event
• [x] Improve cleanup
• [x] Add tests

How was it tested?

• Run interop setup
• Register sidechains on mainchain and mainchain on sidechains.
• Make cross chain transfers using the scripts as mentioned in the readme
• Stop mainchain-node-two for more than 1000 blocks (mainchain-node-one generates the blocks) and then start it and see if it syncs properly and starts sending valid CCUs without any errors.

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 79.24242% with 137 lines in your changes are missing coverage. Please review.

Project coverage is 83.28%. Comparing base (39c521f) to head (228050a).

:exclamation: Current head 228050a differs from pull request most recent head 053a4e0. Consider uploading reports for the commit 053a4e0 to get more accurate results

Additional details and impacted files

[](https://app.codecov.io/gh/LiskHQ/lisk-sdk/pull/9198?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=LiskHQ) ```diff @@ Coverage Diff @@ ## hotfix/6.0.2 #9198 +/- ## ================================================ - Coverage 83.52% 83.28% -0.24% ================================================ Files 606 608 +2 Lines 22890 23101 +211 Branches 3377 3379 +2 ================================================ + Hits 19118 19239 +121 - Misses 3772 3862 +90 ``` | [Files](https://app.codecov.io/gh/LiskHQ/lisk-sdk/pull/9198?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=LiskHQ) | Coverage Δ | | |---|---|---| | [...ain-connector-plugin/src/chain\_connector\_plugin.ts](https://app.codecov.io/gh/LiskHQ/lisk-sdk/pull/9198?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=LiskHQ#diff-ZnJhbWV3b3JrLXBsdWdpbnMvbGlzay1mcmFtZXdvcmstY2hhaW4tY29ubmVjdG9yLXBsdWdpbi9zcmMvY2hhaW5fY29ubmVjdG9yX3BsdWdpbi50cw==) | `91.42% <100.00%> (+12.41%)` | :arrow_up: | | [...-framework-chain-connector-plugin/src/constants.ts](https://app.codecov.io/gh/LiskHQ/lisk-sdk/pull/9198?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=LiskHQ#diff-ZnJhbWV3b3JrLXBsdWdpbnMvbGlzay1mcmFtZXdvcmstY2hhaW4tY29ubmVjdG9yLXBsdWdpbi9zcmMvY29uc3RhbnRzLnRz) | `100.00% <100.00%> (ø)` | | | [...sk-framework-chain-connector-plugin/src/schemas.ts](https://app.codecov.io/gh/LiskHQ/lisk-sdk/pull/9198?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=LiskHQ#diff-ZnJhbWV3b3JrLXBsdWdpbnMvbGlzay1mcmFtZXdvcmstY2hhaW4tY29ubmVjdG9yLXBsdWdpbi9zcmMvc2NoZW1hcy50cw==) | `100.00% <100.00%> (ø)` | | | [...lisk-framework-chain-connector-plugin/src/utils.ts](https://app.codecov.io/gh/LiskHQ/lisk-sdk/pull/9198?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=LiskHQ#diff-ZnJhbWV3b3JrLXBsdWdpbnMvbGlzay1mcmFtZXdvcmstY2hhaW4tY29ubmVjdG9yLXBsdWdpbi9zcmMvdXRpbHMudHM=) | `40.42% <100.00%> (-39.12%)` | :arrow_down: | | [...n-connector-plugin/src/active\_validators\_update.ts](https://app.codecov.io/gh/LiskHQ/lisk-sdk/pull/9198?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=LiskHQ#diff-ZnJhbWV3b3JrLXBsdWdpbnMvbGlzay1mcmFtZXdvcmstY2hhaW4tY29ubmVjdG9yLXBsdWdpbi9zcmMvYWN0aXZlX3ZhbGlkYXRvcnNfdXBkYXRlLnRz) | `93.75% <66.66%> (+13.39%)` | :arrow_up: | | [...ain-connector-plugin/src/certificate\_generation.ts](https://app.codecov.io/gh/LiskHQ/lisk-sdk/pull/9198?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=LiskHQ#diff-ZnJhbWV3b3JrLXBsdWdpbnMvbGlzay1mcmFtZXdvcmstY2hhaW4tY29ubmVjdG9yLXBsdWdpbi9zcmMvY2VydGlmaWNhdGVfZ2VuZXJhdGlvbi50cw==) | `95.65% <94.11%> (-2.31%)` | :arrow_down: | | [...amework-chain-connector-plugin/src/inbox\_update.ts](https://app.codecov.io/gh/LiskHQ/lisk-sdk/pull/9198?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=LiskHQ#diff-ZnJhbWV3b3JrLXBsdWdpbnMvbGlzay1mcmFtZXdvcmstY2hhaW4tY29ubmVjdG9yLXBsdWdpbi9zcmMvaW5ib3hfdXBkYXRlLnRz) | `95.65% <90.00%> (-4.35%)` | :arrow_down: | | [...k-framework-chain-connector-plugin/src/endpoint.ts](https://app.codecov.io/gh/LiskHQ/lisk-sdk/pull/9198?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=LiskHQ#diff-ZnJhbWV3b3JrLXBsdWdpbnMvbGlzay1mcmFtZXdvcmstY2hhaW4tY29ubmVjdG9yLXBsdWdpbi9zcmMvZW5kcG9pbnQudHM=) | `76.92% <90.47%> (-20.45%)` | :arrow_down: | | [...ramework-chain-connector-plugin/src/ccu\_handler.ts](https://app.codecov.io/gh/LiskHQ/lisk-sdk/pull/9198?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=LiskHQ#diff-ZnJhbWV3b3JrLXBsdWdpbnMvbGlzay1mcmFtZXdvcmstY2hhaW4tY29ubmVjdG9yLXBsdWdpbi9zcmMvY2N1X2hhbmRsZXIudHM=) | `95.00% <95.00%> (ø)` | | | [...-chain-connector-plugin/src/block\_event\_handler.ts](https://app.codecov.io/gh/LiskHQ/lisk-sdk/pull/9198?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=LiskHQ#diff-ZnJhbWV3b3JrLXBsdWdpbnMvbGlzay1mcmFtZXdvcmstY2hhaW4tY29ubmVjdG9yLXBsdWdpbi9zcmMvYmxvY2tfZXZlbnRfaGFuZGxlci50cw==) | `87.64% <87.64%> (ø)` | | | ... and [2 more](https://app.codecov.io/gh/LiskHQ/lisk-sdk/pull/9198?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=LiskHQ) | | ... and [1 file with indirect coverage changes](https://app.codecov.io/gh/LiskHQ/lisk-sdk/pull/9198/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=LiskHQ)