Big refactoring

[damip]

Rationale

The program needs to be refactored for improved performance, features and maintainability.

Points of attention

Documentation and specification

• improve documentation
• write specs of the whole program

Client

• Encrypted wallet
  • https://github.com/massalabs/massa/issues/447
• Encrypted communication
  • https://github.com/massalabs/massa/issues/1999
• Add tests
  • https://github.com/massalabs/massa/issues/1347
  • https://github.com/massalabs/massa/issues/1134
• versioning

API

• Versioning
• Add tests
  • https://github.com/massalabs/massa/issues/1182

Tests and benchmarks

• improve testing framework
  • https://github.com/massalabs/massa/issues/1240
  • https://github.com/massalabs/massa/issues/1092
• run full node tests within a simulator
• add benchmarks/profiling in CI
  • https://github.com/massalabs/massa/issues/1114
• evaluate per-crate performance

Project setup

• split crates (export, worker, test_exports)
  • https://github.com/massalabs/massa/issues/1132
• discuss splitting the client in another project
  • https://github.com/massalabs/massa/issues/1952
• discuss switching critical crates to stable toolchain
  • https://github.com/massalabs/massa/issues/1100

Architecture

• ensure checklist https://rust-lang.github.io/api-guidelines/checklist.html
• remove clippy warnings
• split ledger away from consensus and have it interact only with API and execution
• have a shared object "storage" library to avoid block copies
• reduce MPSC usage in favor of RwLocks to allow multiple simultaneous reads
• reduce Async usage
• prioritize tasks
• crate versioning
• improve file splitting
  • https://github.com/massalabs/massa/issues/237
• improve genericity
  • https://github.com/massalabs/massa/issues/1199
  • https://github.com/massalabs/massa/issues/1133
  • https://github.com/massalabs/massa/issues/275
• better error handling
  • https://github.com/massalabs/massa/issues/1138
• discuss new modules and their interactions
  • https://github.com/massalabs/massa/discussions/2177

Config

• discuss a uniform config usage

Versioning

• ledger versioning
• SC versioning
• protocol versioning
• smooth updates

Addresses

• Discuss using prefixes for addresses, block hashes etc...
  • https://github.com/massalabs/massa/issues/1105
• Versioning (see how it's done in bitcoin/ETH)

Crypto

• discuss using Blake3 hash and Scnorr signature
  • https://github.com/massalabs/massa/discussions/1916
• Discuss multisig

Algos

• optimize existing algorithms
  • https://github.com/massalabs/massa/issues/1888
• multistake protection
  • https://github.com/massalabs/massa/discussions/2161
• better caching
• improve ledger storage: on-disk
• discuss storage costs
• correct PoS lookback: https://github.com/massalabs/massa/issues/1295
• with the unified ledger, PoS lookback could be shortened. Study this
• improve endorsement system by emitting them at half-period and PoS-selecting for the endorsement target and not the endorsement inclusion slot
• prevent addresses without funds from flooding pool with high-reward operations by having Storage know about their sequential balance to ensure they have the funds to pay for the ops

Bootstrap

• improve bootstrap (streaming)

Networking

• whitelisting
  • https://github.com/massalabs/massa/issues/1946
• better peer management (peer testing, random listing etc...)
  • https://github.com/massalabs/massa/issues/1220
• hashes in objects, better caching
  • https://github.com/massalabs/massa/issues/1121
• keep serialized objects so we don't have to re-serialize them when sending (=> more RAM less CPU)
• better (de-)serialization with an external tool (ex: nom)
  • https://github.com/massalabs/massa/issues/1200
• send peer list when refusing incoming connection
  • https://github.com/massalabs/massa/issues/2222
• protocol versioning
• better gossip
  • https://github.com/massalabs/massa/issues/1088
• have user-defined categories of peers (bootstrap, whitelist etc...)

Monitoring

• discuss adding monitoring capabilities
  • https://github.com/massalabs/massa/issues/1080
  • https://github.com/massalabs/massa/discussions/1938
  • https://github.com/massalabs/massa/issues/1312

[yvan-sraka]

Improvement in CI (trying to test client end-to-end):

• https://github.com/massalabs/massa/issues/1080
• https://github.com/massalabs/massa/issues/1081
• https://github.com/massalabs/massa/issues/1092
• https://github.com/massalabs/massa/issues/1134
• https://github.com/massalabs/massa/issues/1269
• https://github.com/massalabs/massa/issues/1315
• https://github.com/massalabs/massa/issues/1347
• https://github.com/massalabs/massa/issues/2027
• https://github.com/massalabs/massa/issues/2170

[yvan-sraka]

Fix issues in logging:

• https://github.com/massalabs/massa/issues/1240
• https://github.com/massalabs/massa/issues/1316
• https://github.com/massalabs/massa/issues/2111
• https://github.com/massalabs/massa/issues/2163
• https://github.com/massalabs/massa/commit/1dc0c4ad608f8be9dbeadc492a429b5ef8879ec4#diff-4ca28ecd1b72c3cb9d82b826603333c9da457413bd37dd536765a4f735906083 we should remove serde_json::json!({}) line :)

[qdrn]

Improve transaction pool:

• https://github.com/massalabs/massa/issues/2215

[AureliaDolo]

@damip maybe we could go through what's missing here