[Living Ticket] Scalability related efforts

[okdas]

Objective

Ensure that Shannon scales both on-chain & off-chain.

Origin Document

This issue is intended to be a living document to keep track of all related efforts.

Identified issues and points of investigation

• [x] RelayMiner RAM usage: #551
• [ ] AppGateServer (and Gateway) CPU #Infrastructure
  • [ ] Relays are not going through, and CPU utilization is up to the limit. Need to get pprof snapshots & evaluate.
• [ ] Validator scalability
  • [ ] Ensure the validator's resource usage (CPU, RAM, etc...) is reasonable when the number of claims & proofs grows VERY LARGE. Note: This is why we need distribution of claims & proof
  • [ ] Probabilistic Proofs: make sure these parameters are adjusted properly for both: #Algorithmic
    • [ ] Validator scalability #Infrastructure
    • [ ] Discourding adversarial actors #Algorithmic
• [ ] Relay Mining #Algorithmic #Infrastructure
  • [ ] Ensure gateway consumption is reasonable #Infrastructure
  • [ ] Ensure relayminer footprint is as small as possible #Infrastructure

Things to investigate:

• Replacing the KV store in the SMT (e.g. BadgerDB or other)
• Keeping things in memory or flushing to disk
• Changing parameters

---

Creator: @okdas Co-Owners: @red-0ne @bryanchriswhite @Olshansk

[Olshansk]

@okdas Made some changes, updates & improements to this ticket. PTAL