🔷 [ProjectTracking] ReadRPC rightsizing

[gmilescu]

Goals

Background

ReadRPC is a redesign of the architecture of an RPC node that offers a reduces operational cost while increasing the performance/throughput of the service. It moves away from the regular model of a NEAR node, and it relocated RocksDB as an external DB layer. The goal of ReadRPC is to offer an alternative to running a fleet of RPC nodes by replacing them with a single infrastructure.

As ReadRPC was delivered and placed in production, it became clear that it can replace RPC nodes only for large infrastructure operators, that can afford a ScyllaDB instance. For smaller operators or for app developers that need a slim RPC instance for their own app, ReadRPC is too heavy.

There are two dimensions where ReadRPC can be mare more lightweight: using a thinner DB type and reducing the amount of data it has to manage.

Why should NEAR One work on this

ReadRPC right sizing will make it easier and more appealing to run RPC nodes, and this contributes to having a minimum viable and scalable ecosystem.

What needs to be accomplished

There are two major deliverables included in Rightsizing, and both are required for the project to make sense:

• Database agnosticism, allowing ScyllaDB to be replaces with general purpose DB that is easier to provision and maintain, and offers lower operational costs (e.g. PostgreSQL)
• Data filtering, selecting only a subset of the blockchain state to be stored and managed by the respective ReadRPC instance.

Main use case

The main usecase is to allow RPC operators to adjust the ReadRPC infrastructure, reducing the operational complexity and cost of ReadRPC.

Links to external documentations and discussions

Additional resources will be added here when they become available.

Estimated effort

Engineers assigned: @khorolets and Yurii.

Initial effort estimate is about 4-6 PM (person months). Currently remaining effort is presented in the latest comment of this issue.

Assumptions

There are no specific assumptions that this project is making.

Pre-requisites

N/A

Out of scope

N/A

Task list:

### Tasks
- [ ] https://github.com/near/nearcore/issues/10045
- [ ] https://github.com/near/nearcore/issues/10046

[gmilescu]

2023-11-27

• Rightsizing implementation is still in flight. Coding for Postgresql suport is completed, testing not done yet.

[khorolets]

2023-12-18

• Rightsizing implementation is still in progress. We design a convenient configuration scheme
• Meanwhile, we have been distracted by a couple of things:
  • ✅ Increase the size of the state that can be retrieved with view_state via ReadRPC
  • ✅ Implement the validators method (current epoch or any historical one)
  • ⚠️ Implement a brand new method view_state_paginated to retrieve huge states in paginated way

[khorolets]

2024-01-16

• We are wrapping up the configuration scheme to enable the rightsizing feature relatively soon. Follow the PR here if you're curious
• At the moment, we are a bit distracted focusing on the last fixes to be sure we can go out from the beta
  • It means we work on the analysis of the % of mismatches with the NEAR RPC
  • We apply fixes to the mismatches we can/should fix

[gmilescu]

2024-01-26

• Status: DB agnosticism work is completed, implementing data filtering is stil in progress.
• Remaining tasks: complete work on data filtering
• Remaining effort: The remaining efort is about 1-2 engineering weeks to be dev-ready ready.
• Blockers: None
• Release date: We aim to release rightsizing in late February/early March.
• Note: work on this project was deprioritised in the week of February 22nd to fix issues encountered with ReadRPC in production.

[gmilescu]

2024-02-08

Over the past two weeks we did not make too much progress with this project. We prioritised instead the preparation work for Stake Wars IV (e.g. Explorer, web service for account creation), and we expect that work to continue for about one more week.

The remaining tasks are around finishing the implementation for data filtering (1 engineering week of effort) and then test the whole setup.

We continue to aim for releasing rightsizing in late February/early March.

[khorolets]

2024-02-19:

Weekly Update: Rightsizing Feature in ReadRPC Project

Completed Work:

• Completed the implementation of the Rightsizing feature for the ReadRPC project.
• Added an example docker-compose.yml file to facilitate easy setup. This demonstrates the feature in action within the ReadRPC project.
  • Guide in README section
  • Example docker-compose.yml

Feedback and Community Engagement:

• Requested feedback from other managers and devrel teams, asking our partners to trial the Rightsizing feature.
• Awaiting community feedback to inform and define our next steps.

Next Steps:

• Transitioning focus to other tasks within the project while monitoring feedback on the Rightsizing feature. Processing the feedback input and taking any action items is a matter of separate task(s).

This task is done ✅