stargeizer
commented
4 years ago in context for Harmony to grow and get external contributors, we must provide a professional interface for developers to hook into
However, we would like to support a standard that would help partners & developers onboard faster and we see Rosetta's node API spec as the best solution for Harmony to achieve this.
we need to make our codebase more consumable for external developers
Moreover, this simplifies the implementation of staking functionalities as we do not have to worry about violating the expected returns of the node API.
Daniel-VDM
Node API Overhaul
Original Draft
Overview
The API layer of a Harmony node is its main interface with the outside world. Therefore, it must be consistent, robust, and well-documented in order to provide easy integration with developers who wish to build on Harmony.
Current issues with v1 & v2 of Harmony's JSON-RPC based API
Context
For Harmony to grow and get external contributors, we must provide a professional interface for developers to hook into. Moreover, an easily consumable Node API can attract developers to take a deeper dive into Harmony's codebase and contribute to the core protocol.
Since Harmony has completed open staking, all core functionalities for an API layer have been defined. Therefore, now is an opportune time to correct the issues stated above as we are in the design phase of other major core features.
It is important to note that Harmony has numerous tools and partners that use the current JSON-RPC based API. Therefore, we must be careful to NOT break/change the existing RPC output format during the overhaul.
Goals and Non-Goals
What this overhaul aims to do:
./internal/hmyapipackages to reduce code duplication & dead code.What this overhaul aims NOT to do:
Existing Solution
Harmony uses JSON-RPC 2.0 for its node API (carried over from ethereum).
At a high level, the Harmony node starts an Http server (attached to some given port) for the RPCs and gives it a set of methods (referenced as
APIin the code) that are offered by the RPC interface.Harmony uses the geth RPC package found here to handle the RPC's Http server. Said server gets started by the node on boot (here) by ultimately calling the geth function here. The
APIs consumed by said function call is defined (here) specifically for the Harmony Node.The 'common' backend interface, which connects the RPC layer with the node, can be found in
./internal/hmyapi/backend.go. The implementation of the interface that is used by all Harmony nodes can be found at./hmy/api_backend.goas theAPIBackendobject, which just wraps around theHarmonyobject with a cache. Note that theHarmonyobject is not used in a meaningful way other than just a container of major data structures.Harmony currently supports 2 RPC method namespaces,
hmy&hmyv2. The supposed 'only' difference between the two is thathmyv2returns values in decimal, whereashmyreturns values in hex. However, there are numerous occasions wherehmyv2is missing a method thathmyhas and visa-vera. Moreover, there are various methods that have slightly different signatures, causing further confusion. The primary cause of these issues is that there are 2 distinct copies of each namespace's service and respective methods, which can be found here and here. Because of this code duplication, one has to define a new backend interface in 3 different places and implement it in 2 different places to properly add a new RPC. Couple this with the fact that we don't have any check for consistency between the namespaces, and it becomes easy to see why we have the issues mentioned above.Lastly, Harmony's RPCs often returns the JSON text marshalled version of various data structures. However, we have inconsistent key naming conventions for said structures.
Proposed Solution
At a high level, we will first fix the existing JSON-RPC methods (focusing on
./internal/hmyapi), then we will implement the Rosetta node API.The overhaul will have 7 stages: 1) Lock the current RPC behavior with tests. 2) Refactor JSON-RPC helper functions & method definitions. 3) Implement a new node API (Rosetta). 4) Docker node deployment. 5) Integrate Rosetta's testing tool into our CI. 6) Extend our implementation of Rosetta's node API for staking functions. 7) Setup API documentation generation.
1. Lock the current RPC behavior with tests
Similar to our existing PR testing script, we will orchestrate a series of transactions on a localnet and do RPC calls to assert the format and values of the output. Quite some work has been done here using pytest, and we shall follow/extend this format for the tests. We can also add more go-tests as we see fit.
Moreover, we will script the deployment of a localnet and execution of the test script within a (public) dockerfile & image.
A benefit of doing this first is that we are able to ensure that our existing tools won't break with any subsequent changes. Moreover, we should be able to replace the existing PR testing Jenkins job for a simple
docker runcommand that we can integrate with our TravisCI.2. Refactor JSON-RPC helper functions & method definitions
The goal of the refactor is to remove duplicate code, dead code, and needless abstraction.
2.1 Remove the
BackendinterfaceSince all RPC methods use a
Backend, we can start there to reduce the needless abstraction. Note that we only have 1 implementation of thisBackendinterface with no plans (or need) to create more implementations. Therefore, we should remove the interface and use theHarmonyobject as the interface between the Node and API/RPC layer (for reasons described in 2.2).2.2 Fuse
APIBackendandHarmonytogetherAs stated in the section above,
APIBackendjust wraps around theHarmonyobject. Therefore, we can just define all interfaces between the Node and API/RPC layer within theHarmonyobject.2.3 Generalize code for the 2 RPC namespaces
We have heavy code duplication at the method definition for each RPC namespace here (
apiv1&apiv2). We should undo the code duplication introduced earlier this year and just carry a flag within each API struct (for examplePublicBlockChainAPI) to indicate the RPC namespace. Then, the methods can adjust the output format (hex or decimal) according to this flag.2.4 Remove dead code
There are numerous things leftover from ethereum that Harmony has not implemented (such as
addrlock) that can be removed. We should take this time to check & remove this code.2.5 Rename JSON text marshall key namesThis task will get done at a later time (TBD). Reason in the footnote for 7/24.As mentioned in the issues section, we have inconsistent key naming conventions for the responses of our JSON-RPCs. This is partially due to our inconsistent naming conventions for the JSON text marshall of our data structures. As these text marshalls will be used in the implementation of the additional node API, we should make the naming convention consistent with 'snake_case'. However, we CANNOT break the output format of our current RPCs. Therefore, we will add a 'reformatter' of sorts (when necessary) to keep the current output format.
3. Implement a new node API (Rosetta).
The goal of this overhaul is to provide a consistent, robust, and well-documented API. Rosetta's node API provides a low-enough level API specified using OpenAPI 3.0 and works well for Harmony's needs. Moreover, Rosetta provides a SDK along with tooling and tests in their rosetta-cli that will speed up implementation and ensure correctness. We see Rosetta as a standard that other chains will follow, therefore it would be advantageous for us to use this API specification as it will be easier to integrate with future partners and developers.
Note that the implementation of this node API can hook into the Node's 'backend' interface cleanly with the refactor from stage 2. This will greatly speed up the implementation time of this stage.
4. Docker node deployment.
To satisfy the Rosetta node API expectation, we must provide a dockerfile for running a Harmony node. Expectation is outlined here. Note that the dockerfile must build the node and run it (with certain storage restrictions). Fortunately, we have some work done in our dockerfile here that we leverage along with the dockerfile created in stage 1.
5. Integrate Rosetta's testing tool into our CI.
Rosetta provides tests for the node API via the Rosetta CLI (as seen here). We will integrate said tests into our CI test, either as a stage in a Jenkins job, or as an additional step in the new test suite (stage 1).
6.
Extend our implementation of Rosetta's node API for staking functionsThis is no longer needed with the/callendpoint added in the Rosetta specification.As mentioned in stage 3, Rosetta's node API does not provide a spec for staking related functions. We can integrate staking transactions by defining these transaction types as a supported
operation. However, fetching staking related network information will be done by extending our implementation of the spec via an additional/stakingendpoint. This way, we don't add fields or overload existing fields in the Rosetta spec, thus making our implementation more upgrade friendly for future versions of Rosetta.Note that the Rosetta spec was generated using Open API 3.0, therefore we can extend this spec by defining the
stakingendpoint APIs (example here) and generating the necessary code to implement it (example here). Our intention is to stay as close as we can to the design direction of Rosetta yet remain independent enough for future upgrades of Rosetta's node API.With a new set of staking endpoints in this API extension, we must create our own set of tests to integrate into our CI. We shall add a test suite to the pytest from stage 1 since it will more 'future proof' than duplicating & extending the checks done with the Rosetta CLI.
7. Setup API documentation generation.
Since the Rosetta node API is specified using Open API 3.0, we can use existing tools like Swagger UI and Swagger Inspector to generate documentation (with sample requests & responses) from the API specification. Example here. The sample calls can be done on a 'documentation' node to provide consistent responses.
Milestones
Each milestone will have a pull request associated with it.
Start Date: July 9th, 2020
2.5)/network&/blocknode endpoint services/account&/mempoolnode endpoint services/constructionnode endpoint servicesAlternative Solutions
JSON-RPC v3 instead of Rosetta node API
This solution would be to just create another JSON-RPC version that fixes all of the aforementioned issues (after the refactor in stage 2). This would be a simpler approach and would be faster to implement. However, we would like to support a standard that would help partners & developers onboard faster and we see Rosetta's node API spec as the best solution for Harmony to achieve this.
That said, the design of this overhaul should allow for easy implementation of a v3 JSON-RPC in the future if we feel like we need it.
Abandon v2 JSON-RPC
The solution here would be to drop the v2 RPC to simplify stage 2 of the implementation. This would speed up the implementation time, however, we have to consider key partners and developers that currently use v2.
As we have had the v2 RPC for some time now, we should continue to support it. Furthermore, it is likely that the time saved in the implementation now would be repaid later in extra time spent on coordination, support, and patching our existing tools.
Skip 'backend' refactor and go stright to Rosetta node API implementation
While this is possible to do, it will likely introduce further code duplication, which may lead to inconsistencies between the JSON-RPCs and the node API. Also, we need to make our codebase more consumable for external developers, and it is likely that the RPC/API layer will be their introduction to it.
Furthermore, we still need to support the JSON-RPCs for the foreseeable future and will likely add RPCs or functionality to the new node API in the future. However, it is currently very cumbersome to add such things and plenty of dev time is wasted in the process. Therefore, it is worth it to take the time now to refactor the backend.
Overload or add return elements to the new node API replies instead of extra
/stakingendpointWe want the extensions that we make to Rosetta's node API to be as independent as possible so that it would be easy to upgrade our implementation of the API in the event that the node API spec receives an update. Moreover, this simplifies the implementation of staking functionalities as we do not have to worry about violating the expected returns of the node API.
External Impact
There is a possibility that partners may suffer some downtime when refactoring the RPC's 'backend'. However, extra time has been allocated to define thorough tests that would minimize this possibility. Moreover, we shall do a testnet deployment first and encourage partners (with the help of the bis team) to test their functionality on our testnet. We will also ask P-OPS to test.
Current Scoping & Timeline
Current Stage: 3
7/9: [Stage 1] Localnet deployment needed to be redone to include external nodes as part of the testing process. Changes are done here. Created a new repo that will contain the tests for stage 1. 7/10: [Stage 1] Finished testing pipeline here. Localnet is built, deployed, and tested in a docker image using a shared volume with the main repo. This allows one to test whatever changes they made locally, allowing for fast iteration in the later stages. 7/11-7/18: [Stage 1] Test cases took a lot of time to write as the 2 different RPC versions had differences that required careful tests. Moreover, some time was spend optimizing the tests to be ran as fast as possible since it will be used a lot in the future sections & by other devs as we implement other initiatives. PR of test merge can be found here. 7/24: [Stage 2] There is a lot of technical debt and duplicate code that needs to be refactored/fixed, even within each RPC version. Therefore, as I join the 2 RPC versions back together (thus removing a large chunk of the duplicate code), I will also fix the in-package code duplication (or at least contain it to be fixed at a later time). Moreover, I will also separate services for clarity (now everything is dumped into 2 files, transactionpool.go & blockchain.go). This comes at a cost of time, but I find it to be much more valuable than correcting the naming conventions of data structures at this time. Also, the data structure correction can be done separately after stage 2 by another engineer if available. 8/11: [Stage 3] Due to the 10 day #DeFi hackathon at Harmony, momentum & context has been lost. This has put a substantial delay in the planned timeline. Moreover, familial commitments towards the beginning of September will slow down dev time (a bit). Lastly, the
constructionspec has had additions to it since the initial planning pass. The additions will require a little bit more time than anticipated to implement the endpoint. That said, Rosetta's forums are more active nowadays, which should hopefully help us get over hurdles faster should we encounter any. 8/13: [Stage 3] During the PR process of #3297, there was some regression on the CPU usage (evident in the localnet test). This slowed down progress as time was spent dealing with the issue. Timelines have been updated to reflect this. 8/25: [Stage 3] While the PR (#3312) for stage 3.2 got done in time, there were some edge case bugs that were found when testing with the mainnet DB. Therefore, the review process took longer than normal. Delay for this stage was only 1 day, hopefully this can be made up at the later stages. 8/27: [Stage 3] Some decimal precision errors occurred during further testing of staking operations from the/block/transactionendpoint. Moreover, some block rewards were not visible. The following PR #3321 fixes these issues. All effort was put towards fixing said bugs, thus no progress has been made on the account, mempool, or construction endpoints. Some delay is expected at this point -- but should be minimal. 9/7: [Stage 3] Due to the bridge hackathon (where some efforts were made to research multichain) and some personal time off, progress for rosetta was halted. However, some lessons learned from the multichain research (particularly involving signing txs and Docker) will reduce the time spent on the Construction API and the Docker requirements. The target date has been updated (generously) to reflect the delay.