search

FuelLabs / fuel-core

Rust full node implementation of the Fuel v2 protocol.
Other
58.17k stars 2.76k forks source link

Proposal to move out GraphQL API from `fuel-core` to `fuel-indexer` #588

Open xgreenx opened 2 years ago

xgreenx commented 2 years ago

Overview

The proposal was discussed with the fuel-indexer team and accepted. You can find the proposal's description at the end of this description. The beginning of its description tracks the progress of the implementation.

The progress of the implementation

Unresolved questions

Child tickets:

The initial proposal

Summary

The fuel-core implements CRUD functionality, interaction with the user, indexation, subscription on events(not implemented yet), and implementation of some specific queries. All this API functionality is not required for the blockchain.

This issue is a place where we can discuss which parts of the API functionality we want to move out and what not. Where do we want to implement moved-out functionality, how, and who?

It may affect other teams(indexer team, dev-ops team, documentation team), so they also may, and should participate in the final decision.

Motivation

Moving out API functionality brings benefits along several different axes:

Performance

Right now, many supported queries are done in a non-optimal way(O(N), O(N^2), O(N^2 * log(N))). It is because we don't have indexes for some fields or pairs of fields. It is a vulnerability for the mainnet because anyone with a few requests can shut down/block/overload the node(you can create 1000 dust UTXO and send several queries to get a spendable balance).

Almost all input queries require iteration over all owner's entities in the database with a load of all fields and filtering(by asset id or spent status) them in runtime.

Example of owned_coins_by_asset_id image

We can manually add indexes into RocksDB, but it:

Using the database that supports the creation of indexes solves many problems, but it will be a pain for us to manage all schemas for different databases.

Code-complexity

Integration of API into other parts of the blockchain breaks the abstraction and requires creating workarounds or writing non-optimal code.

If the API functionality is part of the blockchain, it creates the following issues:

If the API is a separate node/binary/daemon, it can have its logic on how to process each block and transaction. It can, optimally, update all tables/entities in the database and use proper indexes(and manage them) to support all required queries. The entry point may be the typical fuel-p2p service. But except for the p2p service, it also will contain Rest API to work with users.

Another good point is that API is independent of the blockchain node, and it is not a problem to make changes in the DB schema to support a new query or data format.

Parallelisation of work

The proper and actual API is essential for a good development experience and requires the introduction of many endpoints. The client team is more focused on the blockchain changes first, so unable to deliver it in time. With a separate API node/indexer, we can have another team that is entirely focused on that.

It makes the development parallelizable from the beginning. Of course, those two teams should sync periodically about breaking changes(p2p protocol, entity layout), and the API team will depend on changes in the fuel-core. It forces the development of fuel-core to be more modular and act as a crate more than the package. But the dependency will be only on several services, so we can minimize the number of conflicts.

Possible solution

Fuel already has a particular product indexer, that indexes blocks, transactions, and events, and the team grows(human resources). The indexer uses SQL database under the hood and supports indexation. Also, in the future, it can be our full history node and support timeline queries.

The indexer auto-generate DB schemas based on the entities description from handlers. But it doesn't have schemas for common data used by the blockchain. Moving API to them also means starting indexation and storing blocks/transactions/data required for the queries. As a consequence, support of additional schemes in the database with manually specified indexes based on the list of supported queries(as an example coinsToSpend).

Possible problems

Unresolved questions

  1. Which GraphQL queries do we want to process in the fuel-core, and which do we want to move out?
  2. Do we need to support some GraphQL queries on the fuel-core? Or better make it a fully p2p node without any endpoint?
  3. How better to start the transition, and who will work on it?
  4. Who is responsible for coordinating future work with the indexer team(like breaching change notifications/discussions)? Do we need a coordinator?
  5. May it somehow affect future integration with the bridge and relayer?
  6. What is the approximate scope of the work? Can we finish it before the mainnet and update all repositories/documentation?
Voxelot commented 2 years ago

I agree with the sentiment that a node only functioning as a pure blockchain validator / block producer shouldn't have to incur extra database hits to build and maintain secondary indexes only used by the SDK or API consumers. However, we also need to balance making our lives easier with a simpler client codebase vs shunting additional complexity onto users. This decision should be informed by our philosophy of:

Efficiency & developer experience is everything

Splitting fuel-core into multiple binaries (p2p only blockchain node vs separate API layer process) has a real cost on DX (developer experience) that is not trivial. It also isn't standard behavior. For example, most blockchain clients such as Ethereum provide at least some kind of basic RPC API built-in. A graphql first approach has been a key selling point of Fuel since the beginning, so we aren't going to use REST or jsonrpc.

While the idea of a headless client is interesting, I think that should be a cli flag rather than a separate binary IMO. Internally we can structure the codebase however we like for modularity, but the final product should be a single binary that uses runtime feature flags to enable/disable functionality such as API endpoints. This gives the ability to run a node in a more efficient manner when desired, without causing major breaking changes or degradation to the DX.

Relying on the indexer for more complex or dapp-specific queries makes sense, however, there should be a core set of APIs available on any fuel-node that allow for some bare minimum usage of the network with or without operating an additional indexer node or additional database. I don't think it makes sense to delegate something like transaction submission to an external process or indexer since it's so heavily related to the transaction pool. The other issue with not exposing RocksDB data via graphql at all, and entirely relying on an API node using some other kind of datastore, is that the client essentially becomes a black box. Most of our integration tests would become useless without incorporating several services.

It is a vulnerability for the mainnet because anyone with a few requests can shut down/block/overload the node

The blast radius for bad API queries is somewhat limited by our planned use of sentry nodes for mainnet. However, that's no excuse to leave ourselves open to intentional or unintentional DOS. Unless we can actually extract these APIs to the indexer, we need to address these performance issues within the node.

The question around which queries to keep in fuel-core node should be based on some criteria:

coinsToSpend would definitely benefit from being a built-in piece of functionality for the indexer node. However, given how core this API is for doing any sort of basic SDK actions, it would be fairly painful on the DX as well.

I'd rather find a way to modularize DB activity related to servicing the APIs (i.e. index maintenance) without needing to do a complete re-architecture or harming our DX. This way, if there is complicated logic related to making API queries faster, it is self-contained and easy to disable at runtime.

xgreenx commented 2 years ago

The developer doesn't need to install/interact precisely with fuel-core. We can have a separate package that starts the indexer and fuel-core together. Indexer has a GraphQL port exposed to the user. Under the hood, it forwards all transactions to fuel-core.

In this scenario, we have three entities:

The user installs/works/interacts/plays with fuel-api-node. The network uses the same fuel-api-node to provide the information. The DX is the same in that case, deployment is more configurable. The problem is that most validators will be fuel-p2p-node, and only small parts of nodes will be fuel-api-node.

But if you are fuel-api-node, you are the entry point to the network, and you can decide which transactions to share and which to ignore=)

Voxelot commented 2 years ago

Connecting things under the hood between fuel-indexer and fuel-core is a good idea. I think the main architectural issue we have right now is that all the APIs for fuel-core are implemented inside of graphql resolvers.

Ideally, we move all the core impl logic inside the graphql resolvers into a Rust API directly accessible on FuelService. Then, suppose we wanted to make a fuel-api-node that embeds fuel-indexer and fuel-core into a single binary together. In that case, fuel-indexer could just accept a trait for all the fuel-core API's it needs (such as fetching blocks or receipts) which could either be backed by the fuel-gql-client (for remote access) or an actual FuelService instance (for embedded).

This also would allow us to explore supporting alternatives to graphql in the future, as the API handlers would just be shims over the Rust API.

As far as extracting graphql from fuel-core altogether into a separate fuel-api-node process, I think this is a big architectural pivot that would put our current roadmap at risk. If there are APIs we are concerned about supporting due to performance concerns, we should come up with ways to rate-limit or restructure them to be more efficient on the current fuel-core architecture for now. Once the indexer is closer to production ready, and we've delivered the bridging milestones, we can revisit these big ideas.

xgreenx commented 2 years ago

Ideally, we move all the core impl logic inside the graphql resolvers into a Rust API directly accessible on FuelService. Then, suppose we wanted to make a fuel-api-node that embeds fuel-indexer and fuel-core into a single binary together. In that case, fuel-indexer could just accept a trait for all the fuel-core API's it needs (such as fetching blocks or receipts) which could either be backed by the fuel-gql-client (for remote access) or an actual FuelService instance (for embedded).

Yep, it is an excellent way to segregate GraphQL and code functionality=)

This also would allow us to explore supporting alternatives to graphql in the future, as the API handlers would just be shims over the Rust API.

Yep, we need to make GraphQL independent during segregation and act as a wrapper around the core functionality. In this case, it can be replaceable like the lego detail in the constructor=D

As far as extracting graphql from fuel-core altogether into a separate fuel-api-node process, I think this is a big architectural pivot that would put our current roadmap at risk.

Yea, it is what I don't like. But in the long perspective, it will simplify our development and support of new queries(and I hope it will save a lot of time).

Once the indexer is closer to production ready, and we've delivered the bridging milestones, we can revisit these big ideas.

I agree. For now, I think we can already create several tasks related to this big idea and start slowly doing them(based on free resources). For example:

These tasks(and maybe you will see some more) can be helpful for the indexer team and us.

And as you said, when the indexer is almost production ready, we can think about the full transition to them and the creation of fuel-api-node.

If there are APIs we are concerned about supporting due to performance concerns, we should come up with ways to rate-limit or restructure them to be more efficient on the current fuel-core architecture for now.

We need to have count limitations for each query based on its performance. Maybe we also need to think about the framework to measure the gas of execution(like we want to have for VM operations)=) But again, we don't have time to do that, and maybe that idea can be useful for the indexer team.

The problem is that sometimes one request(as coinsToSpend) may consume a lot of time. So I see here several essential(for the stability of the node) tasks: