search

near / nearcore

Reference client for NEAR Protocol
https://near.org
GNU General Public License v3.0
2.31k stars 619 forks source link

Subscription to state changes AKA Events #1546

Closed ilblackdragon closed 4 years ago

ilblackdragon commented 4 years ago

Instead of doing events via logs, the idea is to have subscription on changes in the contract state.

E.g. developer subscribes to monitor key (or prefix) and gets changes as blocks are rolling in.

For example RPC can be subscribe <account_id> + <key prefix> and poll or WebSockets for receiving information about changes in key prefix.

Additionally need to make sure that this doesn't overload node by subscribing to too many changes.

Implementation detail includes:

MaksymZavershynskyi commented 4 years ago

@k06a Answering the following question from slack:

but how this would simplify complexity of light client checking LOG contained in tx?

  1. We don't need to implement separate merkelization of logs, since storage is already merkelized.
  2. We want events to have the following two features: be subscribable and be structured. We also want smart contract developers to choose their own format of how they serialize events (e.g. JSON vs binary). The motivation behind the subscription is that we want front-end to simply listen on a websocket for a specific event. E.g. consider an ERC20 contract that has high usage, we want users to be able to subscribe to the events only relevant to their tokens on that contract. Currently, one would have to parse every single log of every transaction that touches this contract. For high-usage ERC20 and users monitoring this contract from a webpage loaded into their browsers through nearlib this is not a feasible option. We could allow users to subscribe to specific logs through a websocket, but then this feature would be at odds with allowing structured logs. If logs are just generic binary blobs (with format defined by contract developer) then there is no way to have a node-level functionality of query specific logs, e.g. relevant to the given users, because node does not know the format of logs that smart contract developer chose.

However, if we allow subscribing to specific keys in the key-value storage, then smart contract developers can have any kind of structure in values. Also, with the keys we can subscribe to a specific prefix.

frol commented 4 years ago

I can give this issue a look, but I still have at least one top-priority item (multi-net support) in NEAR Explorer. I definitely have a lack of knowledge of the nearcore internals, so I will need some walkthrough guidance (with an emphasis on the concerns you have already learned here). There has been a longstanding issue of having the pubsub support in the API, and I am keeping an eye on available solutions for RPC & pubsub under a single protocol (RSocket is "new", but promising; WAMP-proto is quite good, but there is no Rust support; gRPC is a no go for web browsers; thus, we should keep doing the simplest approach available at the moment: plain websockets & HTTP)

frol commented 4 years ago

I am done with the Explorer multinet support (yay!), so I can take a closer look at this one.

@evgenykuzyakov @nearmax @ilblackdragon May I ask any of you to share your vision on a call?

MaksymZavershynskyi commented 4 years ago

Let's have a video call. Please grab the time slot on our calendars. 9AM-6PM SF are my working hours.

bowenwang1996 commented 4 years ago

@frol can you add me as well?

frol commented 4 years ago

Given we always wanted nice PubSub and RPC public APIs, I took some time to investigate the space.

Here are the requirements I came up with:

As always, there is no silver bullet out there in the APIs space, so here are the options I have laid out:

WAMP-proto is the tool specifically designed for PubSub (+RPC) that you don't want to manage in your app (I have been successfully using it for Explorer), but its Rust support needs to be ported to Tokio (0.2 with async/await ??) [the client specification is straightforward, so my safe estimate is 1 week of work to get it running].

Plain WebSockets implementation should be ok, but I fear it is going to be too basic.

What are your thoughts?

frol commented 4 years ago

Just for the reference. After a discussion with @ilblackdragon and @bowenwang1996, I finalized the design as follows: implement WAMP-proto router protocol with only support for ~a broker role~ a dealer [RPC] role with an advanced extension "progressive calls" using Actix WebSocket implementation.

Pros:

Cons:

On another point, we discussed that it would be great to subscribe to the events specifying the starting point to make sure you can always reconnect and receive all the events starting from a specific point, so you don't need to do RPC calls to get complete data.

Also, we concluded that it is better to limit the events to blocks that are finalized.

MaksymZavershynskyi commented 4 years ago

What is a "finalized event"?

frol commented 4 years ago

@nearmax I mean that we don’t want to take care of possible network forking (in which case we would need to produce “reverting” events), so it is simpler to only produce events about the blocks that reached the non-revertable state (I have troubles with the exact blockchain terminology here, so the key is the finality gadget)

ilblackdragon commented 4 years ago

I think now that we have the finality gadget we can create a message from Client that posts to ?? Actor (the one responsible for WebSockets, as this is useful event for other subscriptions like blocks and txs) about final blocks. And then fetch events for that block height.

We will store on node side changes to the state in a form that is easily queriable. E.g. just as a state_root+key

This will need to be garbage collected later with pretty quick schedule.

MaksymZavershynskyi commented 4 years ago

This will need to be garbage collected later with pretty quick schedule.

We can just use Rc or Arc and make it auto-cleanable.

frol commented 4 years ago

Just for a future reference, there is a pretty well-designed WAMP client implementation for Rust async/await + tokio 0.2: https://gitlab.com/elast0ny/wamp_async-rs. We still need an embeddable router implementation for nearcore, but it is still a useful source for inspiration.

k9thebeast commented 4 years ago

+1 on this feature from Stardust

frol commented 4 years ago

@k9thebeast Do you think the new experimental state changes API is sufficient?

ailisp commented 4 years ago

Any future plans on single transaction event? while near/docs#288 is good for "Subscription to state changes", it's inconvenient to get single transaction finish/fail event and "subscribe to specific kind of log/receipt happens", like what other blockchain had such as: https://docs.solana.com/apps/jsonrpc-api#accountsubscribe https://docs.solana.com/apps/jsonrpc-api#programsubscribe https://polkadot.js.org/api/start/api.tx.subs.html#transaction-events

frol commented 4 years ago

This issue has been superseded by #2651. All the future work should be offloaded to an external implementation based on NEAR Indexer framework /cc @khorolets