Support events

[seanyoung]

Problem

In Solidity, it possible to emit events. Events are read-only records which are stored on-chain, which can be used for triggering off-chain programs.

contract auction {
        event BidReceived(address account, int amount);

        function bid(int amount) public {
                // ...
                emit BidReceived(msg.sender, amount);
        }
}

Events are ABI encoded so that using the correct ABI, events can be decoded off-chain. For example, Hyperledger Burrow writes the events to an SQL database using vent

From the BPF program's perspective, events are write-only.

Proposed Solution

Provide a BPF syscall which provides a deposit-event/emit-event type functionality.

Provide an RPC which subscribes to events with filtering functionality.

[gregl83]

This is a hard dependency for my project and other projects requiring off-chain network IO.

As an example, Tron Network nodes have a plugin for piping events into Kafka allowing for off-chain data to be fetched and posted on-chain via handlers calling programs/contracts. With this callback-like functionality data can be persisted on-demand with eventual consistency.

Reliable Event-Driven stream processing enables a lot of product development.

[jon-chuang]

Quite curious @gregl83 , why using RPC subscriptions to certain accounts wouldn't suffice? I'm guessing its because RPC subscriptions don't allow for fine granularity of events?

[gregl83]

Quite curious @gregl83 , why using RPC subscriptions to certain accounts wouldn't suffice? I'm guessing its because RPC subscriptions don't allow for fine granularity of events?

Hi @jon-chuang, super new to the project and trying to evaluate an unplanned pivot to Solana for a WIP POC. From what I gather, the RPC subscribe method operates without a guarantee of messages being received at least once (fire-and-forget). That is, if the RPC subscriber experiences downtime or encounters a processing exception, messages can be dropped. Perhaps there is an embedded queue?

Additionally, there is a lot of flexibility with event sourcing using a high-throughput and durable message bus like Kafka. Replays, filtering, confirmed receipt, concurrency, ordering etc.

[jon-chuang]

@gregl83 , I was wondering if the msg macro, which prints to log, serves your needs?

I was thinking that the easiest way to get this functionality would be if these logs could be filtered on the RPC server side by program pubkey.

[buffalu]

this isn't exactly what you're asking for, but might help you limp by for now.

anchor emit example: https://github.com/project-serum/anchor/blob/master/examples/events/programs/events/src/lib.rs#L10

source: https://github.com/project-serum/anchor/blob/81e03c5e379cd9a98e13ca8e2ef06b00043de7ab/lang/attribute/event/src/lib.rs#L48

[jon-chuang]

@buffalu , I believe emit! uses the msg! macro under the hood and simply prints to log as well.

Just a convenience macro for converting an arbitrary struct to a string.

[gregl83]

@jon-chuang, I believe you're correct, emit! is a convenience wrapper with event type as signature, but thank you @buffalu for the example.

Jon, is there a guarantee that log messages will include metadata such as program id? Do client requests to any node reproduce the log message during replication?

If RPC is indeed fire and forget, then has anyone setup something like logstash which can use Kafka as a sink and handle restarts?

[jon-chuang]

@gregl83

1. logs will print program IDs
2. I'm not sure if logs are dropped by default and if there are methods to sync them from other nodes. But if you request logs from multiple RPC nodes (and dedup via transaction ID) you'd have the fault tolerance you need, I believe.
3. If you're running your own RPC node, it may be possible to save the relevant logs when catching up to the network. But you'd need to ask for a specific snapshot to sync from (and the further you've fallen behind, this could take a very long time)

Edit: this issue could be relevant https://github.com/solana-labs/solana/issues/18197

[charlesvien]

👍

[biserd]

Struggling with the same. Anyone can point me to the relevant documentation for emit() substitute? Thanks in advance!

[fw-aaron]

Struggling with the same. Anyone can point me to the relevant documentation for emit() substitute?

@biserd the substitute is use !msg in your solidity and run a RPC client against your solana node to read those msg!s.

[fw-aaron]

[Wouldn't] using RPC subscriptions to certain accounts ... suffice?

@jon-chuang The way I see it, !msg on blockchain with a RPC client looking for those !msg messages should be sufficient, but there is a detail that I'm not sure how to handle: finality. What happens if I see a !msg using the RPC client and record/aggregate that into my off-chain database, then the transaction is rolled back or the blockchain is forked so that the transaction in question is no longer on the blockchain?

[ryoqun]

note that log based events might be affected by truncation: https://github.com/solana-labs/solana/issues/23653, https://github.com/project-serum/anchor/issues/1613