ethereumProvider API

[frozeman]

Abstract

This draft is to discuss the ethereum API, an object exposed by ethereum native browsers and tools like Mist, MetaMask and status.im.

The API should be small and clean and allow all kind of libraries to attach to it.

The class of the object should be EthereumProvider, the object name ethereum.

API

The ethereum should inherit from a EventEmitter type class, and provide add/removeListener, on/once/off etc.

// callback: a error first callback
ethereum.send(methodName, params, callback)
> undefined

// CALLBACK return
> err, result

// returns subscription notification,
ethereum.on(subscriptionType, handler)

subscriptionType = `eth_subscription`

// HANDLER return
> {
       "subscription":"0x2acffa11d68280c2954daeb77cb849d9",
       "result": {...}
    }

// react on connection start
ethereum.on('connect', handler)

// HANDLER return
null

// react on connection end
ethereum.on('end', handler)

// HANDLER return
Error Object with message (timeout, error, drop, etc)

[frozeman]

@kumavis @jarradh

[kumavis]

I suggest that connect() and the relevant events are removed from the standard. If a special subclass of ethereumProvider needs it, it can implement it.

Why did you include those?

Any thoughts on returning a Promise in ethereumProvider.send(payload, callback)?

Looks good!

[kumavis]

small nitpick ethereumProvider.on('timeout', callback) the callback reference here should be called handler, as callback implies that it should only be called once

[frozeman]

changed to handler.

Hm yeah, if the provider always takes care of connecting and re-connecting automatically we wouldn't need connect()

Concerning promise. i would like to keep this lib small and a promise is an extra dependency. Apart from that i'd follow node.js defaults here, which is error first callback :) But im open for debate..

[kumavis]

here's a version with the connection semantics in a separate subclass

EthereumProvider : EventEmitter

// payload: is a valid jsonrpc 2.0 object
// callback: a error first callback
ethereumProvider.send(payload, callback)
> undefined

ethereumProvider.on('data', handler)
> undefined

// The subscription received in the handler:
{
  "jsonrpc":"2.0",
   "method":"eth_subscription",
   "params": {
       "subscription":"0x2acffa11d68280c2954daeb77cb849d9",
       "result": {...}
    }
}

EthereumWebsocketProvider : EthereumProvider

// Will try to connect, the ethereumProvider should be already connected when the webpage is loaded and firse the "connect" event at start.
ethereumProvider.connect()
> undefined

// react on connection start
ethereumProvider.on('connect', handler)

// react on connection end
ethereumProvider.on('end', handler)

[tcoulter]

My biggest ask is that dapp developers not have to code for specific provider types. For instance, a dapp developer should not have to maintain a specific code block for Metamask-style providers where no connection information needs to be managed, and another code block for Mist that, say hypothetically issues a provider where connection information does need to be managed. (If unclear, I interpret @kumavis's suggestion above as requiring dapp developers to do exactly that).

Creating a library that supports that ask would require either we:

1. Expose a connection abstraction that every provider must implement, for all provider types; or
2. Expose no connection information at all.

My vote is on a modified hybrid approach, as described below, as I think the current connection abstraction is too specific to a certain connection type, and exposes too many temporary error states. Instead, I think we should focus on whether a provider has been "started" and "stopped" -- i.e., is it ready to provide you the information you need, or not. Here's what I propose:

// Implemented by all providers; is a no-op for providers that don't need to 
// connect to anything or don't require any type of polling. If there's base
// code we decide on, it could implement this no-op by default.
//
// The callback is fired when the provider is successfully started.
// This is important for providers that work over an established connection
// or that are in-process and connect to a database backend.
ethereumProvider.start(callback)

Similarly, we could implement a stop method that can also be a no-op if not necessary.

// The callback is fired when the connection is successfully stopped.
// Important for established connections as well as database backends.
ethereumProvider.stop(callback)

Sending data is the same as the examples above, but the handler has changed (more on that below):

// payload: is a valid jsonrpc 2.0 object
// callback: see below
ethereumProvider.send(payload, callback)

Also, on('data') can be more generic. Instead of being limited to subscriptions, which assume a specific transport, on('data') can provide a handler for all data coming through the pipe. See the note on handlers below.

// returns payload and response, see handler details below
ethereumProvider.on('data', handler)

As well, theon('error') event has been modified:

// Will return an error as a result of any payload; this includes connection
// errors as well as rpc errors. See below for handler details.
ethereumProvider.on('error', handler)

All callbacks and handlers on send, on('error'), on('data') are of this form:

function(err, payload, response) {
  // connection error
  // payload (request)
  // response: response body returned by the RPC server
}

Note that err above would only include connection errors as, if RPC clients are following the specification, RPC errors are part of the response body.

An http provider pointed to an invalid non-responsive or incorrect url, for example, would expose connection errors in the err object when a request 400's or 500's. Websocket errors would return a connection err (err) when the connection hasn't reconnected after a specific period of time. The TestRPC provider would return an err when there's a runtime exception that should never have occurred (i.e., the TestRPC provider would run in the same process, so wouldn't need to make use of this error path as a connection error).

I see all of this discussion as akin to leveldown. An output of this discussion should not only be an abstraction that works for any connection type (or none at all) with the littlest effort on its users; but we should also produce an abstract-leveldown-like codebase (we'd call it abstract-ethereum-provider) that lets any data layer fit in and the connection type inconsequential.

[kumavis]

I interpret @kumavis's suggestion above as requiring dapp developers to do exactly that

@tcoulter you seemed to have misunderstood my proposal. I specified a standard EthereumProvider and an example subclass of that standard EthereumWebsocketProvider. The standard is what dapp developers would expect to be injected by the browser. The subclass would be for the case where the there was no standard api object injected and the dapp dev wanted to set up their own fallback that would necessarily require connect configuration be handled by the dapp dev.

[kumavis]

Also, on('data') can be more generic.

Instead of being limited to subscriptions, which assume a specific transport,

its not a specific transport, its a feature of the ethereum json rpc built on top of json rpc notifications.

on('data') can provide a handler for all data coming through the pipe. See the note on handlers below.

its nice having just a handler for server sent notifications, but interesting idea

[kumavis]

Creating a library that supports that ask would require either we:

for the sake of simplicity I advocate for 2. Expose no connection information at all. in the base standard

[tcoulter]

for the sake of simplicity I advocate for 2. Expose no connection information at all. in the base standard

I went down that path too when I originally started writing my original comment. If you as a dapp developer want anything extra it means you have to explicitly code for a specific environment (i.e., the websocket provider), which is why I later chose the hybrid/more general abstraction approach. However, if the custom provider classes are meant to only be used as a fallback, then I could get into that.

its not a specific transport, its a feature of the ethereum json rpc built on top of json rpc notifications.

👍 Good to know, thanks. I find the general on("data") to more useful (at least theoretically), but could go either or as the behavior I'm looking for could easily be shimmed.

[frozeman]

Ok sorry for the late response, but i was on a holiday. @tcoulter @kumavis

So concerning exposing no connection data: IMO thats wrong. I do agree we can consolidate "timeout" and "error" to simply "end". But a library like web3.js can and must be able to know about connection drops, which can happen in mist, as well as any internet connected device. Web3.js and other libraries in this case must know about it and provide subscription regains strategies and lost connection behaviour.

Or the dapp developer itself must be able to act on connection drops as well.

Its not feasable to try to mitigate all of that under the hood in the EthereumProvider itself.

Concerning the generic on("data"):

This makes building libs like web3.js a lot more complicated, as each of those libraries needs to handle the mapping between request and responses and do the mapping of JSON rpc IDs themselves. This is easily taken away by ethereumProvider.send(payload, callback), and i think thats a good thing and makes it more fun to build web3.js like libraries.

Concerning the ethereumProvider.connect()

This is necessary to force regaining connection tries, but we can get rid of it, if we make sure to call the "connect" event on reconnect, and try to reconnect under the hood automatically.

Concerning Separate WS provider

The ethereumProvider above is on purpose generic for all type of providers, be it HTTP, IPC or WS under the hood. Its kind of its own type and can be used in web3 like libs, as they would a WS provider.

If ethereumProvider is not defined the dapp has to load its own WS or whatever provider to get a connection. The browser can't take care of this.

So given the above feedback i would suggest the following API:

// payload: is a valid jsonrpc 2.0 object
// callback: a error first callback
ethereumProvider.send(payload, callback)
> undefined

// returns subscription notification,
ethereumProvider.on('data', handler)

// The subscription received in the callback:
> {
  "jsonrpc":"2.0",
   "method":"eth_subscription",
   "params": {
       "subscription":"0x2acffa11d68280c2954daeb77cb849d9",
       "result": {...}
    }
}

// react on connection start
ethereumProvider.on('connect', handler)

// react on connection end
ethereumProvider.on('end', handler)

[danfinlay]

What happened to your PromEvent pattern? I was fond of that. I'm not as big a fan of a global data event.

I'd much rather that each individual query is subscribable, allowing reactive state updates.

Example:

token.getBalance(myAccount)
.on('update', updateUI)

Also, I think a single PR is too simplistic a form to discuss an entire spec. I would propose the spec proposal be a repository, and then individual features & changes could be pull requests, discussed individually.

[frozeman]

This here is about a basic provider provided by mist, metamask and others, not how web3.js or any other library on top will work ;)

Off Topic: Tho as the subscriptions we currently have are rather limited in the RPC, things like you propose there, are nice but won't be available in web3.js 1.0. The PromiEvent is only working on the send function right now: http://web3js.readthedocs.io/en/1.0/web3-eth.html#id62 And for contract methods: http://web3js.readthedocs.io/en/1.0/web3-eth-contract.html#id21

[frozeman]

To get back to this and finalise it soon. @kumavis @tcoulter

Here is an idea from @MaiaVictor which is what you @kumavis probably mentioned in one of these calls, to have a simple function object which can handle the calls, without exposing formatting the RPC calls yourself.

I could imagine something like this, but with the subscription addition: https://github.com/MaiaVictor/ethereum-rpc

[frozeman]

@kumavis @tcoulter i talked to @MaiaVictor and took in his suggestions to strip away the RPC itself, so that we can in the future change the transport layer without the need for changes in this API here.

The provider would take care of generating valid JSON and IDs and you only pass in RPC method names and get results back.

The end and connect is still there as its necessary for dapp developers and high level libraries like web3.js to detect connection drops and handle them well, or for apps to show connectivity statuses to their users, and/or warnings.

So i would propose the following:

// payload: is a valid jsonrpc 2.0 object
// callback: a error first callback
ethereumProvider.send(methodName, params, callback)
> undefined

// CALLBACK return
> err, result

// returns subscription notification,
ethereumProvider.on(subscriptionType, handler)

subscriptionType = 'eth_subscription' or 'shh_subscription'

// HANDLER return
> {
       "subscription":"0x2acffa11d68280c2954daeb77cb849d9",
       "result": {...}
    }

// react on connection start
ethereumProvider.on('connect', handler)

// HANDLER return
null

// react on connection end
ethereumProvider.on('end', handler)

// HANDLER return
Error Object with message (timeout, error, drop, etc)

[VictorTaelin]

I agree with that spec. I understood on("data") wrong previously and it is actually fine like that. I'd personally have a simpler API (i.e., eth.method(a,b,c) or eth("method", [a,b,c]), and leave the on("connect") and on("end") events out as I think it would look prettier to just have it on the browser. In practice, that'll just cause each DApp to implement its own "connection lost" message on top. I don't see much critical use to it. But I can live with both of those things. This proposal is simple enough to feel safe depending on it.

Just a small refinement, what you think of:

ethereumProvider.on("eth_subscription", function (subscription, result) { ... })

Or something like that? So it has a direct pairing with eth.send("method",...).

~~

Looking forward to see what the other guys think, so we can just go ahead and work on top of this standard.

[frozeman]

I like the idea of ethereumProvider.on("eth_subscription", ..., as we also have ethereumProvider.on("shh_subscription", ... and so on. Will add this.

[tcoulter]

As I'm thinking about this, all the proposals above (even mine) seem to extremely complicate the task of making an ethereumProvider. I'm starting to think we should have ethereumProvider.send(...) be the only requirement for the ethereumProvider spec. Given the most recent spec above, we're currently asking all ethereumProvider writers to support the following:

• A method calling function:send()
• An event system for subscriptions (that handles a subset of all available provider methods): eth_subscription and shh_subscription
• An event system for transport level state changes that don't apply to all possible transport types (i.e., transports like RPC, REST, in-memory providers like the TestRPC, etc. could care less about these state changes; it really only applies to websockets)

If I wanted to create a simple provider for testing, i.e., as a mock provider within my application tests, there's a lot I'd have to create just for it to be compliant. I think this is too much of a burden.

Instead, we should rally around one method, and have it be the only requirement to write an Ethereum provider: ethereumProvider.send(method, params, callback).

I agree with your concerns that transport level information is important. To get around this, I suggest we do two things:

1. Add more provider methods like transport_connected, that can be called from within a dapp, which can provide the dapp with information about whether or not the transport is connected. This will have to be polled on a regular basis via a timeout, but it won't require a network request for all transport types; i.e., this will be answered by the transport itself in the case of websockets, or in the case of a mock provider it can simply return true. Some providers, like an http provider, may also choose to simply return true, as the send method will fail if a specific request fails.

2. For events, and anything you'd like to be subscription-based, this can easily be added via a wrapper around the ethereumProvider. e.g., it's not hard to write an object that inherits from EventEmitter and wraps a provider's send method, emitting an event after each callback is triggered for the methods that you're interested in (i.e., eth_subscription and shh_subscription). In fact, using this model, I can easily write a provider that triggers an event for every method, which suddenly makes my request-and-response-type dapp into event-based dapp:

var inherits = require("util").inherits;
var EventEmitter = require('events');

inherits(EventedProvider, EventEmitter);

function EventedProvider(provider) {
  this.provider = provider;
}

EventedProvider.prototype.send = function(method, params, callback) {
  var self = this;
  this.provider.send(method, params, function(err, result) {
    callback(err, result);
    // Send all the information just in case a handler could use it.
    self.emit(method, params, err, result); 
  });
}

Of course, the above code is for Node, but it's very easily bundled for the browser.

The above code would provide subscription events for eth_subscription and shh_subscription, as well as events for every other provider method ever called which is super handy. And people could use the EventedProvider if they'd like their dapp to have events (perhaps using some evented wrapper, of course) -- and they won't need to implement an event system for providers that don't need events, like the ones I mentioned above. This could easily be extended to poll transport_connected under the hood to provide the connect and end events, and the poll interval desired. Now, I know "polling sucks", but remember, most of the time this will be answered by the transport itself, which means it will all happen within the current execution environment (i.e., no network requests, etc.). Rarely if ever should transport_connected send a request down to the Ethereum client.

As an added bonus of this much simpler specification, it means providers can be composed. For instance, I can make a caching provider that takes a different provider as input, and suddenly I have a caching layer with no extra work. Adding an event system and a mechanism for transport state change handling makes this much harder, as if you were to compose them you'd have to propagate those events all the way up.

To sum up, the features desired by the above spec are good, but if we simplify the spec further we can reduce overhead and get a lot more benefit. We can still have all our cake and eat it too with regards to transport level changes, though, by adding the transport_connected provider method, and any other transport-related methods required.

[VictorTaelin]

I agree with all you said. I mean, if you read my proposal, it is exactly what you proposed, ethereumProvider.send(method, params, callback) (except I also removed the .send). In other words, ethereumProvider becomes basically a global function that calls the RPC.

Fabian, though, argued we need on("data") because of streams. I completely agreed with his point: if ethereumNode only offered pooling-like operations, then we couldn't get stream-like performance from it. Now, reading what you said and under further thoughts, I noticed this is false. We can still have the exact same performance characteristcs when PUB/SUB streams are available, even if we don't have on("data"). Or, in other words, given an ethereumProvider with only direct RPC methods, we can reconstruct one with on("data") with the same performances characteristics. As such, there is, indeed, no reason to have events on ethereumProvider at all.

[VictorTaelin]

So, in short, I'm 99% in favor of your proposal (1% is that additional .send removal). I'll implement it on Ethereum-RPC, and I'll also implement a wrapper with that builds the longer spec (i.e., on("eth_subscription",...), on("bzz_subscription",...), on("connect",...) and on("end",...)) in terms of the simpler one, to test the hypothesis that it is possible without performance losses.

I'll be waiting to see what everyone else thinks.

[kumavis]

here is what i think most stacks will look like

• high level userspace library ( e.g. framework integrations, fancy subscription mgmt )
• eth api ( rpc methods exposed as functions, e.g. eth.getBalance(address) )
• json rpc request handler ( request and response are paired by .send, notifications are emitted as event )
• connection duplex stream ( requests go in, responses and notifications come out )
• transport ( websockets, http, ipc, carrier pigeon )

web3.js has been more of the (high level userspace library) variety, with a lot of API surface and a lot of change. This meant cool features (like web3.Contract) but breaking changes. When standard APIs do this they break the web. So web3.js should continue to grow and experiment, and the standard should be at some lower level (I think we all agree so far).

The goal here is to determine what the API standard for Ethereum Browser Environments should be, and thus the actual transport between the app using this API and the logic that process those requests is out of scope. That narrows our layer options to:

• eth api
• request handler
• connection duplex stream

providing something at the eth api level would be more akin to existing browser APIs, but it requires you to know what methods the rpc endpoint provides. Perhaps some sort of handshake could be done here on connection, like the introduction of a "web3_availableMethods" rpc method.

the current web3 provider behaves like the request handler layer. this layer is fairly generic and provider some utility over the raw connection duplex stream. It groups response with their requests, and provides a way to listen for server-pushed notifications. At this level we're at the level of the official json rpc spec, no ethereum semantics or custom functionality. We could add a little more custom functionality on top that grouped server-sent notifications with their original subscriptions, but i would advocate that we handle that at a higher level

The connection duplex stream level is certainly the purest. Data goes in, data comes out. Sort of an unusual api for a browser standard, but very unixy.

So what do we do?

If our goal is to establish an standard Ethereum Browser API, then the eth api seems to make the most sense. (e.g. something like eth-query ) But the eth rpc method space is still in flux, lots of new things are being introduced and deprecated. Makes it a difficult choice.

The request handler layer (essentially the original proposal of this thread) would be a stable standard but certainly an unorthdox one.

Which of those two im leaning towards changes day to day. Do you agree with my identification of the layers of abstraction and their pros and cons?

[danfinlay]

I agree that it's unusual just how pure this new proposed standard is. It's almost an over-reaction to criticisms of the web3 library's size. No other browser standard wraps multiple other methods, from callbacks to events, into a single function.

This method would make the API almost completely useless to people not importing external client libraries, which is a new direction for a web API.

That said, it definitely achieves the goal of "minimal surface area" for user-space libraries, and gives developers a lot of room for future iteration, and keeps page load time minimal.

Maybe it's worth considering: Which minimal API is more useful to developers?

While an availableMethods call could load methods from a remote process, this forces all other feature detection (which is often done at load time) to be done after an async resource load, which could actually result in slower page loading than just injecting functions for each capability in the first place.

So while I can't really complain with the current proposal as a very low-level access method, I think at least slightly longer term I will be in favor of some kind of global object that has methods for each current feature.

[danfinlay]

I think @tcoulter's concern is also valid, part of this spec is to make it easy for people to develop clients against it.

Since that layer is easier to implement, and the other layer can be built on top of it, maybe we really can just make the pure-function API first, standardize it, and then start experimenting with other APIs to inject as well.

[VictorTaelin]

Thanks for the analysis, @kumavis. I agree with your categories. In an ideal world, where the RPC API wasn't supposed to ever change, the best choice would be to standardize the eth api layer, as you put it. But, as you said, it isn't as stable and new RPC methods will be added.

This is confusing me, though. We're assuming the request handler layer (what we're proposing here) is somehow more stable and flexible than eth api. Why, though? There are two types of changes. Non-backwards compatible changes (e.g., removing eth_getFilterChanges) would break both DApps using ethereumProvider.send("eth_getFIlterChanges",...) and DApps using ethApi.getFilterChanges(). Backwards compatible changes (e.g., adding eth_something) are possible on both: on ethApi, add a new method and inject that instead; on ethereumProvider, just call the new method. So, on both cases, they're equivalent. One isn't more flexible than the other, the request handler is still exposing the same api, just indirectly. Am I missing something?

To be honest, at this point, I'm deeply confused with all the options and mindsets here. If I was alone in doing that I'd, before anything, focus on designing an RPC API that wouldn't need to change. I'd then just expose the API to the browser (i.e., something like kumavi's eth-query) and tell DApps they can rely on it. And that's it. Seemed so simple to me.

Now I feel like I'm being more destructive than productive on this conversation. I don't want to interfere anymore. I'm happy with most of those solutions. In the end, none of those details is a major issue. But losing time is. Let's not have our own blockchain size debate. I'd say, let's go with Fabian's last proposal. He has good reasons to think it is good, and, perfect or not, it is clearly sufficient. I just hope we can find some consensus soon.

[kumavis]

@MaiaVictor you are entirely correct about breaking changes with method removal ( and method behavior change ). One difference though is that if a new rpc method is introduced, it would be accessible via the json-rpc interface but not the eth API.

Perhaps the happy medium is something very close to what we have now: eth api (simple web3 / eth-query) with the json rpc request handler exposed ( web3 provider )

Let's not have our own blockchain size debate.

🤣 yes I agree -- though its damn hard to change things once people start using them so lets try to get it right this time

[frozeman]

Ok now read through all of that.

So first of a few informations, because i think not everybody caught that yet. There will be the new Pub/Sub API coming -> https://github.com/ethereum/go-ethereum/wiki/RPC-PUB-SUB (and in fact its already one year! implemented, and i guess parity too), which is thought to replace eth_newFilter and eth_getFilterChanges.

The reason why is to standardise notification streams, which in the new API we already have:

• logs
• newHeads
• pendingTransactions

And we could add a lot more.

The old filter was not flexible to add more and unclear to understand, we all just got used to.

The current way means, duplex streams are the new default, e.g. WS and IPC If we can convince the nodes to add polling support we can add HTTP back that list, but i think this will take a bit of time.

So given that fact we have now push, and this needs to be part of the standard a on('notifications|data|eth_subscription', .. is necessary, to receive PUSH notifications.

I also really like @MaiaVictor idea to strip away the RPC layer, as its not really necessary to know how methods on nodes are called, but just that you can call them.

I also like the last approach i posted, eg. ethereumProvider.send(methodName, params, callback) as if new methods get added we don't need to change the provider, but high level libs like w3.js can start using it. We don't want to update mist or metamask's provider all the time, because there is a new RPC method.

Therefore i really like the last approach. Its simple clean and allows for flexibility.

Concerning the connect and end events:

Those are necessary, because subscriptions are based on sockets, e.g. if your IPC/WS socket disconnects your node will cancel the subscription. So dapps and high level libs like w3.js need to be informed to be able to regain subscriptions, or fetch missing information. Currently we only have eth_getLogs as a way to retrieve past logs, but we might add a generic way to retrieve missed logs, when we add polling to subscriptions. All of these improvements would be possible with this new provider. So it is backwards compatible and should be safe for the future.

At the same time it allows dapp developers to even use it directly, which is far more complicated when we would require them to build JSONRPC objects, or even give them a raw duplex and they need to match responses themselves.

With this solution is think we have a great middle ground.

If you guys have better solutions, i would ask you to post example APIs, like i do. so we can agree father, because don't forget "people don't read"! ;) So the easier to digest, the better. so for readability i post the proposed solution again:

// payload: is a valid jsonrpc 2.0 object
// callback: a error first callback
ethereumProvider.send(methodName, params, callback)
> undefined

// CALLBACK return
> err, result

// returns subscription notification,
ethereumProvider.on(subscriptionType, handler)

subscriptionType = 'eth_subscription' or 'shh_subscription', 'data'? `notification`?

// HANDLER return
> {
       "subscription":"0x2acffa11d68280c2954daeb77cb849d9",
       "result": {...}
    }

// react on connection start
ethereumProvider.on('connect', handler)

// HANDLER return
null

// react on connection end
ethereumProvider.on('end', handler)

// HANDLER return
Error Object with message (timeout, error, drop, etc)

[danfinlay]

Yeah I'm basically fine with that.

The bonus method that @kumavis mentioned that I'd support is if the RPCs also had an availableMethods endpoint, that could maybe return something like ethjs-schema returns.

But that could be its own proposal. In the meanwhile this is a nice low level provider api.

[frozeman]

The problem with availableMethods is that this would make the low level eth-api not useable until those are checked. As well as nodes under the hood can change to different ones, which would make a recall of this function necessary.

This is not so an issue for metamask, but in mist users can switch nodes easily.

[VictorTaelin]

Minor pick but shouldn't it be "disconnect" rather than "end"?

[VictorTaelin]

Also, just to be very honest: I've been thinking a lot about this, and, while I'm fine with this approach and will not propose to change it, I'm also quite convinced it is the wrong long-term solution. Let me elaborate.

Ethereum is a smart-contract network. It is a state machine capable of processing arbitrary transactions that update its state accourding to a transaction function. That state machine is kept alive by users all around the globe, making it decentralized. Due to its turing-completeness, it is able to simulate other state machines inside itself. A contract is just a state-machine with its own transaction function. A DApp is just a view for a contract's state. So, what a DApp needs to be fully functional?

1. A way to view the contract state. (read)

2. A way to submit a transaction. (write)

And that's why inejcting Web3 sucked: it included much more things, and we couldn't replace any of them without breaking several DApps. It seems that we realized this problem and our response was to build ethereumProvider, but I think it is giving us a false sense of "low-level". Problem is, while ethreumProvider is a small lib, it is actually a door to 2 big high-level libs. Any DApp using it still depends on the entire RPC and subscription APIs. If, for example, eth_getFilterLogs get removed, any DApps using it will break! Now, 95% of the RPC methods and the SUB APIs are not essential. If you're making a decentralized Youtube, why would you mess with uncle blocks?

eth.getBalance(Word160 address) => Promise Bytes
eth.getStoragetAt(Word160 address, Word256 index) => Promise Word256
eth.getLogs(Word256 fromBlock, Word256 toBlock, Array<Word160> address, Array<Bytes> topics) => Promise Array<Triple<Word160, Array<Word256>, Array<Word256>>>
eth.sendRawTransaction(Bytes signedTransaction) => Promise ()
eth.sign(Bytes message) => Promise Bytes
eth.getAccount() => Promise Bytes160

The API above, for example, would allow a DApp to assume it can do the essential stuff: reading the state and sending transactions. And that is it. Can you think in any product that couldn't be decentralized with that? Youtube, Reddit, Uber, Patreon? All look doable. Note this only defines what a DApp can do; the underlying implementation could vary a lot for performance. A getLogs implementation could use subscriptions, take care of reorgs and keep the requested logs cached in memory to promptly give it whenever queried.

Note: I'm not proposing we actually do this. That'd be a bad idea given the current state of things, and the difference wouldn't big anyway (and of course I could be wrong). In any case, I just felt like pointing ethereumProvider as a direct bridge to RPC and subscriptions will force us to support all the existing APIs forever, and that could make us less capable of making certain changes.

Edited: made it much more terse

[danfinlay]

First off, I really like that intro and I like where you're coming from and I'm sorry you think you won't change any minds. Your proposal looks good, but I do think I see the one big difference.

Just to clarify, your primary difference in opinion is that you don't think subscriptions will be more performant than polling. I think depending on the transports involved in the different implementations, it could actually make a big difference. Supporting an evented API allows the provider to optimize subscribing logic, wherever that logic is occurring, allowing the client-side JS to focus on rendering data as it's available.

Can you expand on why you don't think subscriptions add performance?

[VictorTaelin]

No that is not true at all, subscriptions absolutely add performance, we just don't need to expose them to the DApps. Where did you get the impression I said that? In fact a major part of the idea requires subscribing to a stream of storage diffs, so eth can always have an up-to-date copy of your contract's state in memory, so that whenever your app calls getStorageAt (probably several times a second) it just fetches from memory rather than sending an request over the wire.

[frozeman]

Just had a lengthy talk with @MaiaVictor and i think his idea is not well communicated. So Let me try to quickly explain.

The idea is only to expose a bare bone API which would work for ANY decentralised state machine. So we don't rely on ethereum specifics, like the protocol based on blocks/uncles or even the tx signing, but make it generic.

E.g. a API like this could be enough to cover almost all cases.
Then the provider could take care of re-running (reactive?) those functions when the state changes.

eth.getBalance
eth.getStoragetAt
eth.getCode

eth.call -> reactive
eth.sendTransaction

eth.sign
eth.getAccounts

eth.sendSignedTransaction ? to ethereum specific already?
eth.getTransactionCount ?
eth.gasPrice ?

To keep your dapp up to "state" ( 😄 ) people fire logs from their contracts, and the re-call contract functions to see if things changed.

This could certainly be made simple with reactive functions etc.

This is also one reason why we added the subscription, because we can in the future have things like:

web3.eth.subscribe('storageChange', '0x123456789',  function(result){
     /// result = {storageIndex: 2, previous: '0x2345..', current: '0x576432'}
})

But given a lengthy discussion, a bare bone but "complete" API is not yet possible IMO. It also would require many UX improvements in Mist and MetaMask to e.g. make sendTransaction return nothing (as the Mist/MetaMask UI would always take care of showing the pending status), etc.

So i would stay with the current proposed API and evolve such tools (like web3.js) on top of that, so that we one day can come up with a small, but generic API which fits all dapp needs!

So can we all please agree on the proposed API in comment: https://github.com/ethereum/interfaces/issues/16#issuecomment-299803228

So we can move on?

[VictorTaelin]

@frozeman I edited it, hopefully it is more clear now.

Just in case it is not clear, I'm all for the proposed API above and would appreciate if we could all just agree and mode on. I do think it will have future issues, but there is no pragmatic alternative, all given.

[danfinlay]

That API looks good to me, and if Fabian wants to get going, I think it addresses the biggest things we're aware of.

Funny to see us balancing "iteration speed" and "planning a non-breaking API", since those seem so dramatically opposed.

I don't really think we were remotely approaching "block size" disagreement level, btw. I appreciate everyone's willingness to chip in their ideas.

By the way, I really like the subscribe('storageChange', address, callback) pattern, but for clients that don't have a local VM, often they don't know the address of some storage they're requesting, which is why most UIs today use eth_call polling for those kinds of data queries.

Do you think a storageChange event exposes enough to build a non-VM-loaded ui on top of, or do we maybe need a way of abstracting the VM work, like eth.subscribe('call', params, callback)?

In any case, that could be added in a non-breaking way, so it's not a blocking question at all, you have my general support for this proposal.

[tcoulter]

My stance:

1. I agree with many of @MaiaVictor's statements:

   I'm also quite convinced it is the wrong long-term solution

   subscriptions absolutely add performance, we just don't need to expose them to the DApps.

2. I'm very against imposing an API on all ethereum providers that force them to expose an evented interface for transport state changes. I think it's short sighted - websockets are only one of the many transport types Ethereum should (and does) support.

3. I'm against the notion that because go-ethereum is switching to websockets, everything should switch to websockets. (That's how I've interpreted some of the above; definitely correct me if I'm wrong.)

4. I definitely don't think this is our block size debate, and I think this discussion is helpful.

As an exercise before anything is decided, we should take it upon ourselves to build multiple Ethereum providers, at least as proof of concepts, that support different transport types or no transport at all. For instance, at the very least we should be able to build the abstract-leveldown of Ethereum providers, and use that to build an in-memory provider, perhaps one that returns default data for testing. This should be our baseline. Building providers should be easy, and if we change the spec we should maintain that ease of use.

In the last two years I've created or had my hands in these five providers:

• truffle-hdwallet-provider - intercepts transactions and signs them based on an hd wallet mnemonic
• web3-provider-engine - provider that composes many different providers; everything from processing eth_call's in memory, to caching, transaction interception, etc.
• ethereumjs-testrpc - a whole Ethereum blockchain in memory, wrapped as a provider
• blockapps-web3 - a provider that converts the RPC calls over to BlockApps REST API.
• hooked-web3-provider - the first incantation of intercepting provider methods, precursor to the provider-engine.

The provider interface is the interface for interacting with Ethereum, no matter if you're using go-ethereum or not. Go's websocket interface will only be one type of provider, and as the ecosystem grows there will only be more. My vote, if there is voting, is on the simplest method possible, which is @MaiaVictor's ethereumProvider() method.

--

Two asides:

1. I recently talked to a big software company I can't disclose who will use Ethereum in their large blockchain offering, and interfacing with it will be through a custom provider. It'll have to be, since they're wrapping up Ethereum requests in a larger request, which will have to be done at the transport level.

2. Here are some other providers out in the wild. Note how they're all different and address multiple concerns in specific environments:

• https://github.com/JoinColony/hooked-web3-ddp-provider
• https://github.com/Upchain/upchain-web3-http-provider
• https://github.com/SilentCicero/ethdeploy-provider-zero-client
• https://github.com/ethjs/ethjs-provider-signer
• https://github.com/kumavis/web3-stream-provider
• https://github.com/Neufund/ledger-wallet-provider
• https://github.com/raineorshine/web3-fake-provider
• https://github.com/Neufund/server-wallet-provider
• https://github.com/jesuscript/web3-raw-provider
• https://github.com/wighawag/web3-html-provider
• https://github.com/kobigurk/web3-http-provider-basic-auth
• https://github.com/guenoledc/multi-http-provider

[danfinlay]

Adding events is not because of web sockets, it's because subscriptions are the correct abstraction for the UI's relationship to that data. Any transport could be used, or not used, this is about the JS provider and what it exposes.

[tcoulter]

I'm referring to things like this:

// react on connection start
ethereumProvider.on('connect', handler)

// react on connection end
ethereumProvider.on('end', handler)

// HANDLER return
Error Object with message (timeout, error, drop, etc)

Which I interpret to be transport-level data. Correct me if I'm wrong. For instance, if the websocket connection drops out, end is called no? Conversely, do http providers need to fire a connect event on initialization?

[frozeman]

The main reason why we proposed the subscription API is that the filter API is a mess and not extendable. Subscriptions make that clear again and allow it to extend in the future like with storageChanges or callChange types. Also having Push is just natural in the coming web. I also don't think that my proposed API is WS or IPC specific, but works for any duplex connection. The on(... is also only for subscriptions and connection events. The responses come through ethereumProvider.send(methodName, params, callback).

btw the subscription API is not only geth, but parity has it too already i think.

The connection events are necessary as subscriptions are based on sockets, and if the sockets go down the subscriptions cancel. A dapp, or web3.js needs to know when that happens so it can react on the following connect event later to re-subscribe, and fetch missing information if necessary.

I also spoke already with bas to adda eth_pollSubscription, do allow HTTP support for those. As it looks they don't see it as a priority, but we can change that i guess. It would also be great to be able to regain subscriptions, and it will queue up notifications for a period of time, like 1min after drop of connection.

All that could be done with the proposed provider.

Not having push option, means we need to rely on polling for the next 100 years :)

[frozeman]

And to answer your question @tcoulter If the connection drops it fires the end event, if then the node reconnects, it calls connect

[tcoulter]

Like @MaiaVictor, I'm not suggesting we don't have subscriptions, or websockets, or duplex connections. I'm suggesting that information doesn't need to be exposed to the dapp unless requested. A polling-like API interface doesn't imply a polling connection.

That said, my main reason for posting the above wasn't to argue the API. "You can't fight the sea". Instead, my main point is this:

If we change the API, build the abstract-leveldown of the Ethereum provider. Make it just as easy to incorporate any storage mechanism and any transport type as it is to write a backend for levelup.

---

If the connection drops it fires the end event, if then the node reconnects, it calls connect

Right. What should non-duplex transport mechanisms do? When should they call connect? Should they?

[VictorTaelin]

My last point, though, is that even an ethereumProvider is illusorily small because it is just a door to an API which includes tons of very specific functions, some of which will be deprecated, some which already are. eth_getUncleByBlockHashAndIndex, eth_getBlockTransactionCountByNumber, db_putString, eth_getFilterChanges, eth_newPendingTransactionFilter, newHeads, syncing and so on. By injecting ethereumProvider we're indirectly promising support for all of that.

I think the real future-proof way to do it would be to identify the minimum set of operations required for a DApp to work, and expose that. Since DApps are just decentralized state machines, then we essentially just need sendTransaction, getStorageAt. So, that's what I'd expose. That is promising we support only be bare minimum we need to, making future changes even on the entire Ethereum protocol much less painful.

Now, again: that is 100% not practical right now for a ton of social and technical reasons. In practice I think we'll just start building on top of this ethereumProvider and, eventually, this debate will come again.

[frozeman]

Thanks @MaiaVictor

@tcoulter the HTTP connection would not call the connect event, as doesn't do IPC or WS when a dapp is loaded. It will only be called if it disconnects at some point. Its a necessity of the way subscriptions are implemented, unfortunately.

Those are also necessary to halt dapp operation, as clicking buttons etc would all fail.

HTTP would not fire those events, as it couldn't fire the connect event again.

I dont understand the level down api yet, but i prefer push rather than polling and would better make polling run inside the provider and expose push, even on http :)

[tcoulter]

@tcoulter the HTTP connection would not call the connect event, as doesn't do IPC or WS when a dapp is loaded. It will only be called if it disconnects at some point. Its a necessity of the way subscriptions are implemented, unfortunately.

Won't that break dapps that expect to respond to connect events? I assume many dapps will use the first connect event to fire off further initialization.

It has to call it as far as I can tell.

i prefer push rather than polling and would better make polling run inside the provider and expose push, even on http :)

You can't fight the sea. :) Let's make an abstract-ethereumprovider so all the glue is standard. (That's what I meant by saying let's make en Ethereum provider version of abstract-leveldown -- I don't actually mean integrating Ethereum with levelup/leveldown, though that's not a terrible idea in some cases.)

Edit: typos

[VictorTaelin]

@tcoulter I think I see what you mean. In practice, we'll probably build our DApps on top of as few assumptions as possible anyway, right? So, we'll probably have aDAppNetwork interface (what you call abstract-ethereumprovider, I guess) with just the bare ability to spawn and interact with decentralized services. We'd then build high-level libraries on top of that interface (not ethereumProvider as suggested here). Then we'd just make a DAppNetwork instance out of this ethereumProvider, and our DApp will be live on Ethereum. In case ethereumProvider changes (or even Ethereum itself is replaced!), our DApp just swaps that instance and updates. That's how I plan to do it, at least. (We'd still depend on EVM exactly as it is now, though, but that's an entirely separate issue.)

[frozeman]

You guys lost me completely. @tcoulter i would not suggest a dapp to use "connect" as a init function. Its only there to handle dis/reconnects from the underlying node and loose of subscriptions.

[VictorTaelin]

@frozeman wait, what happens if I try to call .send on anethereumProvider that uses ws under the hoods and is currently disconnected?

[danfinlay]

I imagine you'd get a descriptive error describing the disconnected state.

[frozeman]

Yes.

Additional reason for the "end", "connect" event, are network changes under the hood. Apps need to be aware of those things, and e.g. check the network type they arte on again, etc.

[kumavis]

heres a different take that would require the connect to happen first:

explicitly attempt to connect to ropsten (fail if not available):

const ropstenProvider = connectToEthereum({ networkId: 3 })

attempt to connect to default (decided by browser):

const ethProvider = connectToEthereum()

metamask is looking at simultaneous multichain support, so having dapps be able to specify what network they are trying to connect to is really useful