Defining non-standard RPC method for transaction list

[axic]

The official RPC doesn't (and probably will never) support a method to retrieve transaction list associated with an address.

I think it could make sense having a non-standard method for that. There are multiple ways to create that data:

• using a data provider like etherscan/etherchain/ethercamp
• hooking into the block polling
• building someone's own service for that
• testrpc should be able to provide it too

The reason having a fixed API is useful:

• client code needs to be written once
• current data providers can be used
• there's a skeleton to work with when creating someone's own private service

Suggestion: zeroclient_getTransactions which returns an array of transactions

Sample response: http://api.etherscan.io/api?module=account&action=txlist&address=0xde0b295669a9fd93d5f28d9ec85e40f4cb697bae&sort=asc

[danfinlay]

I like that approach. It makes the provider-engine into a sort of web3 middleware ecosystem, where you can plug extra APIs in that you’d like.

On Mar 9, 2016, at 3:37 PM, Alex Beregszaszi notifications@github.com wrote:

The official RPC doesn't (and probably will never) support a method to retrieve transaction list associated with an address.

I think it could make sense having a non-standard method for that. There are multiple ways to create that data:

using a data provider like etherscan/etherchain/ethercamp hooking into the block polling building someone's own service for that The reason having a fixed API is useful:

client code needs to be written once current data providers can be used there's a skeleton to work with when creating someone's own private service Suggestion: zeroclient_getTransactions which returns an array of transactions

Sample response: http://api.etherscan.io/api?module=account&action=txlist&address=0xde0b295669a9fd93d5f28d9ec85e40f4cb697bae&sort=asc http://api.etherscan.io/api?module=account&action=txlist&address=0xde0b295669a9fd93d5f28d9ec85e40f4cb697bae&sort=asc — Reply to this email directly or view it on GitHub https://github.com/MetaMask/provider-engine/issues/49.

[axic]

cc @kumavis @tcoulter @ryepdx

[kumavis]

yeah i like this, but we need a better rpc prefix there

[axic]

Some ideas: util, explorer, middleware, localeth

[kumavis]

dora

[kumavis]

I do like explorer_getTransactionsForSender explorer_getTransactionsForReceiver

[kumavis]

this should eventually go into an EIP/RFC

[axic]

explorer does sound nice!

Why the differentiation between getTransactionsForSender and ForReceiver?

[kumavis]

getTransactionsForSender: give me all txs that the provided address has sent getTransactionsForReceiver: give me all txs that the provided address has received (potentially including all messages triggered by a tx)

[axic]

@kumavis most block explorers don't make a distinction between that. You do that client side by comparing the from/to in each transaction.

[tcoulter]

From https://github.com/ethereumjs/testrpc/pull/45:

I'm worried about adding application specific functions that don't directly relate to the TestRPC. i.e., evm_snapshot and evm_revert exist to allow the TestRPC to execute unit tests faster by checkpointing and reverting state. However, we're now adding a function whose sole purpose is to make writing your application easier even though no real Ethereum client is likely to support it. I don't think this is within the scope of the TestRPC.

I think this feature is great for the provider engine as the provider engine is meant to provide features on top of what's available in your Ethereum client. But the TestRPC has a much narrower scope.

[tcoulter]

My thoughts on this have been evolving, and I've mostly come to this conclusion:

• This isn't a good feature for the TestRPC. You can see much of that discussion in https://github.com/ethereumjs/testrpc/pull/45.
• This is only a good use of the provider engine if the plan is for the RPC method to be accepted by the Ethereum Foundation (all Ethereum clients, etc.). Here's why:
  1. You're using the provider engine as your method of wrapping up two APIs, etherchain's and etherscan's. It'll work, but given the other issues below this seems out of place.
  2. This method isn't yet supported by any Ethereum client, which means it's not supported by Web3.
  3. To use this method, you'll have to call it directly through a lower level function (i.e., engine.sendAsync({method: "explorer_...", ...}) rather than a more direct API. Or else you could "monkey patch" a function into Web3, but that's hairy on its own.
  4. These services have nothing to do with Web3 or the Etheruem client, since you're not getting data from the client itself. You are instead accessing data from external third party services.

Because of the above reasons, it's far better to create a new npm module that abstracts out these two apis, similar to the pkgcloud module for services like Rackspace, AWS, etc. This module would be generally useful, rather than coupled to the provider engine, and it would have a more direct API since you wouldn't need to call it via engine.sendAsync() or through a custom function added to web3. Adding it to the provider engine is the easiest way of doing things, admittedly, but generally it's less elegant and less useful.

This is where I currently stand. I'd do a complete 180, however, if you told me we were going to lobby @frozeman, Jeff, Gavin, etc. to get the method added to Ethereum clients and to the JSON RPC API (not unheard of). I think this is very valuable functionality for a client to have, but if it's not standard and will never be we should make different development decisions.

[tcoulter]

Sorry - accidentally closed.

[axic]

A lengthy discussion over at geth about the same topic: https://github.com/ethereum/go-ethereum/issues/1897.

[kumavis]

@axic thanks for the link!

I have most of an rpc polyfill for address -> txs sent but its a bit heavy on the number of requests required to fulfill it (hundreds) -- but works on any existing archive node!

PS: doesnt work for address -> txs received tho : /

[axic]

@kumavis yeah, it takes a big toll on RPC :(

Experimented a bit too. I still think it would be a real addition to testrpc, albeit maybe should be somewhat hidden and only made available through provider-engine.

[danfinlay]

If there were an RPC-middleware service that were responsible for exposing and caching those sorts of responses, it should be wrappable around any RPC, be it geth or testrpc.

[axic]

@flyswatter well provider-engine is the middleware, isn't it? And it certainly can be implemented, although it should have a more persistent storage. Based on personal experience as well as from that other thread, it takes a lot of effort to build up that data, even if a node sees all transactions and not speaking about retroactively trying to retrieve all blocks.

[tcoulter]

FYI: Very happy to add eth_listTransactions to the TestRPC when geth implements it (and possibly before)

[danfinlay]

I was just thinking how since you might want to consume an RPC server (like testrpc), and also customize an RPC, there might be room for an RPC-centric wrapper for provider-engine, that allows intercepting and forwarding network requests the same way provider-engine does with javascript methods.

On Apr 26, 2016, at 11:59 AM, Alex Beregszaszi notifications@github.com wrote:

@flyswatter https://github.com/flyswatter well provider-engine is the middleware, isn't it? And it certainly can be implemented, although it should have a more persistent storage. Based on personal experience as well as from that other thread, it takes a lot of effort to build up that data, even if a node sees all transactions and not speaking about retroactively trying to retrieve all blocks.

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/MetaMask/provider-engine/issues/49#issuecomment-214850088

[axic]

@tcoulter well I do not see that coming along anytime soon. I fully understand your point of not misleading devs with a method in testrpc, which isn't available elsewhere.

Would it be possible to have a non-publicly exposed method in testrpc to keep this data, which could be somehow accessed by provider-engine?

[kumavis]

fwiw, the testrpc would handle the polyfill well