Cryptocurrency middleware / abstraction layer?

[ghost]

This is a proposal for an open middleware / abstraction layer that any service (not just Hive) wishing to utilize multiple cryptocurrencies / value tokens could take advantage of.

Supporting multiple currencies in a single wallet is a royal pain, and here's why:

• Multiple full nodes running simultaneously is a fail for obvious reasons.
• SPV implementations are often incomplete, unmaintained or (usually) nonexistent.
• Supporting more than a few SPV nodes (probably) bloats the application.
• Juggling multiple SPV wallets in-app is (mostly) impractical.

I would like to explore the possibility of a kind of middleware layer that makes it possible to create one wallet that effectively supports any kind of currency or token, including:

• Bitcoin and derivatives such as Litecoin, Feathercoin, and Dogecoin
• Custom tokens created by Mastercoin, Counterparty, and Ethereum
• Whatever comes next

Let's say we create an extremely simple API covering a limited subset of features for each general implementation that we wish to support. How can we as wallet/app makers find out about these servers in the first place? Perhaps by using a kind of distributed tracker: a blockchain ("Trackerchain") specifically designed to track token servers and requisite APIs.

Pretend for a moment that I'm the maintainer of feathercoind. I've got a feathercoind server setup that is ready to accept connections. To register it into the new blockchain so it can be found by others, I create a transaction and embed into it the necessary data with the IP and short names of the token it supports. It is of course signed too, giving user (or the app) the power to validate the data as genuine -- or to reject it outright. An additional library or daemon (perhaps) on this end serves to do the de-abstraction of incoming data into something my local interface can work with.

On the wallet end, I simply pluck out of said Trackerchain, IPs of the servers that are offering what I want, and I request connections to them. Perhaps they offer RESTful APIs, perhaps some are running through Tor. I have no idea. But in essence I am suggesting creating an SPV-like client so limited in scope that a common API could be used for any token. Provided that the abstraction layer worked correctly, wallet software could/should begin working with new tokens as soon as they are registered, no client code changes required.

Is this possible to do? Is this possible to do securely? I should add that I am suggesting an approach as decentralized as I can possibly imagine, but my imagination may indeed be severely limited.

Please: This is not a place to argue the validity of so-called "altcoins". I predict that we are going to see 10,000+ new issuances by the end of this year, largely emerging from the metacoin technologies currently being rolled out. Of course most of these will be junk, but the "1% may not" thought process applies, and indeed we can benefit from any memeing coin with this approach.

CC @larskluge @mikehearn @kyledrake @petertodd @davidjohnstonceo

[tgerring]

Suggest to separate wallet and network components to accomplish this. Currently, the reliance on bitcoinj for both network and wallet functionality limits our ability to implement other coins without a full SPV implementation.

If Hive's wallet component knows and maintains the cryptographic secrets of each token, then abstracting the network interaction via software becomes an easier task such that the backend existing as SPV or third-party RESTful service matters much less.

[ghost]

Definitely assuming a separation. The concept goes even further, implying that what we run locally as a wallet has no idea about any network except how to download and read the Trackerchain, and of course deal with what is provided by the resulting APIs.

That said, that do you mean by "cryptographic secrets of each token", and can you elaborate on your thoughts about abstracting interaction via software?

[tgerring]

I've brought it up a few times before--Hive should handle the cryptographic secrets & signing/spending/decrypting (e.g. private keys for Bitcoin/Litecoin/PGP/Ethereum/etc.)

If we define & build a software programming interface such that we can use dependency injection to plug into different networks, then implementing an altcoin becomes as easy as writing a wrapper for the software library or API that exposes network functionality. We could optionally rely on the library to generate/manage keys.

Very basic, incomplete example:

Interface BlockchainNetwork {
  [Req] Attribute: NetworkName
  [Req] Method: GetTransactionHistory(Address)
  [Req] Method: BroadcastTransaction(SignedMessage, Destination = null)
  [Opt] Method: GetNextAddress(BranchDescriptor = null)
}

This Interface than can be wrapped around each library/API so we have a consistent way to interact with all the networks without the UI needing to know much about them.

[ghost]

I would just underline this:

"Provided that the abstraction layer worked correctly, wallet software could/should begin working with new tokens as soon as they are registered, no client code changes required."

Requiring client-side changes for each new token is not the solution that I am looking for; it will not help to launch what is new and very much resource-strapped. If it is absolutely the only way to do it, and we are sure of that, then so be it... But this other, "autodetect" method would be greatly preferred.

[mikehearn]

Abstractions like this only work if the things they're abstracting are very similar or identical. Most alt coins today are a waste of time and tweak only very basic things, like the inflation schedule, so from the perspective of a wallet that trusts a third party server to accurately represent system state they're all the same: you can treat their transactions just like Bitcoin transactions and it will all work.

But more interesting and useful alt coins do actually change the way transactions work, and at that point your abstraction would break. You wouldn't be able to auto-register such a coin because Hive wouldn't know how to sign things.

Forking bitcoinj for alt coins is not very hard: the dogecoin guys managed it, for example. They then made their own version of MultiBit based on it, along with a dogecoin specific wallet. Certainly they could make a DogeHive without any real effort beyond making new icons (and maybe code signing).

Executive summary: the only alt coins the proposed abstraction would work for are the ones nobody really cares about, and the interesting ones would still require modifications at the code level.

[ghost]

Mike,

You are highlighting technical benefits as if it makes a damn bit of difference to the majority of holders. Dogecoin, Auroracoin and others should already have shown that this argument is wearing pretty thin. It's the incentive structure (and frankly, the marketing) that matters, and that is where the valuation largely derives. So as long as it is possible to create a new coin and have the market say it's worth something, people will continue to do this. Personally I want to take advantage of it to promote certain wallet software, others may have other ideas, hence something open to everyone.

Clever way to drag us into the conversation I specifically asked not to have by the way. :-)

Let's just pretend that we're talking about Litecoin, Dogecoin and Bitcoin. Do you think this could be pulled off securely in anything close to the manner I'm suggesting? Would you have any suggestions to improve or otherwise harden the scheme?

[ghost]

Maybe I should clarify. I'm assuming that we're years away from the time when great advances like distributed contracts (as one example) have common use by everyday people, and that most of us will largely remain in the space of simply trading and/or holding tokens.

If indeed we are talking about creating transactions to move tokens between wallets as a "limited subset" of features supported by the proposed API, then I'm not sure why it couldn't be abstracted in such a manner, even between two distinctly different worlds. I grant you that the most interesting stuff would not be covered, but I think that's okay, because it still satisfies what will likely be the most common use case: Moving the bloody things around.

Vaguely happy to learn otherwise though. Go on... :-)

[mikehearn]

You could have centralised servers that "do something" when Bitcoin-style transactions are submitted, sure. But it wouldn't be a decentralised wallet or coin in that case.

Like I said, forking bitcoinj isn't very hard. The main reasons it has to be tweaked normally are different fee rules (which are hard to express via some kind of config file because they can change anything), and the different block headers/work algorithms. Your proposed abstraction would avoid the need for the latter changes by not being SPV, but fee rules would still be something that could trip you up.

[ghost]

Good point about the fee rules.

[mikehearn]

A bit of history: when I first wrote bitcoinj I created a NetworkParameters class. I intended it to be a single place where differences between coins could be centralised, so the library could support variants of Bitcoin if there was a need for them. Last year I added the same thing to the C++ side (CChainParams).

But this wasn't as useful as I'd hoped. When the alt coins did start to arrive, it turned out that each one tweaked something in a unique and slightly different way to the last. Sometimes these tweaks were not really predictable because they made no technical sense or were downright dangerous, so I hadn't abstracted them out into NetworkParameters. Other times it's just because they went in different directions and, well, they were different. In the end I figured, you know what, we have an API. That seems like a reasonable abstraction. If people want to they can just use git to maintain patches on top of that. Most apps wouldn't need changing.

So far it's worked out OK. If people are minting auto-coins by patching Bitcoin Core, they can certainly go ahead and patch bitcoinj as well, and it happened a lot already:

https://github.com/bitcoinj/bitcoinj/network/members

Some documentation on how to make the most commonly requested changes might help a lot.

[javgh]

If I understand the proposal correctly, it would essentially be a client, that can talk to Electrum-style servers and gets new addresses and meta-data for additional Electrum-style servers from the trackerchain. New coins would need to provide their own Electrum-style server for their coin and then register it in the trackerchain.

I guess that could probably work for all clones of bitcoind. Beyond that, I'm not sure if a lowest common denominator approach will not cripple those altcoins too much, that try to do novel things. I guess you might still be able to send around ether, but why are you buying ether, if not to feed it to one of those fancy turing-complete contracts along with some input data?

Regarding the trackerchain: Would that be a separate chain? If so, what incentives do miners have to keep it secure? Or is it sitting on top of the Bitcoin blockchain? Does the client need to be a Bitcoin (SPV) client, to read the tracker data or would there be another Meta-Electrum-style server, that you can ask for trackerchain data?

I think that bundling a number of SPV clients into one, as the more decentralized alternative to these Electrum-style servers, does not necessarily have to be too bloated. I don't know, how many different coins each individual user would be using. But I imagine that it would not be more than a handful. So if you have, let's say, 5x the space requirement of today's Hive wallet, then that doesn't necessarily seem like a deal breaker to me. It's true though, that you wouldn't have this "autodetect" feature then.

[ghost]

Very interesting Mike, thank you for sharing that. And might I add, quite an impressive number of forks already!

@javgh I presume that many people will be buying Ether to trade and to speculate. However, I would of course like to extend this idea into the metacoin world of Ethereum as well, if it is in any way possible.

Re: Trackerchain, I guess I imagined it as a separate chain, yes. Or it could be a metalayer on top of Bitcoin. Maybe that would be easier. I guess the client would need to be SPV too, so it's a good point.

Re: the SPV bundle, the point at this layer is not to decide for users, which tokens are legitimate or important, and which ones are not [1], hence Trackerchain and "autodetect". So I'm excluding the "bundled SPV" approach, unless we can also include all Counterparty, Ethereum, NXT, etc tokens in the fold, and somehow make that autodetect bit work too.

Is there a best of both worlds solution here?

[1] We can do this at the curated wallet layer (Hive, etc), if it really turns out to be the best idea.

[XavierHawk]

Hey guys totally 5 year old non tech end user here.

I had an idea to make a wallet controller rather than an actual wallet.

say you have Doge, BTC, LTC, AUR, and CREDITS...instead of having to shuffle between all those wallets to use them...you have a wallet controller with all the pass words, pass phrases, 2FA etc built in...and it just controlled what was spent, received, etc..

is this even possible? if so it would win the wallet wars....

[mackuba]

Hmm... that sounds theoretically possible, but I don't really feel qualified to plan how the architecture of this should look like...

One thing I would be worried about is the security of such solution. The SPV clients rely on only the full node having full data, but they're carefully implemented in such a way that they have just enough information to make sure the node isn't cheating them. Even a bitcoinjs-style wallet, which is even more lightweight than a bitcoinj-style wallet (since JS doesn't directly interact with the network at all), has at least all the most internal core algorithms implemented on the client side. So if the given altcoin's client implementation is completely under external control, that would be a potential security issue.

On the other hand, if some most critical parts of an altcoin (like private key generation and management) would be implemented client-side in the wallet, like in bitcoinjs, then that kind of defeats the purpose of this idea, because adding new altcoins would potentially require code changes in the client to support them, if they use even slightly different algorithms in some places.

Also, isn't a whole new blockchain an overkill in this case? Some simpler DHT-style distributed database could also work here. But again, I'm not an expert.

[mackuba]

Suggest to separate wallet and network components to accomplish this. Currently, the reliance on bitcoinj for both network and wallet functionality limits our ability to implement other coins without a full SPV implementation.

As I've said before, as far as I can tell, the "wallet" and "network" parts of bitcoinj are so interconnected that it would be hard to separate them as two different things; and it's not about badly written software, it's just that all the layers (https://code.google.com/p/bitcoinj/wiki/HowThingsFitTogether) interact with each other and it would be hard to define a clear dividing API between the two parts in such a way that either of the components could be easily replaced. @mikehearn, correct me if I'm wrong.

[mikehearn]

You can instantiate a Wallet without the network of course, but then it's your responsibility to fill it up with the information it needs (transactions, primarily). I don't think they're that tightly bound together, but so far nobody tried to replace the network part with something else, like direct server fetches.

Agree that a blockchain is the wrong tool for distributing NetworkParameters data.

If the vast majority of a coins users were relying on a central server to tell them the state of the network anyway, then you wouldn't have a decentralised currency anymore. You'd have a bank. A large number of untrusted servers being accessed by competing wallets is what makes Bitcoin decentralised, so I'm not sure there are any shortcuts.

Anyway, forking bitcoinj is easy peasy. If you want coingen.io to do a bitcoinj/MultiBit/Bitcoin Wallet build at the same time I'm sure Matt would take the money for it :)