Feasibility for replacing Bitcoinj with Bitcoin Core SPV

[ManfredKarrer]

Bitcoin Core SPV (Jonas Schnelli's branch) should have similar resource requirements like BitcoinJ. Investigate if that is true (downloaded data at first start up) and if it is feasible investigate best integration option. We could ship it as binary like we do with Tor and communicate via RPC. Investigate if RPC is best option and if it is fast enough (it is not very fast).

[ManfredKarrer]

Neutrino might be an interesting option as well: https://github.com/lightninglabs/neutrino

[chirhonul]

Yes, I'd suggest we look into Neutrino (BIP157/BIP158)-based setups instead of bloom filters (BIP37). BIP37 is generally seen among core devs as fundamentally incompatible with privacy, and also imposes disproportional cost on full nodes (DoS vector). The main drawback of the GCS-based light node protocol is that CPU cost can be higher for the client, which could make things slow for a mobile device, but as Bisq is a desktop that should not be an issue for us.

I'd be happy to contriute here, once I understand more how the bisq -> bitcoinJ integration is built.

[ManfredKarrer]

@chirhonul Are you familiar with the Jonas Schnelli's approach? https://github.com/bitcoin/bitcoin/pull/9483

We had a past tech session where i gave an overview about the Wallet/BitcoinJ part: https://www.youtube.com/watch?v=JcySykHuWIQ&index=4&list=PLFH5SztL5cYOtcg64PntHlbtLoiO3HAjB

[chirhonul]

Yes @ManfredKarrer, I have been following the bitcoin/bitcoin#9483 PR.

I will take a look at the tech session recording, thanks!

[chirhonul]

I've watched the video. It had some interesting details that I now understand better, thank you for linking it!

Here is one project that may be interesting, since you mentioned in the video that integrating Bitcoin Core with Java would be hard / impossible, at least for Android. The ABCore project achieves this, using the Android NDK (https://developer.android.com/ndk/) to have a thin Java layer wrapping the Bitcoin Core C++ codebase inside an Android .apk. It's possible to install it and download a (pruned) blockchain on an Android device. I have tried it, it takes a very long time to sync the chain but does work:

• https://github.com/greenaddress/abcore

Since Android Studio is based on IntelliJ and uses Gradle, it might not be so hard to use the same setup to compile and package up C++ code like Bitcoin Core and bundle it inside the Bisq binary, if the RPC latency would be significant.

I understand the desire to continue working with the BitcoinJ fork if possible and gradually improve the code in our fork or upstream (or a new project forming a permanent fork, if necessary). I am not familiar with BitcoinJ at all basically so that would take some warm-up period for me personally, but I do have a fairly good understanding of BIP44, BIP37, BIP157/BIP158 etc. I would be up for either working on improving our BitcoinJ fork, or planning for a migration towards client-side filtering, as well as thinking about what JVM-based lightweight client may be useful to the wider ecosystem beyond Bisq. We can maybe discuss details on short-term tasks I can start with as well as broader strategy in other channels.

[ManfredKarrer]

Ah thanks for the info regarding the ABCore project . Have not heard about that. Though as a full Bisq version for Mobile is not really planned we have lower requirements than most other projects.

We might be able to integrate Bitcoin Core the same way like we integrate the Tor binary (Bisq starts the shipped Tor binary as new process and talks via a TPC interface to it, for Bitcoin Core we could use RPC and/or ZeroMQ). I think that would give the good performance of a native app and we are always in sync with latest reference client.

If that approach does not work (if the SPV model of Bitcoin Core is not feasible regarding resource behaviour) we should maybe get in touch with Samurai Wallet and Mycelium developers. They share a lot our mindset regarding privacy and at least Samurai Wallet has already implemented SegWit.

I am in touch with a BitcoinJ developer who has signaled to be interested to work on Bisq after he has completed a current project (ends around August).

[oscarguindzberg]

I have investigated this subject in detail.

Super short summary: I found no good replacement for bitcoinj that we could use today. bip157/bip158 is where the industry is going, so it is worth keeping an eye on it. Contributing bip157/bip158 support to bitcoinj would be an interesting option to evaluate if someone decides to invest time on this.

In general light wallet design faces a dilemma: Privacy (prevent others knowing what addresses are mine) and Trust (don’t trust third parties) vs. Low resource consumption (storage, cpu, bandwith).

An ideal bitcoinj replacement for bisq should have these features:

• Light - does not requires huge amounts of bandwidth/cpu/disk (ie no bitcoin core).
• Hierarchical Deterministic/BIP32 support (good example: bitcoinj).
• Tor Support.
• Stable, production tested and actively supported.
• Censorship resistant and Privacy - don’t allow the server owner or MITM to know what txs are mine. Worse example: Coinbase. Bad examples: known-to-be-broken bloom filters/bip37/bitcoinj, electrum, copay. Good examples: bip157/bip158. Perfect case: bitcoin core.
• High-level API: e.g. bitcoinj's wallet class.
• Mobile support is not a requirement (bisq is just desktop)

Resources to understand bitcoin privacy conscious wallet start-of-the-art

• Jonas schnelli status of wallets june 2018 https://www.youtube.com/watch?v=XORDEX-RrAI&t=57m15s and https://bitcoin.jonasschnelli.ch/BOB_jonasschnelli_csatfow.pdf and https://en.bitnovosti.com/2018/07/06/jonas-schnelli-current-and-future-state-of-wallets
• Bitcoin as a Privacycoin: This Tech is Making Bitcoin More Private https://bitcoinmagazine.com/articles/bitcoin-privacycoin-tech-making-bitcoin-more-private
• Full Node Level Privacy Even For Mobile Wallets? Transaction Filtered Full Block Downloading Wallet https://medium.com/@nopara73/full-node-level-privacy-even-for-mobile-wallets-transaction-filtered-full-block-downloading-wallet-16ef1847c21
• Bitcoin Privacy Landscape In 2017 — Beyond Coin Mixing, General Guidelines And Research: https://hackernoon.com/bitcoin-privacy-landscape-in-2017-zero-to-hero-guidelines-and-research-a10d30f1e034
• Anonymous Bitcoin: https://medium.com/@nopara73/anonymous-bitcoin-1fae5d1e33b7

Some ideas to make a light wallet:

• Download block headers until wallet creation date and full blocks from wallet creation date.
• UTXO set commitments (root hash of UTXO set in the block header / coinbase tx).
  • You have to trust previous blocks follow consensus rules.
  • You can do full validation from that point on, if you download the rest of the blocks.
• Checkpoints: Have hardcoded checkpoints in the source code. Don’t download block headers before the checkpoints.
• Encrypt communication between the wallet and the server (BIP 151, electrum, etc).
• Pruning: remove old blocks (or data in general) as you already validated it and it is past the finality threshold (ie 100+ blocks).
• Client-side filtering: (BIP 157/158)

Now, the projects I evaluated...

Bitcoinj

• Benefits
  • Already implemented in Bisq.
  • Tor Support.
  • Stable, production tested.
  • Unconfirmed txs friendly.
  • High-level API: e.g. wallet class.
• problems
  • Bad performance for arbitrators. Once you get lots of keys and txs in your wallet things are very slow (probably similar performance to not using bloom filters at all).
  • Unmaintained: no release for 2,5 years
  • No segwit support.
  • Bloom filters problems
    • No privacy at all.
      • Confidential information is sent unencrypted over the internet, so it is subject to MITM attacks.
      • User addresses could be easily obtained by every blockchain analysis company.
      • See https://groups.google.com/forum/#!msg/bitcoinj/Ys13qkTwcNg/9qxnhwnkeoIJ , https://eprint.iacr.org/2014/763.pdf, and https://jonasnick.github.io/blog/2015/02/12/privacy-in-bitcoinj
    • Bitcoinj Bloom filter “seed” generation bug removes more privacy.
    • Server can omit txs.
    • Easy to DoS a bitcoin core node that serves bloom filter, so probably most bitcoin core nodes will deactivate support sometime in the future.

Jonas schinelli’s “hybrid full block / SPV mode” on bitcoin core

• Modify bitcoin core to download block headers until wallet creation date and full blocks from wallet creation date.
• https://bitcoincore.org/en/2017/03/31//prioritized-block-download-using-hybrid-spv/
• Downloading the “old” blocks is optional: Wallet could do it to avoid following a longer but consensus-breaking chain.
• Idea: add pruning support to require less storage.
• Benefits
  • Full node privacy.
  • It’s bitcoin core, so it avoids potential alternative client bugs.
  • Recover from existing wallet works faster and the user can start using the wallet (even if she is not sure the shown balance/tx belong to blocks with correct PoW but invalid consensus).
  • Also makes the process of new wallet creation much faster since just headers and no blocks will be downloaded
• Problems
  • Required resources are similar or equal to full bitcoin node.
  • Unfinished/Abandoned implementation.
• Unmerged PRs
  • Complete hybrid full block SPV mode https://github.com/bitcoin/bitcoin/pull/9483 (jan 2017)
    • New config param “spv”: When on, core should sync the headers and then request blocks starting from wallet creation date.
    • Overlaps/complements #10794
  • Add simple light-client mode (RPC only) https://github.com/bitcoin/bitcoin/pull/10794 (jul 2017)
    • new config param -autorequestblocks=0. When on, sync the headers, don’t download blocks.
    • External wallet (or internal wallet in the future) can request the download of specific blocks via RPC
  • Deprecated PRs.
    • Introduce auxiliary block requests https://github.com/bitcoin/bitcoin/pull/9171 (nov 2016). Closed in favor of #10794
    • Add Hybrid full block SPV mode https://github.com/bitcoin/bitcoin/pull/9076 (nov 2016). Closed in favor of #9483

Neutrino / bip157 / bip158

• https://github.com/bitcoin/bips/blob/master/bip-0157.mediawiki
• https://github.com/bitcoin/bips/blob/master/bip-0158.mediawiki
• How it works
  • Bitcoin full nodes provide a filter for each block.
  • Filter is created deterministically by the full node based on the block. The filter is the same filter for every wallet.
  • Light wallets check client-side whether the filter shows the block might contain a relavant tx for them, if it does, they download the full block.
• Built to be used by lightning network nodes
• Benefits
  • Similar privacy to a full node: Peers nor MITM can find out what addressed belong to the light wallet.
  • Much less bandwith/storage/cpu required compared to a full node.
• Problems
  • Far from production ready technology.
  • No solution for unconfirmed txs (light wallets should download every broadcasted tx or don’t be aware of unconfirmed txs at all)
  • More bandwith required compared to bloom filters or electrum:
    • Filter size is 20k per block, much bigger than once in a while bloom filter updates.
    • It has to download full blocks if a relevant tx might be in the block, no SPV proof usage.
  • To investigate: Are there problems with the false/positive rate?
    • Is one-size-fits-all false/positive rate a problem?
    • Clients will end up downloading full blocks that don’t contain relevant txs to the wallet. The more addresses the wallet has, the more unnecessary blocks it will download. Is this a non-problem for lightning network nodes that have just a few keys to monitor (lots of keys added and discarded frequently) but a real problem for normal wallets that have to monitor a lot of derived-long-time-ago-but-still-valid keys.
  • How to make sure the filter is valid?
    • There is no filter commitment yet. A soft fork to add a filter hash to the coinbase tx / block header is very far away (maybe in 2 years).
    • Currently there is a (weak?) mechanism to find out if a peer is providing fraudulent filters (ie connecting to several peers, requesting information from all of them and eventually downloading full blocks)
• Server side implementations:
  • btcd
    • https://github.com/btcsuite/btcd
    • full node written in golang
    • Reference server implementation for bip157 / bip158.
  • bitcoin core
    • Not implemented yet
    • Issue asking for bip157 support https://github.com/bitcoin/bitcoin/issues/11863
    • commit to start bip157 support in core by laanwj https://github.com/bitcoin/bitcoin/commit/ea263e1eb030530d3cfe76542c6322de3d1bfa31
    • bip158 support(not bip157) https://github.com/bitcoin/bitcoin/pull/12254
• Client side implementations
  • Neutrino
    • https://github.com/lightninglabs/neutrino
    • Reference client implementation for bip157 / bip158.
    • golang
    • lower level api compared to bitcoinj
      • no class similar to bitcoinj wallet (no coin selection, method to get balances, key derivation)
    • alpha stage
      • docs say “experimental” / “not recommended to use it with real money”
      • almost no user documentation
    • Created for the needs of the LND client
  • Wasabi includes a bip157 / bip158 client implementation (written in c#)
  • JVM
    • Acinq/eclair told they plan to port neutrino to scala/kotlin
    • Muun is slowly building a bitcoin lib in java and they plan to add bip157/158 support.
    • A Bisq contributor might add bip157 / bip158 client support to bitcoinj

Bitcoin scala implementations

• Neither of them is really mature nor feature complete
• https://github.com/ACINQ/bitcoin-lib
• https://github.com/bitcoin-s/bitcoin-s-core

Samurai wallet

• https://samouraiwallet.com/
• This is not a wallet API, but a real wallet focused on privacy.
• Built for mobile, not Bisq use case.
• Lots of interesting privacy features
  • Whirlpool
    • https://github.com/Samourai-Wallet/Whirlpool - port of Wasabi’s ZeroLink
  • Stonewall (replaces BIP126)
    • Wallet creates txs that look like coinjoin txs so it confuses chain analyzers.
    • https://support.samourai.io/article/64-what-is-stonewall
    • https://gist.github.com/SamouraiDev/4ced85a29996dd56781e2bf319b93aaf
    • https://blog.samouraiwallet.com/post/173544815052/full-bech32-support-introducing-boltzmann-and
  • boltzmann
    • Returns the entropy of the transaction which measures the linkability of inputs to outputs.
    • https://github.com/Samourai-Wallet/efficiency-score
    • https://github.com/Samourai-Wallet/boltzmann
    • https://gist.github.com/LaurentMT/e758767ca4038ac40aaf
    • https://gist.github.com/LaurentMT/d361bca6dc52868573a2
    • https://gist.github.com/LaurentMT/e8644d5bc903f02613c6
  • Creates change output to match destination address type (p2pkh, p2sh, p2wpkh)
  • Ricochet
    • Creates 5 txs making the tx dependency chain deeper. That outsmarts very basic chain analysis technics that just look 5 txs before.
  • PayNym/BIP47
    • Share a kind of xpub, others will add the “xpub” to their contact list and every time they send you a tx, it will be a new address just they and you know (requires a 1st tx setup). Others won’t be able to find out that tx is from that xpub
  • Uses its own server to get blockchain data (I guess they used blockchain.info in the past). It has the same privacy problems electrum, copay and mycelium have.
  • You can specify your own bitcoin core node, then uses RPC (but maybe that is just to relay txs)

Wasabi Wallet

• https://www.wasabiwallet.io/
• This is not a wallet API, but a real wallet focused on privacy.
• https://github.com/zkSNACKs/WalletWasabi
• https://zksnacks.com/
• Features
  • Bip 157/ 158 support
  • ZeroLink wallet fungibility framework: https://github.com/nopara73/ZeroLink/
  • Chaumian CoinJoin
• Backend:
  • Custom server that provides bip157/158 filters. I am Not sure how the wallet syncs the block headers. Not sure if there is only 1 server or anyone can run a server or how the wallet could defend against a malicious server
  • https://github.com/zkSNACKs/WalletWasabi/tree
• c# language
• author: Nopara73 (Adam Ficsor)

Wallets using centralized servers

• Most prominent examples: Copay, Mycelium, electrum.
• Benefits
  • Almost instant new wallet creation and seed recovery.
  • Lowest bandwidth/storage/cpu alternative available.
  • Stable, production tested and actively supported.
  • Encrypted Client-server communication.
• Problems
  • The server knows exactly my addresses.
  • The server can not inform users of incoming txs.
  • Users should trust the company server or servers run by random people they don’t know.
  • Personal Electrum Server could, in theory, be run by users, but it requires a local bitcoin core which kills the almost instant on and low resource consumption benefits.

[oscarguindzberg]

Btw, I had a quick look and discarded alternatives listed in https://bitcoin.org/en/choose-your-wallet :

• Bitcoin knots: it is a full node
• Green address: centralized
• mSIGNA: requires a full node
• ArcBit: centralized
• Armory: python. requires a full node
• Bither: java. Uses bloom filters. Uses a custom library written in java (https://github.com/bither/bitherj ), similar to bitcoinj, but not based on bitcoinj. Seems dated.

[ManfredKarrer]

Thanks @oscarguindzberg for the summary! A small note: Bisq uses it's own tor infrastructure so Tor support is not really needed (we don't use BitcoinJ's now abandoned Tor support which was also not supporting hidden services).

[Christewart]

@oscarguindzberg Great list. Me and my company maintain bitcoin-s and are planning on building a lite client with https://github.com/bitcoin-s/bitcoin-s-core. Currently we have rpc and core protocol functionality along with signing logic. Things that still need to be built is a stand alone wallet and p2p networking services. The goal is to allow this library to be used for a lite client for lightning network nodes, but I think it should satisfy bisqs requirements too.

[lontivero]

Just for the record: NBitcoin supports all these features and more. Client side compact filters and its p2p protocol (bip158 and bip157), all existing standard scripts, bech32 addresses, partially signed transactions (reviewing now), rpc client, rest client, schnorr signature (early impl), etc. There was a time when bitcoinj was ahead of NBitcoin but currently NBitcoin is far far away, features are implemented few months after core (sometimes nbitcoin implements bips before core, ex: bip158).

Sent from my Moto G(4) using FastHub

[ManfredKarrer]

@lontivero Ah thanks. A pity it's not in Java....

[schildbach]

FYI: Bitcoin Core plans to switch off server-side filtering by default. The change has already been merged and they plan to release it with the next major version. Node operators can still enable filtering manually.

The relevant thread is on the bitcoin-dev mailing list. Topic: "Bitcoin Core to disable Bloom-based Filtering by default"

[ManfredKarrer]

@schildbach Thanks for the info!

I expected that will happen some day and I can understand the reasoning behind it. Luckily we use our provided nodes so we an handle it, but shows that BIP 37 SPV mode reaches its final days....

[ManfredKarrer]

I agree to that statement: https://github.com/bitcoin/bitcoin/pull/16152#issuecomment-501695811

I think as public network will become infeasible and provided BTC nodes the only working option (beside local node or private node) moving to a federated electrum server model might be better in the mid term. Will also solve many performance issues and the difficulty to find experienced BitcoinJ devs.