Half-node example

[donovanhide]

Hi,

apologies for this not really being an issue, but it might be an interesting exercise to replicate this example using your library packages:

https://raw.github.com/jgarzik/pynode/mini-node/node.py

I'd like to mostly just listen to the bitcoin network with a mind to writing a mining pool in go (obviously a new block solution would have to be submitted back...). There's a lot of code in btcd, and was wondering if you could show an example of a "barebones" approach like jgarzik has?

Also, is it fully the case that btcd does not currently support multiple peers? Do you advise a personal instance of bitcoind to connect to for the moment? Should the peer management be a separate library package?

Thanks for the great code! Have learnt a lot about Bitcoin reading through it!

Cheers, Donovan.

[davecgh]

Hello,

I'll try to address all your questions.

First, if all you want to do is listen to the bitcoin network, check out btcwire (https://github.com/conformal/btcwire). It implements the wire protocol and is fully documented with 100% test coverage.

However, keep in mind that there is much more to bitcoin than simply the network wire protocol. Of particular importance when you start to talk about mining are all of the chain rules implemented by btcchain (https://github.com/conformal/btcwire). That package is also fully documented and has high test coverage as well as passes all tests in the "official" block acceptance test tool.

In addition to the chain rules, from the standpoint of mining, there are more rules which should be considered such as only mining transactions which conform to a stricter set of rules than simply what is allowed into the block chain. This is because it's not economically feasible to mine super expensive transactions without high enough fees to compensate the miner. Imagine mining zero fee transactions that contain 10 ECDSA signatures. That would cost the miner more than they could make from mining the block to begin with.

With all of that said, I think what you're really after from the standpoint of a mining pool is code that is currently not in btcd yet, but is under active development (I'm about 85% done with it give or take). Namely, your mining pool would need access to a transaction pool that contains transactions which have already been proven to adhere to all of the chain rules (for example not being a double spend or paying out more than its inputs), and imposes some of the more strict rules (such as ensuring transactions are in "standard" form, don't contain too many signatures, etc). Those more strict rules also should be tweakable by the mining pool operator by modifying the code to either loosen or strengthen them as needed to make the mining profitable.

btcd will have such a transaction pool and will ultimately offer the ability to query it via JSON RPC to request work (which essentially consists of transactions from the memory pool which need to be included in a block plus a target block difficulty). Once that code is in, all the mining pool would need to do to request work is query btcd via JSON RPC using the btcjson package (https://github.com/conformal/btcjson) and divvy the work up amongst the nodes in the pool using whatever algorithms the pool deems most fit.

On multi-peer support, it is sort of in there, but it's mostly disabled because we don't have the sync code multi-peer aware yet. As a result, if you connect to multiple peers right now, you ultimately end up downloading the all of the blocks from every peer only to have them rejected by the chain processing code because they're duplicates which is extremely wasteful. This is currently being worked on as well (by another team member).

For the mini-node example, I'd have to look through and see what all it does, but I suspect a large part of it is just listening to the network and responding to messages. As mentioned above, btcwire handles all of the protocol stuff. You could review the "Message Interaction" section of the documentation for btcwire (http://godoc.org/github.com/conformal/btcwire) which describes how such a mini-node would need to communicate using btcwire.

[davecgh]

I'm going to close this as I believe everything has been addressed. If you have any further questions on the topic, feel free to comment on this issue.

[donovanhide]

Hi Dave, many thanks for the in-depth response and sorry for the delayed reply! It's interesting to look at how mining has panned out and what practices and tools pool operators have ended up using. I'm guessing most of them have a bitcoind instance and issue getblocktemplate jsonrpc calls at regular intervals and then distribute the work via the stratum protocol. What I find interesting with your collection of packages and daemon implementation is that it could be quite possible to create a "mining pool node".

By that I mean a single go binary that is attached to the network, receives new blocks and transactions without serialising them to disk, does not respond to messages requesting data, applies rules to incoming transactions (ignore zero fee transactions, prioritise high fee transactions) and produces work for miners to process via stratum. Advantages of this approach could be quicker responses to other pools finding new blocks, higher fee rewards, greater ability to interrogate transactions, possibility of choosing transactions based on destination (contract mining), etc.

Currently btcd has the peer and address management baked in. If it was a library instead, with hooks such as onBlock, onTx, onvalidTx, etc. it might be a bit more easy to write a daemon with an alternative purpose to replicating bitcoind. I suppose the challenge is that the lighter a codebase btcd itself is, the more easy it will be to write another daemon that does something other than validate transactions, maintain a wallet and answer queries from other nodes.

It could be argued that such a proposal is a sink in the network and doesn't really add anything to the strength of the network, although there would be few mining pool nodes required to satisfy the demands of the mining community.

Another idea I've been working on is just a listener which distributes the unconfirmed transactions via server sent events, so that you could get a rolling window of who's spending what right now. I know you can do this via jsonrpc polling of bitcoind, but it would be a lot nicer to avoid polling and send straight from a select in go code :-)

Just some ideas on alternative use cases. Would be interested to hear your thoughts and thanks again for the response and open source code!

Cheers, Donovan.

[davecgh]

Hey Donovan,

Your first paragraph is accurate as to how most mining works. They request work from the "chain and transaction manager" (this is almost entirely bitcoind currently, but this could be any other alternative full node such as btcd or other alternatives, but it is paramount that those alternative implementations follow the block chain rules exactly [even the bugs] as even the most minor difference can cause a chain fork).

The "hooks" like onBlock, onTx, onValidTx, etc are basically provided by btcwire and btcchain. For a quick example, I've pulled some example code from the documentation of btcwire and added a basic idea for how hooks could be built on top without too much hassle.

type OnVersionHook func(*MsgVersion) error
func RegisterOnVersionHook(hook OnVersionHook) {
    // Add it to a map/list/slice of hooks to invoke
}

// Reads and validates the next bitcoin message from conn using the
// protocol version pver and the bitcoin network btcnet.  The returns
// are a btcwire.Message, a []byte which contains the unmarshalled
// raw payload, and a possible error.
msg, rawPayload, err := btcwire.ReadMessage(conn, pver, btcnet)
if err != nil {
    // Log and handle the error
}

switch msg := msg.(type) {
case *btcwire.MsgVersion:
    // Loop through the registered hooks for "OnVersion" and invoke them
case *btcwire.MsgBlock:
    // Loop through the registered hooks for "OnBlock" and invoke them
// ....
}

Also, btcchain provides notifications when blocks are orphans, accepted, connected, and disconnected. Docs: http://godoc.org/github.com/conformal/btcchain#Notification

I'll address your other points in a follow up later tonight or tomorrow when I have some more time.

[davecgh]

Alright, now to finish up as I said in the last message.

I believe I understand where you're coming from in relation to your second paragraph. The challenge there is that in order to apply many of the rules to a transaction, you need the entire block chain. A transaction can spend coins from any block and a big part of transaction validation requires access to the referenced transaction to link up the inputs and outputs. I'm not sure how that would be possible without serialization (short of keeping the entire 9GB, and growing, chain in memory). However, looking at it from another angle, the SPV approach could provide some interesting properties where there is some implicit trust as long as you are getting your relayed transactions from a trusted full-node (which is fully validating them).

On the point of server events, I agree it is nice to be able to get a stream as opposed to polling. We have code on a branch for btcd right now that supports learning about chain switches (and could be extended to transactions) via web sockets, which means asynchronous notifications as opposed to polling. It is used in btcwallet (non of which is really usable yet since it's all on branches that are not yet public).

Good discussion.

[shazow]

Hi there, sorry to piggyback on this older discussion but I've been thinking about something similar (and toying with the btcd code trying to achieve it).

I'm trying to create a two-tier node system.

1. The first tier simply listens to all transactions gossip. No blockchain, no validation, no wallet, no state. All network gossip events are published to some pubsub channel.
2. The second tier is a full node with a blockchain and validation rules. When a relevant address gets touched, the second tier gets notified by the first tier and does all the Right Things to perform some operation if the state of the transaction is correct (or gets queued for short-term polling).

This would let us create a more real-time experience without resorting to aggressive polling with potentially thousands of monitored addresses.

I suspect @donovanhide's question is similar to mine: How do we make something like what I described in the first tier while leveraging conformal's excellent codebase as much as possible?

Seems to me the peer management and message passing code is all in btcd, but it's tightly coupled to the blockmanager which is superfluous in this case.

(Edit: Some changes for clarity.)

[shazow]

Right now looks like it's not possible to use parts of btcd as a library without forking it. Largely because,

1. Many of the components of btcd which would be nice to re-use are internally-namespaced (lowercased prefix). Example: newServer(...), btcdMain(...), etc.
2. Even those that are publically namespaced and designed to be reusable (e.g. NewAddrManager(...)) depend on a static module-level cfg configuration struct. This means it's not possible to configure it without being a member of the module and it's not possible to have multiple instances of components running with different configurations.

Are there any plans/interest in making btcd's pieces more reusable outside of the module?

[flammit]

Hey Shazow,

I can't speak for what the core devs are planning w.r.t. btcd refactoring, but FWIW, it's now possible to accomplish what you want in your second-tier node by just interfacing with a btcd via the Websocket API. You can register for live notifications that you'll need to maintain your state without having to deal with the heavy lifting of chain management/txn validation or needing to do any polling. Take a look at https://github.com/conformal/btcd/wiki/Websocket-JSON-RPC-API for more details. If you have any questions, stop by the IRC channel and I'm sure that I or someone else would be more than happy to help there.

[shazow]

@flammit Right, the second-tier node is not a problem. Could use any of the available bitcoin daemons to do that. :) The first-tier node is what I'm interested in solving, though. I think this level of architecture isolation would be valuable for all sorts of applications.

I hope to keep this long-form discussion going here, but I do plan to drop by the IRC channel at some point. :)

[davecgh]

@shazow

Unless I'm misunderstanding, your example of a first-tier node is easily possible via btcwire already. For example, while the following code is clearly quick and dirty, it should work to listen to all transaction inventory gossip after you set the address to a known node IP near the top of main. It took about 10 minutes to write:

package main

import (
    "github.com/btcsuite/btcd/wire"
    "log"
    "net"
)

func main() {
    pver := wire.ProtocolVersion
    btcnet := wire.MainNet
    addr := "nodeaddrhere:8333"

    log.Printf("Connecting to %s\n", addr)
    conn, err := net.Dial("tcp", addr)
    if err != nil {
        log.Fatal(err)
    }
    defer conn.Close()
    log.Printf("Connected to %s\n", addr)

    // Send initial version handshake.
    nonce, err := wire.RandomUint64()
    if err != nil {
        log.Fatal(err)
    }
    msgVersion, err := wire.NewMsgVersionFromConn(conn, nonce, 0)
    if err := wire.WriteMessage(conn, msgVersion, pver, btcnet); err != nil {
        log.Fatal(err)
    }
    if err := msgVersion.AddUserAgent("txlistener", "0.0.1"); err != nil {
        log.Fatal(err)
    }
    log.Printf("Version message sent to %s\n", addr)

    // Read the response version.
    msg, _, err := wire.ReadMessage(conn, pver, btcnet)
    if err != nil {
        log.Fatal(err)
    }
    vmsg, ok := msg.(*wire.MsgVersion)
    if !ok {
        log.Fatalf("Did not receive version message: %T", vmsg)
        return
    }
    // Negotiate protocol version.
    if uint32(vmsg.ProtocolVersion) < pver {
        pver = uint32(vmsg.ProtocolVersion)
    }
    log.Printf("Version response received from %s\n", addr)

    // Send verack.
    if err := wire.WriteMessage(conn, wire.NewMsgVerAck(), pver, btcnet); err != nil {
        log.Fatal(err)
    }

    // Listen for tx inv messages.
    for {
        msg, _, err := wire.ReadMessage(conn, pver, btcnet)
        if err != nil {
            log.Printf("Failed to read message: %v", err)
            continue
        }

        switch tmsg := msg.(type) {
        case *wire.MsgInv:
            for _, inv := range tmsg.InvList {
                if inv.Type == wire.InvTypeTx {
                    log.Printf("New tx inv: %v", inv.Hash)
                }
            }
            continue
        }
    }
}

[shazow]

@davecgh That's exactly what I was looking for when I stumbled on this thread, thank you! You should totally include a snippet like this as an example for btcwire. :)

I still wish that btcd components were more reusable, but I suspect that discussion is probably best left for another thread.

[davecgh]

The core bitcoin components are reusable packages - btcwire, btcscript, btcchain, btcjson, btcutil, btcec, etc. The internal architecture of btcd itself is also modular as you noted, but those components are specific to a full node implementation. Given they are specifically tailored to a full node, I'm not sure that they would be very beneficial. That said, I won't rule out the possibility.

Also, I wanted to reiterate what @flammit said above. It is quite rare that you actually want to deal with the raw transaction gossip as the way it does in the code I posted above. You will not know if those transactions are double spends or otherwise invalid. I suspect what you are really after is what @flammit discussed. You want real-time notifications of transactions that have already been proven correct. btcd provides asynchronous (no polling) websockets for this express purpose.

[shazow]

Ah you're right, I didn't realize that the WebSocket JSON RPC API actually has a full stream of all transactions going through the Bitcoin network. I assumed that it would be like Bitcoind's equivalent which only notified you of transactions relevant to your wallet. Given that btcd has no innate knowledge of a wallet, this makes total sense in retrospect.

You're also right that the core components are already very modular and reusable, that's one of the things I love about this codebase.

I do suspect there are still valuable reusable pieces inside stranded inside of btcd, too. AddrManager, for one. But I'm not super familiar with the codebase yet, so that's just an uninformed opinion.

[davecgh]

For reference, the btcd peer code has now been refactored into a separate package. It makes it much easier to do the type of things this issue is referring to.

[shazow]

:sparkles: :custard: :sparkles: