SRI Template Provider using ZMQ and RPC

[Sjors]

An alternative approach for implementing the Template Provider was suggested during review.

Rather than having Bitcoin Core accept incoming network connections, deal with Noise encryption and stratum messages, an external tool would do this. This tool would get its information from the node via ZMQ and RPC:

1. a new zmq publisher, e.g. -zmqpubtemplate, which publishes block templates as soon as they become available.
2. a new RPC updatetemplatepub to configure the template publisher, e.g. for setting the coinbase output data size
3. a new RPC submitsolution

The external tool should probably be part of SRI, e.g. by adding roles/template_provider.

However, in this scenario it may not make sense to have that role in the first place. It just needs to be code shared by the The Job Declarator Client and the Pool roles.

It has similar implications as the idea of making bitcoind a Job Declarator Client, discussed in https://github.com/stratum-mining/sv2-spec/discussions/85. Except in that scenario we had to keep the Template Provider role around because the pool needs it, whereas here that role could disappear entirely.

I could look into adding those two things in a new Bitcoin Core branch. But someone else would have to take on the SRI side.

There are a few downsides to this approach:

1. It limits the number of connected stratum clients to 1, since there's only one ZMQ template feed. That's probably fine for the standard suggested workflows.
2. It means users have to install two pieces of software. That's already the case, but it removes any prospect of avoiding that such as https://github.com/stratum-mining/sv2-spec/discussions/85.
3. It precludes the ability to make the template provider public facing, even with an external tool (though from the Bitcoin Core perspective, this is already very challenging to do safely)

The advantage is that it (significantly) decreases the complexity added to Bitcoin Core.

Not sure if we actually want to go this route, but we can the discuss the pros and cons here.

[Fi3]

Having the additional role make sense also in this scenario, for example to allow public facing TPs.

[Fi3]

Going this route was my original plan and then we/I ruled it out (I'm trying hard to remember why but I don't). Honestly to me seems that the advantages exceed the disadvantages. We don't even need to refactor the spec, and we will be much more flexible for example adding an extension that now need to be implemented in core will most likley be done in the TP (not core) role much faster

[Fi3]

@thebluematt what do you think?

[Fi3]

Another advantage of separeted TP is that the JDS don't need anymore to talk bitcoind rpc

[Sjors]

for example to allow public facing TPs

Having a public facing TP won't work in this design, or at least you'd have to pick one additional_coinbase_output_data_size number (like 4000) for all clients (probably not too bad).

I'm trying hard to remember why but I don't

Hopefully someone remembers...

[Sjors]

JDS don't need anymore to talk bitcoind rpc

I think it does. The protocol between the JDC and JDS wouldn't change, the latter still needs to check against its mempool.

JDC, JDS and the pool (unless we keep the TP role) will need RPC access.

[Sjors]

Is there (good) Rust support for ZMQ?

[Fi3]

JDS don't need anymore to talk bitcoind rpc

I think it does. The protocol between the JDC and JDS wouldn't change, the latter still needs to check against its mempool.

JDC, JDS and the pool (unless we keep the TP role) will need RPC access.

yep I mean with TP role. It will be easier to implement a JDS (I expect pools to implement custom JDS not using the SRI one). We can just add an extension that do what now we are doing with RPC in the JDS.

[Fi3]

Is there (good) Rust support for ZMQ?

It seems so https://crates.io/search?q=zeromq&sort=downloads

[TheBlueMatt]

It limits the number of connected stratum clients to 1, since there's only one ZMQ template feed. That's probably fine for the standard suggested workflows.

In principle its okay because you can always just run a translator from ZMQ to a multi-TP listener, but...

It means users have to install two pieces of software.

making people run two pieces of software really sucks. Its more crap to debug when it goes wrong, and in general getting people to run one piece of software is hard, two is nearly impossible. Historically any time we've thought about some contrib script in Bitcoin Core or written something for people to run its been run exactly never. If its more than a config flag on software people are already running its generally hard to get people to do it.

That's already the case, but it removes any prospect of avoiding that such as https://github.com/stratum-mining/sv2-spec/discussions/85.

IMO this is a major issue. Its a travesty that its not trivial to take an ASIC and point it at a Bitcoin Core node. Sure, Sv2 support on actual miners is a ways off, but I've heard from many people over the years that they have some home miner for heating or whatever and just want to solo mine, but they've tried two or three pieces of software and didn't get it to work. Instead, people end up paying a few-% fee to some "solo pool" which is ab absolutely absurd state - if people want to improve the decentralization in the Bitcoin network, why the hell don't we have software that lets them.

It precludes the ability to make the template provider public facing, even with an external tool

Not quite sure I understand why this is precluded? Or do you mean because it could be built into the proxy, which would preclude it? I think this is an issue and we should at least build a ZMQ -> template provider thing to support this.

(though from the Bitcoin Core perspective, this is already very challenging to do safely)

Why? Bitcoin Core is designed to accept connections from untrusted parties and interact with them. The TP protocol is almost entirely send, so supporting untrusted connections to it safely should be rather trivial?