Bitcoin Node Maintainer

cbeams commented 6 years ago

This @bisq-network/btcnode-maintainers role is responsible for maintaining the shared configuration for @bisq-network's federation of @bitcoin nodes, as hard-coded in https://github.com/bisq-network/exchange/blob/master/core/src/main/java/io/bisq/core/btc/BitcoinNodes.java.

This role should maintain a shared bitcoin.conf file in this repository, and work with @bisq-network/btcnode-operators to make sure they run the same configuration there.

This role is responsible for responding in a timely fashion to GitHub issues added to this repository, questions asked in the #bisq-btcnode channel, and to ensure that monitoring notifications in #alerts get handled in a timely fashion.

Docs: none, other than the above Team: @bisq-network/btcnode-maintainers Primary owner: @sqrrm Secondary: @wiz

sqrrm commented 6 years ago

2018.06 report

Took over role as maintainer halfway through the month. Nothing to report on nodes.

@ManfredKarrer @cbeams Regarding maintaining the issue, is there a way for me to update the initial issue https://github.com/bisq-network/roles/issues/66#issue-287266942 to keep it relevant?

cbeams commented 6 years ago

@sqrrm wrote:

Regarding maintaining the issue, is there a way for me to update the [issue description] to keep it relevant?

In short, yes, but it'll work a little differently than that. I'll have instructions for all role owners about this soon. Thanks.

cbeams commented 5 years ago

@sqrrm, regarding keeping the description for this role (and the Tor Relay Operator role at #72) up to date, I mentioned above that I'd have instructions for all role owners on this soon, but what I'd actually like to do, if you're willing, is to have you try out the instructions first and provide feedback about it before I send out a broader request to have everyone do the same.

I'll provide a little context here, so that hopefully everything makes sense, but in the end, it should be a fairly simple task, one that will probably take 30 minutes or perhaps an hour for most roles.

First it's a good idea to read through the new Roles doc at https://docs.bisq.network/roles.html if you haven't already. It provides (hopefully) everything that contributors need to know about how roles work.

The relevant section of that doc that I want to focus on here is the Docs section at https://docs.bisq.network/roles.html#docs:

To follow those instructions for this role, you'd put together a pull request against the bisq-docs repository that adds a btcnode.adoc file. That file would include:

an Introduction section that explains what the Bisq federation of Bitcoin Core nodes is for, how it helps protect user privacy, etc.
an Infrastructure section that mentions the bisq-btcnode repository and btcnode slack channel.
a Roles section that details the Duties, Rights, GH Issue and GH Team for both the btcnode Maintainer and Operator roles.
if appropriate, a Processes section that details any processes involved with btcnodes, e.g. upgrades.

The Proposals doc at https://docs.bisq.network/proposals.html provides an example for what such a doc should look like; please treat it as a template.

You'll also want to add an entry to index.adoc to make sure that your new btcnode.adoc doc is discoverable. For right now, you can just put it at the bottom of the list in the Contributor Docs section. I plan to revise the index page soon to better accommodate these kinds of docs.

One of the effects of having these docs is that the descriptions in role issues like this one become simple, uniform, and largely unchanging. There's little need to "keep them up to date", and that avoids permissions problems like the one you had where you were unable to edit this issue (because I was the one who created it). The description ends up being as simple as this:

Docs: https://docs.bisq.network/btcnode.adoc#maintainer
Team: @bisq-network/btcnode-maintainers
Primary owner: @sqrrm

So, please let me know if it works for you to do this, and I'll look out for your PR. Thanks!

sqrrm commented 5 years ago

@cbeams I'll try to get that done, hopefully by the end of this week. Will let you know.

sqrrm commented 5 years ago

@cbeams I tried writing this up but three times I gave up. It's in principle not a hard task but I find administrative tasks energy draining so I just push it as long as I can. This is probably not the case for everyone but for me I would waste days of energy on something that, as you say, should take 30 minutes for someone with the inclination. The issues I face are:

I don't enjoy writing documentation and thus lack motivation but I don't mind the technical details of the role
There is process that needs to be followed and learned but the benefit seems small for the energy spent
The need to learn adoc notation to write a reasonable document

I think it would be better to get someone to write these docs up in a minimal fashion for the maintainers to fix if need be. I suspect someone might actually enjoy doing this whereas I and perhaps others with a similar aversion to administrative tasks would be turned off and even avoid taking on roles that could otherwise be taken.

cbeams commented 5 years ago

Understood, @sqrrm, and it’s good feedback all the same. Thanks.

/cc @m52go

m52go commented 5 years ago

Yeah @sqrrm I don't blame you. I'm willing to write the doc if you're willing to provide the details.

But in order to start, I need basic details covering the required content. Let me know if that already exists somewhere.

Otherwise, we could proceed in 2 ways:

you produce quick written notes that cover the items @cbeams mentioned above. They don't need to be remotely well-formed or even readable (i.e., a "brain dump").
we have a quick 10-15 minute phone conversation where we talk through the content.

Let me know what you think, or if you've got any other suggestions.

sqrrm commented 5 years ago

2018.07 report

During the month there has been no noticeable events.

There is a new minor bitcoind version out but we have earlier decided to only try to move quickly to new major versions. There is also a question if we actually want to move to 0.17.x when it arrives, to be discussed separately when it's more relevant.

The process of moving to new role management and documentation has been slow, partially due to me and partially due to the process. I will try to get this done for the bitcoin nodes next month with the help of @m52go That should work as a template for the other roles.

sqrrm commented 5 years ago

2018.08 report

Nothing noticeable to report.

sqrrm commented 5 years ago

2018.09 report

This month there was a bug in bitcoind https://bitcoincore.org/en/2018/09/20/notice/ requiring urgent attention. All bisq bitcoin nodes were upgraded urgently.

sqrrm commented 5 years ago

2018.10 report

A new major version is out, 0.17.0. Some have upgraded and no issues have been seen so I think it's good if all node operators upgrade at this point.

sqrrm commented 5 years ago

2018.11 report

The documentation for bitcoin node is getting closer to completion, @m52go is working on it with my input.

There is a new minor bitcoind version, no need to upgrade but those that want to can do it.

bisq-network/compensation#173

sqrrm commented 5 years ago

2018.12 report

Bitcoin Core 0.17.1 has been released but I see no urgent need for nodes on 0.17.0+ to upgrade.

Nothing more to report for December

https://github.com/bisq-network/compensation/issues/190

KanoczTomas commented 5 years ago

I see the helpwanted tag on this role, is this still the case? If so what is needed?

sqrrm commented 5 years ago

@KanoczTomas What's needed is the secondary role as node maintainer. I'm not exactly sure what the secondary role entails, but something like a backup for the primary I guess. There is a better role description in the works, don't think it's published yet but it's almost done I think.

sqrrm commented 5 years ago

2019.01 report

Nothing to report for January

https://github.com/bisq-network/compensation/issues/213

KanoczTomas commented 5 years ago

@sqrrm ok I will wait for the docs. I manage several bitcoin nodes and statically build core then distribute it to my nodes. Maybe I might help with something in the future. We will see.

sqrrm commented 5 years ago

@KanoczTomas This is the role as maintainer which is about keeping up to date with node software, config and make sure that the nodes that we run are adequate. The doc is actually ready but not linked, see https://github.com/bisq-network/bisq-docs/blob/master/btcnode.adoc#maintainer

To run a node there is a role for that as well https://github.com/bisq-network/roles/issues/67

Since you build and run nodes you probably keep up to date with bitcoin core developments and taking over the secondary role from @ManfredKarrer as maintainer might be appropriate.

ManfredKarrer commented 5 years ago

@KanoczTomas @sqrrm Would be great if you could take over secondary role. Thanks!

KanoczTomas commented 5 years ago

@sqrrm @ManfredKarrer I can take the secondary role, what I lack from the document is how is the proper version of bitcoind shared and how is the confederation bitcoin nodes status "looked over" by this role.

KanoczTomas commented 5 years ago

I suggest adding the minimal version to https://github.com/bisq-network/bisq-btcnode (like minimal_version.txt ot whatever) ... any thoughts on this?

sqrrm commented 5 years ago

@KanoczTomas We currently don't have a strict policy on how the nodes are run. The main idea has been that the operators upgrade to the latest main version reasonably fast. The minor version are optional unless otherwise discussed. So for 0.16.3 it was obviously urgent that everyone upgraded.

How the operators get their binaries and deploy them has not been discussed but I welcome a minimal doc. Anything you add will likely be an addition to what we're currently doing.

There is a monitoring tool to check on node status at http://seedmonitor.0-2-1.net/

sqrrm commented 5 years ago

On CVE-2018–20587

A summary can be found at https://medium.com/@lukedashjr/cve-2018-20587-advisory-and-full-disclosure-a3105551e78b

I don't think any action is needed from our side as long as the nodes are running in a safe environment with only one user. It's up to node operators to handle their individual op sec. Perhaps we could add some recommendations on this to make sure everyone is more aware? @Emzy @KanoczTomas ?

KanoczTomas commented 5 years ago

@sqrrm I agree, besides the node operator should be running the node without a wallet (disablewallet=1, or disabling it during compilation). This would lover financial loss risk. To mitigate for dual stack issues operators that do not use ipv6 should disable the complete stack, hence bitcoind will not try to bind to it. We could recommend having bind and rpcallow only to localhost. I can create a document with best practices for bitcoin node operators, will send for review once ready.

Emzy commented 5 years ago

I agree, there is no direct action needed. Best practice would be to have only the seed node + bitcoind on the server running. You could check if there is something else listening on the bitcoind rpc port (8332) before staring the bitcoind.

netstat -lpnt | grep 8332 The output should be empty if bitcoind isn't running.

sqrrm commented 5 years ago

@KanoczTomas @Emzy Great, let's add a section to the doc on some reasonable best practices on how to run a node. I think we can update the bitcoin.conf template and add a build section to https://github.com/bisq-network/bisq-docs/blob/master/btcnode.adoc as well.

sqrrm commented 5 years ago

2019.02 report

CVE-2018–20587 reported, decided that no action was needed. @KanoczTomas planning on taking over as secondary node maintainer

https://github.com/bisq-network/compensation/issues/228#issue-415543390

KanoczTomas commented 5 years ago

@sqrrm I can take the secondary role from March 1.

ManfredKarrer commented 5 years ago

Thanks @KanoczTomas ! I added you as secondary.

sqrrm commented 5 years ago

2019.03 report

Nothing to report for March

https://github.com/bisq-network/compensation/issues/253

KanoczTomas commented 5 years ago

2019.03 report

Although not Bisq related directly, v0.18.0rc2 was tagged. I recommend doing a gitian build by members of the Bisq community to further strengthen the trust in source->binary translation. As I believe most of node operators run binaries from bitcoin.org/bitcoincore.org.

I think at least node maintainers should have their signitures in gitian, what do you think @Emzy @ManfredKarrer @sqrrm ? I created a pull requested to bitcoin-core (not merged yet) https://github.com/bitcoin-core/docs/pull/47 as the docs are bit inconsistent.

Doing a docker gitian build is the easiest, reach out to me if you need help with it. Rc3 should be soon tagged.

\cc https://github.com/bisq-network/compensation/issues/237

ManfredKarrer commented 5 years ago

@KanoczTomas Basically good idea, but I fear I will not have time for that. Too much other high prio stuff....

ghost commented 5 years ago

@KanoczTomas wrote:

should have their signitures in gitian

What does that mean Tomas ? Could you please recommend good doc on gitian/reproducible build. I'm also interested by this topic.

KanoczTomas commented 5 years ago

@HarryMacfinned you make the build, then it creates assert files for which you then create a pull request https://github.com/bitcoin-core/gitian.sigs/tree/master/0.18.0rc2-linux/KanoczTomas - the output that I committed. When you download the binary from bitcoin.org the hash has to be the same as in the assert file. You can go to the repository and check who built the binary, so you can make sure it is the same and other people independently built it. @ManfredKarrer the docker build is a run and leave running thing (it takes several hours to build) than uploading it to the repo. But I understand you will not have time. I will certainly do it for every release. @HarryMacfinned I will create a document with steps how to do a docker build, as the official one has some parts not working and docker is not explicitly documented there.

sqrrm commented 5 years ago

Version 0.18.0 is out. From what I see it's not urgent that we all upgrade but we should start upgrading as we see fit. Each node operator upgrading when it suits them makes for a staggered upgrade.

sqrrm commented 5 years ago

Cycle 1 report

There is a new bidcoind version out, node operators to upgrade as they see fit.

Regarding @KanoczTomas' suggestion that we add our signatures in gitian, that sounds reasonable to me. I'm not familiar with the process and will ask your help.

https://github.com/bisq-network/compensation/issues/282

sqrrm commented 5 years ago

Cycle 2 report

Nothing worth noting this month. I haven't done anything on gitian yet.

https://github.com/bisq-network/compensation/issues/293

sqrrm commented 5 years ago

Cycle 3 report

There was a note from Luke Dashjr on the mailing list about vulnerabilities being disclosed. We should all make sure to be on at least v0.17.1

Two relatively minor vulnerabilities will likely be disclosed sometime soon.

The first vulnerability, CVE-2017-18350, was introduced in v0.7.0 (released in 
2012 September), and affects all versions released until the fix was included 
in v0.15.1 (released in 2017 November). No versions prior to v0.15.1 are 
expected to be fixed.

The second vulnerability, CVE-2018-20586, was introduced in v0.12.0 (released 
in 2016 February), and affects all versions released until the fix was 
included in v0.17.1 (released in 2018 December). As of today, this fix has 
NOT been backported to older versions. When/if v0.15.3 and v0.16.4 are 
released, they may also include a fix, but due to the minor severity of this 
vulnerability, it does not merit a dedicated release on its own. (The git 
branches are also NOT fixed at this time.)

Please be sure you have upgraded to a fixed version no later than August 1st.

https://github.com/bisq-network/compensation/issues/311

sqrrm commented 4 years ago

Cycle 4 report

Nothing to report this month

bisq-network/compensation#325

sqrrm commented 4 years ago

Cycle 5 report

Nothing to report

https://github.com/bisq-network/compensation/issues/356

wiz commented 4 years ago

Since it seems @KanoczTomas is no longer actively contributing, if there is no objection I'd like to takeover the backup btcnode maintainer role

sqrrm commented 4 years ago

Cycle 6 report

@wiz has taken over as backup btc node maintainer as @KanoczTomas has not been available.

https://github.com/bisq-network/compensation/issues/375

wiz commented 4 years ago

Cycle 6 report

After receiving multiple reports from users that Bisq was not broadcasting some transactions to the Bitcoin network, I started investigating the BTCNodes, suspecting that some of them may not be relaying the transactions properly.

I hacked together a Bitcoin Service Check script for my Icinga2 based monitoring system to periodically check (once per minute) if the BTCNodes hard-coded into Bisq were operating properly. The monitoring system immediately found that several nodes had issues:

For @mrosseel's 2 nodes btc.vante.me and btc2.vante.me I found that the clearnet IP addresses were out of date, and updated them with the current DNS resolved IP addresses.

For @tbocek's node bitcoin4-fullnode.csg.uzh.ch I found it was unreliable at times, and removed it entirely.

For @KanoczTomas's node http://btc.ispol.sk I found it was unreliable at times, and got less reliable under heavy load, suggesting it was resource exhaustion or a rate limit. Since @KanoczTomas was unreachable, we temporarily disabled it using a network filter. However, since @KanoczTomas has since returned and fixed his node by removing the connection limit, we are planning to re-add it soon, after verifying it with the monitoring alerts system that it is stable.

/cc https://github.com/bisq-network/compensation/issues/380

wiz commented 4 years ago

Cycle 7 report

All btcnodes have been stable recently. This cycle 3 btcnode instances started running electrs and mempool-space for TX tracking and fee estimation on onion hosts, which will soon be integrated into Bisq as part of https://github.com/bisq-network/bisq/issues/1077

http://xxh4i4lbbrh6mzy2.onion/api/v1/fees/recommended @wiz http://22qaao7zikvzqehg.onion/api/v1/fees/recommended @Emzy http://hring5rmamttlpup.onion/api/v1/fees/recommended @devinbileck

/cc bisq-network/compensation#398

sqrrm commented 4 years ago

Cycle 7 report

@wiz has done good work on investigating tx issues and fee estimation.

An old issue was disclosed on the dev mailing list, showing the importance of running the latest node. It also highlights the trouble we might find ourselves in once SPV support is removed. If we then stay on old versions due to the SPV support we might be running nodes with known issues.

CVE-2017-18350 is a buffer overflow vulnerability which allows a malicious SOCKS proxy server to overwrite the program stack on systems with a signed char type (including common 32-bit and 64-bit x86 PCs).

The vulnerability was introduced in 60a87bce873ce1f76a80b7b8546e83a0cd4e07a5 (SOCKS5 support) and first released in Bitcoin Core v0.7.0rc1 in 2012 Aug 27. A fix was hidden in d90a00eabed0f3f1acea4834ad489484d0012372 ("Improve and document SOCKS code") released in v0.15.1, 2017 Nov 6.

To be vulnerable, the node must be configured to use such a malicious proxy in the first place. Note that using any proxy over an insecure network (such as the Internet) is potentially a vulnerability since the connection could be intercepted for such a purpose.

Upon a connection request from the node, the malicious proxy would respond with an acknowledgement of a different target domain name than the one requested. Normally this acknowledgement is entirely ignored, but if the length uses the high bit (ie, a length 128-255 inclusive), it will be interpreted by vulnerable versions as a negative number instead. When the negative number is passed to the recv() system call to read the domain name, it is converted back to an unsigned/positive number, but at a much wider size (typically 32-bit), resulting in an effectively infinite read into and beyond the 256-byte dummy stack buffer.

To fix this vulnerability, the dummy buffer was changed to an explicitly unsigned data type, avoiding the conversion to/from a negative number.

Credit goes to practicalswift (https://twitter.com/practicalswift) for discovering and providing the initial fix for the vulnerability, and Wladimir J. van der Laan for a disguised version of the fix as well as general cleanup to the at-risk code.

Timeline:

2012-04-01: Vulnerability introduced in PR #1141.

2012-05-08: Vulnerability merged to master git repository.

2012-08-27: Vulnerability published in v0.7.0rc1.

2012-09-17: Vulnerability released in v0.7.0. ...

2017-09-21: practicalswift discloses vulnerability to security team.

2017-09-23: Wladimir opens PR #11397 to quietly fix vulernability.

2017-09-27: Fix merged to master git repository.

2017-10-18: Fix merged to 0.15 git repository.

2017-11-04: Fix published in v0.15.1rc1.

2017-11-09: Fix released in v0.15.1. ...

2019-06-22: Vulnerability existence disclosed to bitcoin-dev ML.

2019-11-08: Vulnerability details disclosure to bitcoin-dev ML.

https://github.com/bisq-network/compensation/issues/403

sqrrm commented 4 years ago

Version 0.19.0.1 was released and the bloom filter was disabled by default. The flag peerbloomfilters=1 needs to be set to allow bitcoinj to work.

@wiz and @Emzy have already upgraded a node each, once confirmed to run properly we can all upgrade at a leisurely pace.

sqrrm commented 4 years ago

Cycle 8 report

New version was released, looks like it's running fine for those that have upgraded. Care need to be taken with the bloom filter.

https://github.com/bisq-network/compensation/issues/441

wiz commented 4 years ago

Cycle 8 report

So far Bitcoin v0.19.0.1 seems to be running with no issues, so I think we can ask the btcnode operators to upgrade. All the instances seem to be stable lately with no issues to report.

/cc https://github.com/bisq-network/compensation/issues/448

sqrrm commented 4 years ago

Cycle 9 report

Nothing much to report. The nodes seem stable and with the seednode installation script it's now possible to install a node in one command on a fresh ubuntu 18.04. @wiz Perhaps we should think of splitting that script into 2, bitcoin node/seednode or add an option to only install the bitcoin part.

https://github.com/bisq-network/compensation/issues/461

wiz commented 4 years ago

Cycle 9 report

/cc https://github.com/bisq-network/compensation/issues/470

bisq-network / roles