Discussion about recent unacknowledged security vulnerability reports

Recently two vulnerability reports to the security AT bitcoincore.org mailing list went unacknowledged. I'm concerned that might be symptomatic of a problem that could prevent future reports from being properly addressed and I think it warrants discussion.

Preface

This bitcoin-core/meta repository was created for discussion of moderation issues. This isn't quite that, but it was recommended to me as possibly the best public place to discuss this concern. I'm happy to relocate discussion to a more appropriate venue at the maintainers' or moderators' request.
In the following description, I've tried not to identify specific individuals (except for myself). I think discussion should focus on policies and processes, not people.

Incidents

Note: all times are in my local timezone.

Incident 1:

(Day 0, late afternoon) Researcher privately emailed a vulnerability report to the security mailing list.
(Day 4, early afternoon) Researcher publicly disclosed the vulnerability.
Note: days 1 and 2 were on the weekend, so less than two full business days elapsed between the private report and the public disclosure.
A member of the security mailing list told me that they had planned to reply to the researcher but the researcher's public disclosure preempted that action.
When I learned through private discussion that the researcher had publicly disclosed the vulnerability so quickly, I confirmed the timeline with the researcher and then publicly criticized them. I also refused them credit for their discovery in a description I wrote of it and suggested a security-related priviledge of theirs be revoked. This led to my involvement in the second incident.

Although I think a response to a vulnerability report within this timeframe would have been nice, even if it was just an acknowledgment that the report was received, I don't find it a particular cause for concern. However, it does set the background for the subsequent incident.

Incident 2:

(Day 0, mid afternoon) Researcher privately emailed a vulnerability report to the security mailing list.
(Day 4, early morning) Researcher confirms out-of-band that the report was received by a member of the security mailing list.
(Day 9, early morning) Researcher emails me about the lack of response.
(Ibid.) I email the security mailing list suggesting a "prompt response" and noting that "if you don't reply either way (or promptly find a third way), [the researcher] will have justifiable grounds for criticizing the usefulness of privately reporting vulnerabilities to this list. That undermines the security of the project as a whole and the recent gains the project has made by disclosing old vulnerabilities."
(Ibid.) I receive an acknowledgment from a member of the security list that my email was received.
(Day 12, early morning) Researcher publicly discloses the vulnerability.
~~(Day 12, late morning) I contact a member of the security list and am informed that, as far as they know, nobody ever replied to the researcher.~~ [Edit: that wasn't actually the question I asked, so I misinterpreted the answer I received. See this reply.]

Some opinions:

Both vulnerability reports were concise, well written, and correctly described the default behavior of Bitcoin Core and plausible behavior of network participants. They are nothing like what I would describe as spam.
The researcher and several Bitcoin Core contributors previously disagreed about the importance of a class of concern, and the researcher appeared to be using these reports to highlight what they thought was a logical inconsistency in the thinking of the contributors.

Expected behavior

I expected a member of the security mailing list to read each report and send a simple reply, such as "Thank you for your report. We think this particular concern could benefit from public discussion. Could you please open an issue on bitcoin/bitcoin or discuss any required protocol changes on the Bitcoin-Dev mailing list".

If it required more than a few days for members of the security mailing list to reach a conclusion like that, they could emailed the researcher to acknowledge the report and request more time for consideration.

Concerns

Nothing in the researcher's original report clearly indicated that they had not received a reply from the security list. If I hadn't involved myself in that incident, I might also not know that they also didn't receive a reply (after a much longer time period) in the second incident.

This leaves me concerned that other researchers may not be receiving replies and have simply not spoken up about it.
Whether it's this one researcher not receiving a reply or it's multiple researchers not receiving replies, I think this project having a history of not replying to vulnerability reports creates a risk that future researchers will not disclose vulnerabilities. I find this especially disheartening when several contributors recently worked so hard to write up and disclose past vulnerabilities---in part to highlight the benefits to researchers of responsibly disclosing vulnerabilities to the security mailing list.

I privately mentioned this concern to the security mailing list on day 9 of the second incident and yet (as far as I can determine) still nobody replied to the researcher, which is what led me to create this public issue.
I can't find any information about what a security researcher should expect when they email the security mailing list or send a PGP-encrypted message directly to a developer. That's true of most free software projects I've worked with, so this is not a problem that's specific to Bitcoin Core, but I do think it's a problem. It seems unfair of this project to place expectations on researchers (e.g. private disclosure) without telling them what they can expect from Bitcoin Core.

Suggestions

One or two members of the security list should be responsible for promptly acknowledging the receipt of any report that isn't spam, insane, or inane.
The acknowledgment should mention who is the person-in-charge for the incident. It's that person's job to poll the rest of the team (and any necessary outside parties) for next actions, and it's their job to communicate with the reporter (if the reporter isn't CC'd on internal communication).
Obviously, there's a major benefit to keeping the security team small. However, once a vulnerability has been triaged as non-critical, a trusted contributor who isn't a member of the security team can be can be assigned as the person-in-change of that incident. If there are interpersonal problems between members of the security team and the reporter, it should be possible to find a person-in-charge who is either neutral or, at least, diplomatic.
Whatever response policy is currently in effect or is adopted in the future should be concisely described for posting on https://github.com/bitcoin/bitcoin/tree/master?tab=security-ov-file#readme and https://bitcoincore.org/en/contact/ .

I privately mentioned this concern to the security mailing list on day 9 of the second incident and yet (as far as I can determine) still nobody replied to the researcher, which is what led me to create this public issue.

Replies were sent to the researcher following your email to us, but we did not tell them to not disclose.

(Day 4, early morning) Researcher confirms out-of-band that the report was received by a member of the security mailing list.

I had considered that my out of band response to the reception of the report was sufficient and did not warrant further response via the list. That's my bad.

The security list has a couple of issues that does exacerbate this. The list is a forwarder which forwards everything, and also preserves the "From" field. Using "Reply" instead of "Reply All" means that responses will not also be sent to the list, so frequently if a response is sent, the other members of the list do not receive a copy of the response. This has often led to situations where it is not entirely clear that a response was sent. In this particular instance, I was under the impression that someone else had responded to the report on day 4 even though I had not received a reply from the list.

For myself, I had not seen the email until I was notified out of band. Since the list forwards everything, including spam, I have occasionally discovered that my email provider's spam filters have sent every email to the list to spam. I believe this is what happened in this instance.

I also want to note that the researcher in question has behaved (or perceived to behave) antagonistically towards some members of the project, including some who handle security reports, and this may have resulted in some people not wanting to engage with them at all.

Replies to @achow101

I privately mentioned this concern to the security mailing list on day 9 of the second incident and yet (as far as I can determine) still nobody replied to the researcher, which is what led me to create this public issue.

Replies were sent to the researcher following your email to us, but we did not tell them to not disclose.

Ah, I see! Sorry for my misunderstanding! I feel better about the situation already. I will shortly make some strikeout edits to my OP above.

The security list has a couple of issues that does exacerbate this. The list is a forwarder which forwards everything, and also preserves the "From" field. Using "Reply" instead of "Reply All" means that responses will not also be sent to the list, so frequently if a response is sent, the other members of the list do not receive a copy of the response. This has often led to situations where it is not entirely clear that a response was sent. In this particular instance, I was under the impression that someone else had responded to the report on day 4 even though I had not received a reply from the list.

Do you think this is something that warrants improving? For example, setting up a private issue tracker (ideally using libre software on a self-hosted server) that can create new issues from incoming email, forward them to the triage team, and allow them to be assigned and tracked from that point forward. I'm happy to try to find funding for that.

I also want to note that the researcher in question has behaved (or perceived to behave) antagonistically towards some members of the project, including some who handle security reports, and this may have resulted in some people not wanting to engage with them at all.

I totally understand that. However:

For a severe issue, I hope people would be able to temporarily put aside past problems until it can be resolved.
For a non-severe issue, I hope a system that assigns someone to communicate with the reporter and also follow up (potentially privately) with other members of the security team can overcome interpersonal problems.

Thanks for your reply!

The security list has a couple of issues that does exacerbate this. The list is a forwarder which forwards everything, and also preserves the "From" field. Using "Reply" instead of "Reply All" means that responses will not also be sent to the list, so frequently if a response is sent, the other members of the list do not receive a copy of the response. This has often led to situations where it is not entirely clear that a response was sent. In this particular instance, I was under the impression that someone else had responded to the report on day 4 even though I had not received a reply from the list.

This could be fixed if everyone is asked to send the email to one of the developers and keep security@bitcoincore.org in Cc.

Expressing myself as a security reporter credited with at least one public security issue affecting bitcoin core (CVE-2021-31876) and with echos of other reporters who have shared issues to the security list, and a bit of wider non-bitcoin infosec experience from kernel space, I think the initial characterization of Dave is fair, reasonable and neutral.

I also believe that achow101 further providance of information on the internal of the mailing list is very accurate from my experience, even if one can deplore that the administrative setup of the communication channel does not match in operational security what is done by other open-source projects of the same standing. Again, from my list interactions with achow101 and in consideration of its answer on the mailing list about incident 2, I think its reaction has been relatively reasonable.

On the security reseearcher behaving antagonastically towards some members of the project as said by achow101, I think we shall not forget (A) that it's a decentralized project with no BDFL so contributors are in competition to advance their technical ideas on merits, antagonism is part of the process and people shall be at their ease with this dynamic, as long it stays in the norms of courtesy and civility. (B) we have many stake-holders in the bitcoin core development process, some non-profits and some with commercial interests or even sometimes contributors doing this purely as a hobby. This is a reality that shall be noted, it can happen that moral or economic interests of some active contributors, which on other dimensions are fully technically competent, manifest themselves beyond what is admitted by open-source or professional ethic standards.

Generally about (B), if you're a open-source contributor to the bitcoin field, it's better to express your thinking in private politely to said people who you think are less then adhering to ethical standards, and give them time to inflexe or bring more hihglighting on such apparent conflict of interest. In the lack of correction, I think it's perfectly legitimate for a security researcher to start to act a bit more musculary to formalize things politely.

I've seen myself many times the circus of the LND implementation doing a malestrom of public relationships on all medias to advocate the robustness of their lightning software. While at the same time this implementation, of which the development has been mostly driven by a ill-managed VC-funded startup with a direct interest to delay the publication of security vulnerabilities.

All that said, I can only recommend if you're a security researcher in the bitcoin space and if you wish to minimize or compartimentalize all antagonomism that can arise from external interests while discussing a security issue with a vendor team they are at 2 least methods (a) use a pseudonymous and adopt has best as you can a neutral style of writing or (b) give ample timelines to the software vendor team to take position and if willing so implement and deploy a mitigation, timelines based on historical experiences and infosec practice.

On the additional remark of Dave and 1400000bytes, that we just have to "ask" to send email to one of the developers or the mailing list, I deeply think this is not as simple. The whole dynamics in infosecs relies more on some hacker ethos, rules of precedents and ethical guidelines, one does not ask for "trust". As of today, they're historical people on the security mailing list, or who used to be generally involved in bitcoin security fixes, that I do no trust and with whom I have no wish to share security sensitive information related to the base layer, or with wide ecosystem impact.

(And no level of public shaming or other threats will be likely very efficient to pressure security researchers which are acting under their own name. As a security researcher acting on the boundary between public and private on significant worldwide security risk, there is always the marginal case of being physically tortured black site fashion by a criminal organization or an intel agency to extract you security sensitive information -- a depleasant thought you have to become used to).

This is not as simple that putting in between other people to set aside interpersonal problems. If someone has committed hostile material actions towards you, including at period when vulnerabilities were explicitly under embargo, you're certainly not take the risk of share anything sensitive with same, especially if the full mitigation and public disclosure is more "cowboy" that you're initial expectations (and sometimes that happens as novel technical facts can lead to reconsider the level of severity of the issues).

That it is why ethics matters, and few months ago I explicitly open an GH issue to suggest that the security list members, or at least the default endpoint be all public (and ideally with a GPG key) with #29366. That way, a security researcher can at least peak security team member more clearly, and if there is a security handling policy that one wished to be respect (as it can happen) it's more easy to do so.

An even more concrete suggestion could be to remove the concept of a single mailing list itself, and just have every security team member maintaining their own email infrastructure with the email / key fingerprint committed in the git tree. It's more technical work, though I believe the more robust direction on the long-term.

You don't trust an anonymous mailing list endpoint, like you trust security team members that you can regularly meet at conferences, meetups, and other professional meetings with whom you can always discuss eyes in the eyes (-- and sometimes it's a benefit when vulnerabilities are really out-of-nowhere even if I deeply think all security mitigations proceeding should stay full asynchronous, at the very least to handle situation like pandemics).

I'll re-quote what I answered to gmax on the gist establishing the novel security policy of bitcoin core about what is my current appreciation of the people hanlding the bitcoin security mailing list:

"I was just under the impression than the current people handling the establishment of this policy and advocating for short-pace timeline in matters of full disclosures of old issues were acting as "spoiled childs" which would have inherit an old stock of bitcoin security-sensitive issues, without getting the scarces of experience to act with the same level of ethical care than their elders."

No level of magic open-source funding, whatever level of $$$ will buy you competent, knowledgeable, reactive and ethical security team members of a codebase where the funds at stake can be estimated to be bigger than the treasury of most of nation-states in the world. Especially, ethics this is not something you buy with money.

All that said, I'm fairly available to improve this front as if there is one lesson I've drawn from years of security research in the bitcoin space, this is more of a team work which sound smoothly on combining offensive and defensive researchers and contributors with a wide background, mostly converging on ethics and sharing a common history. Reasonable security process is not a one man job in its corner.

Without further effort and dedidcation of bitcoin core on this front, in my personal perspective for someone with my level of skills to go take out libbitcoin kernel, wrap it with a p2p / mempool rust stack and move on. A very time-consuming and uncertain endeavor, when all that time could be rather spent on analyzing new consensus changes or hunting cross-layer security issues.

On the additional remark of Dave and 1400000bytes, that we just have to "ask" to send email to one of the developers or the mailing list, I deeply think this is not as simple. The whole dynamics in infosecs relies more on some hacker ethos, rules of precedents and ethical guidelines, one does not ask for "trust". As of today, they're historical people on the security mailing list, or who used to be generally involved in bitcoin security fixes, that I do no trust and with whom I have no wish to share security sensitive information related to the base layer, or with wide ecosystem impact.

I think the solution is simple:

- To report security issues send an email to [security@bitcoincore.org](mailto:security@bitcoincore.org) (not for support).
+ To report security issues, send an email to one of the developers from this list and keep security@bitcoincore.org in Cc (not for support).

You can even report anonymously by using a throwaway email and mention everything in the disclosure.

Reading the 1400000bytes suggestion, again I'll re-iterate what I already said. "The whole dynamics in infosec relies more on some hacker ethos, rules of precedents and ethical guidelines, one does not ask for trust. As of today, they're historical people on the security mailing list, or who used to be generally involved in bitcoin security fixes, that I do not trust and with whom I have no wish to share security sensitive information related to the base layer, or with wide ecosystem impact".

The linux kernel have been following the enforcement of ethical guidelines in matters of hardware security isusses, where you have security issues shared among a wide number of operating systems (vanilla linux, microsoft windows, android) due to commonality of vectors of exploitations arising from weakness in modern CPUs (AMD, Intel, ARM). Notable e.g to avoid early leaks before all affected systems are reasonably patched. This is documented here

On the suggestion to share any issue to the email address socially associated with sipa and with the security mailing list, personally I don't trust sipa - at the time being - for the reasons as I said so on the bitcoin core issue (i.e 29366) suggesting to put in public all the default recipient of the sec list with a key fingerprint for each. There are still others recipients I trust on the bitcoin core security list, though ultimately my point is that all recipients should be in clear to have better accountability. As a security researcher, if you're finding high-impact issues you're not going to forward them to an internet rando.

The suggestion of using an anonymous throw-away email and put everything in the disclosure is a short-sighted view of how a security reporting, mitigation and full disclosure work. In my bitcoin or lighting experience, you have multiple exchanges as sometimes the attached test won't work on the exact commit of the source code shared by the recipients or on the same exact platform or whatever.

bitcoin-core / meta