Potential stagnation of open issues on h1 bounty program

[phra]

Hello,

I've recently open few issues on the 3rd party modules h1 bounty program and I'm noticing a bit of delay in responses from the team, e.g. a bug that was fixed two months ago after I contacted the developer myself is still in triaged status without any response but h1 staff ones.

I understand the amount of effort involving managing the program itself but leaving open issues for a long time without any interaction can damage the whole initiative, especially when the only party not actively participating in the resolution are the program managers themselves.

Do you have any thoughts on how we can improve the bug hunters' experience by providing a smoother resolution and achieve more prompt reactions?

I maybe have a couple of suggestions already, that are:

1. predefining a standard flow/timeline that the staff can follow in order to contact the maintainer for a fix and/or disclosure of (un)patched issues in a reasonable amount of time? (like google project zero)
2. increase the number of people actively collaborating to the bounty program

I will be happy to hear from you what are your thoughts on this particular topic.

PS: I can eventually be available to help with the program in my spare time.

EDIT: regarding 1., I noticed that a process is already defined here.

[sam-github]

@phra Conversation is ongoing in https://github.com/nodejs/security-wg/issues/604 thta you may find relevant.

[phra]

thanks @sam-github

[lirantal]

I apologies for that @phra. We are way behind on many reports and have taken actionable steps previously too to ensure we prioritize correctly with the limited time folks here have but that's still hard.

We are welcoming others to join and have recently added a couple more but that's still not scalable with the amount of reports we're getting. I'm happy to hear any ideas you have on #604 as Sam mentioned.

[phra]

i have commented on #604

[mcollina]

I would let you know that the number of issues in the bounty program is too big and the situation is becoming untenable. There are weeks of full time work to exhaust that pipeline and no one who is going to do them.

[MarcinHoppe]

Should we stop accepting new reports until we decide on the new direction and act on that decision? I think we probably should.

[MarcinHoppe]

@nodejs/security-wg I'd love your opinion ☝️.

[mcollina]

Should we stop accepting new reports until we decide on the new direction and act on that decision? I think we probably should.

I think we should keep accepting them for projects/organizations that opted in.

[MarcinHoppe]

Fair point, I agree with that.

[DanielRuf]

I think we should keep accepting them for projects/organizations that opted in.

I agree with this.

Many of the current reports are probably the same findings in different packages and so far most maintainers did not react / answer after I have informed them (so far mostly the low priority bucket ones) so in my opinion this would be helpful to accept these which opted in.

[esarafianou]

I think we should keep accepting them for projects/organizations that opted in.

Where is a list of projects/organizations that opted in?

[phra]

Where is a list of projects/organizations that opted in?

@esarafianou i think they are referring to the repo/orgs list shown on Node.js third-party modules h1 page.

[MarcinHoppe]

We add assets for every new package we have a report for. This list does not represent packages where maintainers opted in to be part of the program.

[phra]

@MarcinHoppe ok, so is this list available to be reviewed? :smile:

a question regarding packages that instead were removed from the list (i have reports for packages that were in the list but aren't anymore): does it mean that the maintainers opted out or something else?

[MarcinHoppe]

The best link I think there is is this one:

https://github.com/nodejs/security-wg/blob/master/processes/bug_bounty_criteria.md#modules-list

@lirantal @vdeturckheim is this the correct list of packages that have the maintainer opt-in?

[lirantal]

This one that have confirmed: https://github.com/nodejs/security-wg/blob/master/processes/bug_bounty_criteria.md#confirmed

[esarafianou]

This list seems to be a year old. If we are to allow packages whose maintainers have opted-in, I think it's valuable to reach out to the maintainers once again to verify that they still wish to be in the program.

[mcollina]

I think the situation of the H1 program is getting severe. I've tagged this as tsc-agenda.

I strongly recommend to shut the program down.

[MarcinHoppe]

I agree, I think the fact that it is open sets the expectations we are currently not able to meet.

[MarcinHoppe]

@mcollina any feedback from the TSC?

[mhdawson]

Sorry meant to post this to the issue, from the minutes:

• Discussion, general support from those in the meeting that TSC members support shutting down the program.
• Myles can check if they have a replacement at GitHub for places people can report vulnerabilities.
• @Trott will reach out to some involved parties on the team to discuss winding it down.

[adam-nygate]

Hi all,

We'd be happy to support the program over at www.huntr.dev.

We're rebuilding our disclosure process at the moment, including support for private disclosure and incentivising security researchers and maintainers for securing open source code.

Perhaps I could introduce myself and what we're building at a future meeting?

[mhdawson]

@adam-nygate +1 to doing an introduction.

One question in advance. From the part on the web site on disclosing a vulnerability, it seems to say to open a pull request. Does that mean the vulnerability is immediately public?

[adam-nygate]

@adam-nygate +1 to doing an introduction.

Great @mhdawson, could you let me know where I can register to attend the next meeting?

One question in advance. From the part on the web site on disclosing a vulnerability, it seems to say to open a pull request. Does that mean the vulnerability is immediately public?

Yes, we built our current disclosure process pretty quickly and around GitHub Issues (which are public by default), we were hoping for GitHub to bring out private issues or to expand the functionality of their security advisory feature, but now our plans are to build out a new disclosure process that supports both full and coordinated disclosure, allowing the Maintainer of a project to choose.

[MarcinHoppe]

How does this compare to HackerOne?

[adam-nygate]

How does this compare to HackerOne?

HackerOne isn't really designed for open source, and so it applies processes designed for disclosure against enterprises to the open source ecosystem. An example of this is that HackerOne isn't designed for the code to be in view of the attacker, nor for them to be able to collaborate (or even entirely develop) the fix. We're designing our processes entirely around open source, encouraging security researchers not only to find vulnerabilities but also to fix them.

Separate to this and more importantly for your team, we support in the triaging, so hopefully can take some of the pressure off the security-wg.

[MarcinHoppe]

Thanks! Feel free to correct me here, but if I understand this correctly, then Huntr provides a platform similar to HackerOne Community to handle vulnerability reporting and coordinated disclosure, as well as staffs the response / triage team?

[adam-nygate]

HackerOne Community + triage support (lessens the burden on security-wg) + patch incentivisation/facilitation (lessens the burden on maintainers).

Here are a few links to show an example of our current process (and how we differentiate): Disclosure (pre-triage) Bounty (post-triage) Fix acceptance by the maintainer

[MarcinHoppe]

Thanks, I have a clearer picture now. I wonder how is this funded and if it is a sustainable process, but perhaps this is a question that is not relevant for this particular conversation.

[adam-nygate]

@MarcinHoppe happy to explain further and our future plans in the call 🙂

[reedloden]

I would love to join the call as well, if that's possible. :-)

[mhdawson]

meetings are open so no need to register, just use the zoom link in issue created for the meeting a few days ahead of the meeting. The calendar with the upcoming meeting is in: https://calendar.google.com/calendar/u/0/embed?src=nodejs.org_nr77ama8p7d7f9ajrpnu506c98@group.calendar.google.com

The next meeting in the calendar is Nov 30.

It's too early for me to make, but hopefully we can get @esarafianou setup to host by then.

[mhdawson]

FYI @MylesBorins as I know you had some thoughts related to the program as well.

[mhdawson]

Thanks, I have a clearer picture now. I wonder how is this funded and if it is a sustainable process, but perhaps this is a question that is not relevant for this particular conversation.

I do think it is relevant if the suggestion that the program be handed over. That implies we have trust and confidence with it as a solution going forward. The alternative is to wind down and let reporters/maintainers find alternate solutions on their own in which case it's clear they should do their own due diligence before going with a particular organisation.

[mayakacz]

I think there are two chunks of work here.

1. Understand and address the current backlog.

   • The last mention above is "weeks" in July 2020. What would we need to surge and at least address everything that's already in the queue? Is that the right thing to do?

2. Scope the program going forward and find a new home for it.

   • Take time to make the 'right' decision. Addressing the backlog gives space to identify any needs for a new program and potentially buys a bit of time. This also means that transitioning the program will be (hopefully) smoother.
   • Review the list of projects in scope.
   • Confirm that we want to continue the program.

For (2), I would propose a review of the pros/cons of each solution and go from there. I also would like to better understand what's not working with HackerOne today.

GitHub/npm would also be a potential place to accept this vulnerability information, as we already accept and triage reports for packages in the public npm registry: https://docs.npmjs.com/reporting-a-vulnerability-in-an-npm-package. (Disclosure: I work at GitHub.)

[mhdawson]

I don't think anything is "not working" with HackerOne. It's a good solution, the challenge is that we don't have people with enough time to triage/respond to the issues that are coming in.

[mcollina]

There is nothing wrong with HackerOne - it's great. The program has been a success in the community: a lot of people trusted a group of independents (instead of vendor-focused) to manage the vulnerabilities for the ecosystem. The program has been a failure because we do not have enough people to help: it's essentially overwelming. On top of that the fact that some popular modules are not maintained makes this even harder - how much time before disclosing it?

From my point of view all of this is not activity that can be done without a financial backing. It needs dedicated staff (1-2 people at least) for some time to clear the backlog of hundreds of vulnerabilities... and then some daily activity.

The challenge is that the program is still open. For some projects it is the primary way of reporting vulnerabilities. We need to tell them to set something else up - maybe with a recommendation. We should close it now and decide what to do next, any new vulnerability will make it worse.

I think we have two options:

1. hard close: all vulns go /dev/null. If possible we send a message using some API to crunch them all.

2. trust a vendor to process them: we need to select one that will volunteer to process and address them.

[MarcinHoppe]

I want to reiterate what @mhdawson and @mcollina said: HackerOne the platform is great and HackerOne the company have been great partners in the program.

The concerns I have with keeping the program running are:

1. Sustainability: it has proven hard, if not impossible, to keep up with the volume of submissions based on volunteers.
2. Scope management and incentives: currently the program is open for every submitter and every npm package. This means that researchers have incentive to report issues in unpopular packages with low code quality, because this is where vulnerabilities are the easiest to find.

I would be very happy to discuss it in one of the future WG calls.

[mayakacz]

Thanks, this really helps clarify. The issue seems to be support for L2 triage, response, interaction with maintainers, etc. - not the platform. That's in line with my expectations.

From my point of view all of this is not activity that can be done without a financial backing. It needs dedicated staff (1-2 people at least) for some time to clear the backlog of hundreds of vulnerabilities... and then some daily activity.

I think that's right.

Can you provide some information (whatever is possible to provide publicly) on the volume of requests here? Or an estimate of the team size you think would be needed to support this?

[mcollina]

There are currently 261 open issues.

[MarcinHoppe]

A quick update about program status: we disabled submitting new reports to the program.

Right now we are trying to figure out the next steps and wider comms.

[profnandaa]

Kindly update on this thread the way forward once done. My 2 cents on this would be that let it be a PAUSE but not a STOP. I think 3rd party libs form a good chunk of the attack surface and we can't just ignore. Perhaps we go back to the suggestion around which libs can meet the bar for submission depending on metrics such as stars, used_by, etc.

We are only as strong as our weakest links, and 3P libs is that link... IMHO.

[profnandaa]

I'm willing to help build an auto-triaging API for this that H1 can call, let me know.

[mcollina]

I don't think this program has a chance of being resurrected, but I might be wrong.

[mhdawson]

My understanding/view of the current situation is as follows:

1) We've tried to implement the program since the database was transferred over from the Node security project back in 2016 (https://www.programmableweb.com/news/nodejs-foundation-to-oversee-nodejs-security-project/announcement/2016/12/01) 2) At the time we had companies/individuals who wanted to work on building/maintaining a neutral reporting process and database. 3) Over the years interest in maintaining the database has waned and at the same time we have other good alternatives such as https://docs.npmjs.com/reporting-a-vulnerability-in-an-npm-package and https://snyk.io/vulnerability-disclosure/ which are staffed to accept vulnerabilities, work with maintainers and disclose responsibly 4) There is some interest in scoping a smaller program to help members of the OpenJS Foundation handle vulnerability reports

In my opinion next steps would be:

1) Talk to npm/Snyk to see if there is a way we can hand over the current backlog that was reported through H1. 2) Close down the program with a message along the following lines:

Back in 2016 an ecosystem vulnerability database was donated to the Node.js Foundation (https://www.programmableweb.com/news/nodejs-foundation-to-oversee-nodejs-security-project/announcement/2016/12/01).

A lot has changed since then. Some good third party vulnerability reporting options have emerged and the team handling the reports to the ecosystem program for the Node.js project has had a hard time keeping up. At this point we are closing down the program with confidence that there are a number of good alternatives which are available for package maintainers and those reporting vulnerabilities.

A subset of the team will be looking at refocusing their efforts on helping members of the OpenJS Foundation with the handling of vulnerability reports either through the third party programs mentioned or through a much more limited program. If you are interested in helping out with that effort please reach out to X to get involved.

3) Identify those who want to spin up a new effort under the OpenJS Foundation and help them get started.

[mhdawson]

If/when we have agreement we need to pull in the Foundation staff to help work on the message that will go out.

[vdeturckheim]

That's a good summary and plan @mhdawson . +1 for me.

[mcollina]

+1 for me.

[MarcinHoppe]

Well written. +1 from me as well.

[cjihrig]

+1 from me.