pub needs a dependency security audit feature

[kibantony]

pub needs a feature similar to npm audit that will scan a given project for known vulnerabilities and security issues by checking package versions against a vulnerability tracking database.

[jonasfj]

Wouldn't we also have to build such a database?

note. I'm not saying we shouldn't do this, only that it seems like we have lot of other things to do first.

[kibantony]

Yes. I'm not sure how pub works internally, but presumably you could add a table to the existing package database to track vulnerabilities per package. It would be nice to easily view and report vulnerabilities at pub.dartlang.org.

Alternatively you could rely on the something like the NVD, but that is problematic for a variety of other reasons.

In this age of constant security breaches, this is a critical feature. It's very unsettling to security minded engineers (all engineers?) to include packages from unknown sources with no means to report or audit for vulnerabilities. We can do this for nearly every other part of our stack, (Java, Node, etc.) but not Dart/Flutter. This will only get more critical as Flutter takes off and Dart becomes more mainstream, as I'm sure it will in the coming months.

[jonasfj]

I agree, this would be pretty neat to do. It's just not obvious to me how it would work.

Who can report a vulnerability? Who should review reported security issues? Who flags a package-version as containing a vulnerability?

I've been thinking a bit on how we could do a review system... I'm not sure what it would look like, maybe it's just starring favorit packages, or rating 1-5 stars or writing a text review in markdown.. inspired by how apps are rating the Android Play Store. Maybe, a vulnerability report is a just a particularly bad rating :)

Would vulnerability reports need to private? If so how would we warn users? How long we would keep vulnerabilities secret? Would they need independent reviews/verification?

It's not obvious to me how to do this. I would personally like to avoid supporting a pub audit thing that gives a false sense of impression, if the vulnerability reporting system isn't robust enough.

Note. I do think a pub audit command that could give insights into where the dependent packages originate from would be useful. But we likely need more metadata server side to see that two packages are published by the same author.

---

Note. You can report security issues to package uploaders, their emails are listed on the package page. It's not much, but you can contact the authors this way.

[kibantony]

These are good questions. I can give you specific answers (see below) which may or may not be entirely useful. I'm sure we can agree that once a platform reaches a certain threshold of maturity, explicit vulnerability reporting/tracking/auditing isn't just a neat feature, it's absolutely vital. I would argue that Dart/Flutter has already blown well past this point and will continue to accelerate.

I have no real insight into the organization, but the fact that you're asking questions of this nature leads me to believe there isn't much ancillary support coming from Google. In my answers below I cite npm as the standard example. It's not clear to me how well that applies since npm is backed by a single corporate entity that provides deeply dedicated support and resources. I would hope that Google would provide similar levels of support to Dart/Flutter given the direction it's headed. If that's the case then the short answer to all of your questions is: Escalate up to the larger team/management the need for first class security team and ancillary support to address the need for direct vulnerability reporting/tracking/auditing for Dart packages.

As for specific answers to your questions, this sort of thing is already being done by other language and dev tool teams, so one thing we can do is take a look around and see what's working for them. The closest example to pub/Dart would probably be npm. Looking at npm's policy we can consider some possible answers to your questions.

Who can report a vulnerability?

Anyone.

Who should review reported security issues? Who flags a package-version as containing a vulnerability?

Not the package maintainer. Presumably Google's security team. Or, if that's not feasible, a new security team formed as part of the Dart team. Are there any security engineers already working on Dart? (I hope so!)

Would vulnerability reports need to private? If so how would we warn users? How long we would keep vulnerabilities secret? Would they need independent reviews/verification?

Again npm's policy seems perfectly reasonable here. Reports are reviewed by the security team, independent of the package maintainer. They are kept private between the security team and the package maintainer for some reasonable amount of time (45 days in npm's case), after they are publicly disclosed (presumably at pub.dartlang.org).

[jonasfj]

...explicit vulnerability reporting/tracking/auditing isn't just a neat feature, it's absolutely vital. I would argue that Dart/Flutter has already blown well past this point and will continue to accelerate.

I think we're all in agreement :)

My point was that developing package moderation and vulnerability management policies for pub is a non-trivial process that involves many stakeholders. And it's not obvious that we should simply mirror what other package managers have done.

---

I guess you're suggesting that we simply have an email you can send vulnerabilities to, which we would review and forward to package owner. And if not fixed within 45 days we would publish the report. (which perhaps seems easier than I initially thought)

I think that mostly involves publishing a policy, ensuring we have the right team receiving emails, and setting up some way of associating notices to specific package-versions. I filed https://github.com/dart-lang/pub-dartlang-dart/issues/2160, and closing this as it's not obvious what the CLI tool aspect will look like before the data is present serverside.

[kibantony]

Thank you @jonasfj. 👍

[bach942]

@jonasfj and @kibantony you might be able to use Snyk's database or have them create a plugin to scan the dependencies. They do this for other languages but is similar to npm audit, npm audit fix. Docs.

[oravecz]

Google now has the reporting process. https://dart.dev/security

Google has the data reporting format. https://github.com/ossf/osv-schema

Google has an open source database for this purpose. https://github.com/google/osv

Google doesn’t have the priority to surface Dart into these processes or tools. https://github.com/google/osv/issues/62

Google has started using GitHub Security advisories (per repo?) to publish security issues. https://github.com/dart-lang/sdk/security/advisories

Google may be embracing GitHub’s advisory reporting, but not showing in Github’s centralized database. https://github.com/advisories

Google has posted a handful of vulnerabilities to CVE (same 5 as shown in dart/sdk repo) https://cve.report/software/dart/dart_software_development_kit

Google (and no other commercial or open source vulnerability scanners) exist for Dart at the time of writing. (pub.dev package vulnerabilities, or unsafe coding practices)

Google has no mechanism for tracking vulnerabilities across the thousands of libraries in pub.dev.

[stefanschaller]

@kibantony Very good points. Thanks! There is just one thing that I personally not agree or let's say don't understand. Why should we keep issues private for 45 days? Just to give the maintainer time to fix it? But maybe another developer could also fix it. That's the power of open source, right?

@oravecz Do you know if there is any progress since your last comment? I need something like the vulnerability scanner that works similar to npm audit.

[oravecz]

Last week this article on supply-chain security was published. https://medium.com/dartlang/partnering-with-github-on-an-supply-chain-security-485eed1fc388

I admit to not knowing much about GitHub's Advisory Database, but it seems that it won't prevent bad actors, but rather people who opt-in, self-police and report their own vulnerabilities. While I think this will help for core Dart and Flutter code, the third-party libraries will be missed.

It would be good of Google to publish their GoogleDoc on a complete supply-chain security roadmap if such a thing has been considered.

[zmoshansky]

I'm very happy to see Github integration and improvements in the supply chain security.

I think it's still important to have an npm audit esque tool for those whose code is in a private cloud/non-github host. However, I don't see a reason a tool like that couldn't rely on GitHub's Advisory Database.

Without some of these tools, ISO27001 and similar enterprise standards will be much tougher to manage.

https://www.iso.org/isoiec-27001-information-security.html

[oravecz]

The main problem with GitHub Advisory database is that the package authors themselves have to "opt-in" and self-report vulnerabilities. Google does not require the advisory database as a criteria for listing in pub.dev. There is currently no carrot and no stick, so most package authors will ignore the capability.

[oravecz]

I'll also admit not knowing much about how vulnerabilities are reported in the npm world. I'm aware of the NVD, CVE, and I know that the GitHub Advisory Database now powers the npm audit command. Snyk.io uses their own proprietary database, but I think it and GitHub started with NVD to begin with. Throughout all of these databases there are fewer than 5 Dart-language vulnerabilities reported last I checked.

[zmoshansky]

There also seems like an opportunity for a group like (Google's Project Zero, OSV, etc) to do more active fuzzing & scanning of packages listed on pub.dev. Likewise, for pub.dev publishing the results/compliance to hopefully incentivize compliance.

Even if there aren't many reported vulnerabilities, for certification, one needs to be able to respond quickly if any are reported. Hopefully over time, it becomes more valuable.

[jonasfj]

At a high-level the thinking I currently have about security advisories is:

• Ingest advisories from github into pub.dev
• Serve advisories through API on pub.dev
• Highlight advisories in dart pub get, dart pub upgrade and dart pub outdated.
• Vague idea that a dart pub audit feature would be nice, exact semantics and CLI interface is unclear.

The thinking is that the Github Advisory database is a good way to get started. It'll allow package authors who publish on github to file advisories for their packages. And as I understand it there is a review process of these advisories before they are included in the github.com/advisory database. I'm not sure if the reviewers will be Dart domain experts (maybe when the ecosystem is larger), but at a minimum there is a good chance that spam will be filtered out :D

The main problem with GitHub Advisory database is that the package authors themselves have to "opt-in" and self-report vulnerabilities.

I'd call it a limitation. But that might also make it a good way to get started.

I think it's important that we enable package authors to notify downstream package consumers of security advisories. Advisories written by package authors are hopefully going to be high quality and relevant for package consumers.

Reporting security vulnerabilities in third-party packages: I think this is a very hard problem. There are many reasons why we might want to allow such reports, like:

• Package authors may be unaware of the vulnerability,
• Package authors may not provide steps for how to report vulnerabilities,
• Package authors may be absent (whether temporarily busy with life or simply lost interest).

In the future, this might be addressed by creating a Dart advisory database that complements the Github Advisory database. But it probably involves a lot of non-trivial work like:

• Establishing a committee to review PRs with advisories (and possibly edit the advisories to ensure quality).
• Establishing a responsible disclosure process.
• Figuring out a review process when package authors disputes an issue.
• Automation for synchronizing with the Github Advisory database.
• Automation for verify package ownership.

When/if there are disputes it can become hard to strike a reasonable balance between trusting package authors and keeping package consumers safe.

Importantly: I think we should be careful to establish solid guidelines for reviews and processes to ensure high-quality advisories; before we even consider making a Dart advisory database. If we are not careful we can accidentally create a collection of low quality advisories that mostly just annoy package consumers, who will treat it as noise to be ignored.

Just to be clear: The Github Advisory database is a good start. But it doesn't preclude us from establishing our own Dart advisory database if we want to do so in the future. But one step at the time might be wise.

---

side note: I did a bit of lurking at what golang is doing, and they are not just attaching advisories to a package version, which I think is what we will initially do, they are attaching advisories to specific symbols, etc. This has pros/cons: less noise, but also you might get an advisory when you import and use a function rather than when you add a dependency on the package. It also makes creating advisories harder, or so I suspect, but it's something to keep in mind.

[cpfeiffer]

I haven't seen any mention of owasp-dependency-check's experimental dart support, so here it is: https://github.com/jeremylong/DependencyCheck/issues/2729 In particular, see this comment.

[danrimbault]

Has there been any updated progress on this? The linked comment in the last comment on this issue mentions Q4 2023 for a dart scanner?

[jonasfj]

Advisories for pub can be published to: https://github.com/advisories?query=type%3Areviewed+ecosystem%3Apub

The pub client will now warn of security advisories: https://github.com/dart-lang/sdk/commit/bfe68f9ff552e3a11dcc218ee502feeee6299340

I don't think this has shipped yet, but will probably ship in Dart 3.3.

There is an issue for writing documentation here: https://github.com/dart-lang/site-www/issues/5458

I don't know if we should do a dart pub audit feature, probably we should also surface advisories in dart pub outdated.

Mostly, I imagine that a dart pub audit feature is useful for CI, as it could fail if there is a security advisory affecting one of the dependencies (and it's not ignored with ignored_advisories -- a mechanism to allow an application author to allowlist an advisory once they've reviewed it and determined that it doesn't affect their code / like dependency_overrides it has no effect on direct- or transitive-dependants).

[jonasfj]

@szakarias might have additional context, and know if there are other things we're still missing.

[parlough]

I don't think this has shipped yet, but will probably ship in Dart 3.3.

As a note, unless it's cherry-picked to 3.3, it is currently set to land in Dart 3.4 (if no reversions). You can try it out on the latest dev and main channels :)

[danrimbault]

Thanks for the fast response @jonasfj! And great news about the change for security advisories. Would be great if it shows in dart pub outdated. But first prize in my opinion would be a full dart pub audit feature that throws an error to the terminal for CI as you say. Thanks for the update too @parlough. We will watch for 3.4 release.

[jonasfj]

@parlough oh, right, yes, my bad, it'll probably be in 3.4 :D

@danrimbault, yes... but https://github.blog/2022-10-06-githubs-supply-chain-security-features-now-support-dart/

I'm not sure most people will need it. The security features Github has are probably more convenient.