How to selectively import only some audits?

[faern]

Hi! I started exploring cargo vet a few days ago and tried it on a fairly small repo of mine. I added Mozilla's repository as an import source, since I consider them a trustworthy organization in general. I thought they could be a good source to import some audits from. Upon running cargo vet all their audits that overlap with my dependency tree were silently imported and the tool gave me a green check. I would have expected it to ask me selectively for each audit if I wanted to import it or not.

When I look into supply-chain/imports.lock I have, among other things:

[[audits.mozilla.audits.log]]
criteria = "safe-to-deploy"
notes = "One dependency removed, others updated (which we don't rely on), some APIs (which we don't use) changed."
...

The notes clearly state that they did not read the entire thing, only selective parts. How can Mozilla themselves trust this audit? They will not get notified in the future if they change their usage of log so that they do use the APIs that they did not audit. This seems like a flaw in how they choose to use safe-to-deploy or a flaw in what guarantees vet is supposed to give. I don't want to trust this audit, since I have no idea what APIs they did audit and which ones they did not audit.

I can of course choose to not import Mozilla's audits at all. But that's such a heavy-handed solution with many downsides. What if they also have this audit in the same repo:

[[audits.mozilla.audits.extremely-complex-cryptography-stuff]]
who = "Very Security Professional <epic-crypto-person@example.com>"
criteria = "safe-to-deploy"
notes = "I made a thorough audit of all aspects of this crate. I also made a formal proof that it's correct"
...

I really don't want to miss out on this amazing free audit possibility. So it would be great if audit imports could be selective IMHO. Or am I missing the point in how cargo vet is supposed to be used?

[faern]

I think it would be really hard to have any organization qualify under the definition "I blindly trust whatever this org classify as audited now and in the future". As compared to "This is a source I want to browse audits from whenever I happen upon a crate I have not yet audited myself".

[mystor]

It is possible to exclude certain audits from being imported from a source using the exclude key in config.toml, as documented here: https://mozilla.github.io/cargo-vet/config.html#exclude. For the audit you identified, it might look like this:

[imports.mozilla]
url = "https://raw.githubusercontent.com/mozilla/supply-chain/main/audits.toml"
exclude = ["log"]

Also, thank you for bringing that audit to our attention. This unfortunately fell short of our auditing standards, and we're taking steps to make sure incomplete audits like that don't happen again.

[faern]

Thank you for pointing out exclude! I had read the entire tutorial, but not the reference. exclude can be somewhat of a solution to my problem, but IMHO not an ideal one. The same import source could very well provide a very satisfactory base audit of log but not a diff audit I want to import. From my understanding exclude does not allow me to handle this?

Is the usage of import sources intended to be like I described as "I blindly trust whatever this org classify as audited now and in the future"?

I understand that maybe it's too late to change this tool in such a drastic way now(?), but what do you think in general about my idea to selectively import audits instead of the current state? Is there any downside to that that I'm missing? I don't think it would be more work than today. Today whenever I commit changes to imports.lock I have to read through every entry carefully anyway to make sure what I trust is satisfactory.

[bholley]

Thank you for pointing out exclude! I had read the entire tutorial, but not the reference. exclude can be somewhat of a solution to my problem, but IMHO not an ideal one. The same import source could very well provide a very satisfactory base audit of log but not a diff audit I want to import. From my understanding exclude does not allow me to handle this?

I believe not.

Is the usage of import sources intended to be like I described as "I blindly trust whatever this org classify as audited now and in the future"?

Not entirely, because the workflow still requires that you review and commit the new imports to your tree. If you have concerns about any of them, you can perform and record your own audit for that crate, at which point the local audit will take precedence and the imported audit will disappear. Or, alternatively, you can decline to merge the changeset that adds the dependency in question.

I understand that maybe it's too late to change this tool in such a drastic way now(?), but what do you think in general about my idea to selectively import audits instead of the current state? Is there any downside to that that I'm missing? I don't think it would be more work than today. Today whenever I commit changes to imports.lock I have to read through every entry carefully anyway to make sure what I trust is satisfactory.

It's not clear to me what practical benefit this feature would provide. There's already a reliable means to review your imported audits, allowing you to accept them, re-do them, or reject them (along with the code they cover).