Workstream 1 RFC: Establish risks and controls through threat modeling

[yilmi]

Authors

• @arber-salihi
• @ollol88
• @yilmi

Summary

This RFC proposes the creation of a threat model to identify and analyze potential security risks across the AI supply chain. The goal is to agree on the scope of the AI Supply Chain, applicable risks and areas of focus to then address vulnerabilities ensure security properties of AI systems from data acquisition to model use are preserved.

This threat model should serve as a guide for defining and implementing appropriate security controls and mitigations at each stage of the AI lifecycle.

On top of helping workstream contributors, such threat model should also be included in the whitepaper to help the reader gain some critical context when evaluating recommendations.

Priority

P0: This is critical to include in the next release from this workstream as it supports focusing on the most critical areas for the "Software Supply Chain Security for AI Systems" workstream.

Level of Effort

Medium to Large depending on the scope defined.

Drawbacks

Depending on the scope we agree on, we may end up trying to boil the ocean and never reach a useful output. Another possibility is that we may remain too high level, not reaching any meaningful/applicable controls. We should be able able to mitigate this by narrowing down the scope to AI/ML use cases and be as practical as possible when eliciting threats.

We may also end up reinventing without any added value compared to existing frameworks. This should however be mitigated by the initial research on the topic (looking for existing threat models, frameworks). We should also be careful to keep an fresh and critical mind to limit the influence of previous work and identify potential gaps in existing literature.

Alternatives

Pick an existing threat model specific for AI, and use it as is. In all cases a threat model and a list of risks must appear in our white paper to provide context and justify the recommended controls.

Reference Material & Prior Art

AI specific

• https://storage.googleapis.com/gweb-research2023-media/pubtools/7769.pdf (Google paper on Securing the AI Software Supply Chain)
• https://plot4.ai/library (threat library Plot4.ai)
• https://owaspai.org/docs/ai_security_overview/#threat-model
• https://atlas.mitre.org/matrices/ATLAS
• https://www.nccgroup.com/us/research-blog/analyzing-ai-application-threat-models/#abstract

Software Supply Chain Security

• https://slsa.dev/spec/v0.1/threats
• https://openssf.org/blog/2023/09/27/threat-modeling-the-supply-chain-for-software-consumers/ and Threat Model document

Unresolved questions

Have we validated the scope of the threat model? Are we only looking at threats that apply to the model or want also to look at how the model is used

[bhudson33]

Definitely agree that we should pick an existing threat model to start with and then see how that flows against our goals. Once that is complete, we should then randomly pick at least 1 more threat model to see how it holds up as well. This should provide enough evidence as to whether we have the correct ideals or not.