Bobsimonoff
commented
1 year ago More details Description The following part of the description is really related to Training Data Poisoning.
To mitigate this risk, LLM applications should perform adequate data sanitization to prevent user data from entering the training model data.
While the following are all very true, I am not sure they correspond to OWASP’s definition of a vulnerability; “A vulnerability is a hole or a weakness in the application, which can be a design flaw or an implementation bug, that allows an attacker to cause harm to the stakeholders of an application”:
LLM applications have the potential to reveal sensitive information, proprietary algorithms, or other confidential details through their output. This can result in unauthorized access to sensitive data, intellectual property, privacy violations, and other security breaches. It is important for consumers of LLM applications to be aware of how to safely interact with LLMs and identify the risks associated with unintentionally inputting sensitive data that it may be returned by the LLM in output elsewhere.
The consumer-LLM application interaction forms a two-way trust boundary, where we cannot inherently trust the client->LLM input or the LLM->client output. It is important to note that this vulnerability assumes that certain pre-requisites are out of scope, such as threat modeling exercises, securing infrastructure, and adequate sandboxing. Adding restrictions within the system prompt around the types of data the LLM should return can provide some mitigation against sensitive information disclosure, but the unpredictable nature of LLMs means such restrictions may not always be honoured and could be circumvented via prompt injection or other vectors.
Examples of Vulnerability The following example could be Training Data Poisoning depending on the actual root cause:
- Incomplete or improper filtering of sensitive information in the LLM’s responses.
This seems like a Training Data Poisoning problem.
- Overfitting or memorization of sensitive data in the LLM’s training process.
In this case, the example itself refers to the root cause, Insecure Output Handling (Possibly Overreliance).
- Unintended disclosure of confidential information due to LLM misinterpretation, lack of data scrubbing methods or errors.
Example Attack Scenarios
This example appears to be caused by excessive agency where user A was allowed to see user B’s data because of Excessive Permissions being set, as per the description of Excessive Agency.
- Unsuspecting legitimate user A is exposed to certain other user data via the LLM when interacting with the LLM application in a non-malicious manner.
This example could be Training Data Poisoning if the PII is in the model itself or Excessive agency if the LLM as access to data in other systems, due to permission issues, and finally, it could represent Insecure Plugin Design if the problem is in the plugin’s security architecture. In this case, I’d argue that Prompt Injection is the attack vector, not the vulnerability.
- User A targets a well crafted set of prompts to bypass input filters and sanitization from the LLM to cause it to reveal sensitive information (PII) about other users of the application.
This example is more correctly classified as Training Data Poisoning.
- Personal data such as PII is leaked into the model via training data due to either negligence from the user themselves, or the LLM application. This case could increase risk and probability of scenario 1 or 2 above.
emmanuelgjr
GangGreenTemperTatum
leondz
In summary, this “vulnerability” is problematic because it mostly doesn’t represent a root cause, but a result or symptom. In the 2021 OWASP Top 10, they reoriented from symptoms to root causes. They actually renamed sensitive information exposure to cryptographic failure to focus on root causes.
There is, however, one aspect of this vulnerability that seems to be a true root cause. Regardless of the input it gets, large language model outputs are unpredictable. That unpredictability doesn’t necessarily come from training data poisoning, third-party data sources, or prompt injection. It comes from the stochastic nature and natural language processing capabilities of the large language model as well as ambiguities and inaccuracies inherent in natural language. I would argue that this aspect is a key point mentioned in this section that should be carried forward, however, I believe it makes more sense in Overreliance.