Recommended Updates to Excessive Agency

[Bobsimonoff]

Recommend trialed instead of trialled

favor instead of favour

etc. instead of etc

Add 'to' to this phrase: Limit the permissions that LLM plugins/tools are granted to other systems the minimum necessary So it reads: Limit the permissions that LLM plugins/tools are granted to other systems to the minimum necessary

Consider reworking the following: Excessive Agency is the vulnerability that enables damaging actions to be performed in response to unexpected/ambiguous outputs from an LLM (regardless of what is causing the LLM to malfunction; be it hallucination/confabulation, direct/indirect prompt injection, malicious plugin, poorly-engineered benign prompts, or just a poorly-performing model). The root cause of Excessive Agency is typically one or more of: excessive functionality, excessive permissions or excessive autonomy.

To be: Excessive Agency is the vulnerability that enables damaging actions to be performed in response to unexpected/ambiguous outputs from an LLM (regardless of what is causing the LLM to malfunction; be it hallucination/confabulation, direct/indirect prompt injection, malicious plugin, poorly-engineered benign prompts, or just a poorly-performing model). Excessive agency is a vulnerability of excessive functionality, permissions, and/ or autonomy. This differs from Insecure Output Handling which is concerned with insufficient scrutiny of LLM outputs.

Consider these adjustments to Common Examples of Vulnerability

1. Excessive Functionality: An LLM agent has access to plugins which include functions that are not needed for the intended operation of the system. For example, a developer needs to grant an LLM agent the ability to read documents from a repository, but the 3rd-party plugin they choose to use also includes the ability to modify and delete documents.
2. Excessive Functionality: A plugin that was trialed during the development phase was dropped in favor of a better alternative, but the original plugin remains available to the LLM agent.
3. Excessive Permissions: An LLM plugin has permissions on other systems that are not needed for the intended operation of the application. E.g., a plugin intended to read data connects to a database server using an identity that not only has SELECT permissions, but also UPDATE, INSERT and DELETE permissions.
4. Excessive Permissions: An LLM plugin that is designed to perform operations on behalf of a user accesses downstream systems with a generic high-privileged identity. E.g., a plugin to read the current user's document store connects to the document repository with a privileged account that has access to all users' files.
5. Excessive Autonomy: An LLM-based application or plugin fails to independently verify and approve high-impact actions. E.g., a plugin that allows a user's documents to be deleted performs deletions without any confirmation from the user.

[rot169]

Thanks @Bobsimonoff, I'll be merging most of these in the next revision. Just wanted to check with you regarding one of the current examples; are you suggesting the removal of this item? (And if so, what's the rationale?)

Excessive Functionality: An LLM plugin with open-ended functionality fails to properly filter the input instructions for commands outside what's necessary for the intended operation of the application. E.g., a plugin to run one specific shell command fails to properly prevent other shell commands from being executed.

[Bobsimonoff]

Maybe it is just my reading or the wording of the example, but I looked at the description of Excessive Agency:

An LLM-based system is often granted a degree of agency by its developer - the ability to interface with other systems and undertake actions in response to a prompt. The decision over which functions to invoke may also be delegated to an LLM 'agent' to dynamically determine based on input prompt or LLM output.

and

Excessive Agency is the vulnerability that enables damaging actions to be performed in response to unexpected/ambiguous outputs from an LLM (regardless of what is causing the LLM to malfunction ... list of causes for LLM malfunction)

I felt that this doesn't feel like Excessive Functionality / Excessive Agency, this feels more like Insecure Plugin design. This isn't unexpected/ambiguous LLM output nor is it related the the LLM having agency, it sounds like a security shortcoming in the plugin.

[rot169]

Hmm, I see where you're coming from. There's certainly a degree of overlap, although I feel the difference with Excessive Agency is that it's specific to a particular LLM application, whereas Insecure Plugin is application-agnostic. I'd certainly welcome wider debate on the topic, as I keep see-sawing in my own mind!

[Bobsimonoff]

Right - I think you are saying what I am thinking... However, the following is an inherent flaw in the plugin regardless of what LLM application it lives in. Maybe a different example would help, if we have to work this hard to understand it, consider the readers.

Happy for others to share their thoughts

The following has nothing to do with LLMs other than it is an LLM plugin, but not a flaw in the LLM side of things:

Excessive Functionality: An LLM plugin with open-ended functionality fails to properly filter the input instructions for commands outside what's necessary for the intended operation of the application. E.g., a plugin to run one specific shell command fails to properly prevent other shell commands from being executed.