Add mechanism for tool dependencies in AGS (installation, secrets)

[victordibia]

AssistantAgent can take a list of tools. These tools are typically python functions which themselves can have dependencies and secrets. For example, a search tool might use a specific search library e.g. googlesearch and need a search api key.

In a no code environment, it is useful to have some mechanism to

• [ ] install specified libraries and
• [ ] load specified keys to make them available when the tool is called.

How

Create a ToolHelper class that is used with the component_factory in loading tools.

• For each dependency, check if the dependency is installed (attempt to import)
• If not installed, install it (or validate and provide some confirmation workflow where the user can agree to explicitly install)
• load the tool

Challenges / Considerations

• Tool calls get executed in the same environment as the application (AGS in this case).
  • UX should communicate this to the user
  • One assumption here is that the user designs/understands the tools they create and implicitly are comfortable with outcomes and side effects of their execution. The opposite of this is with a general purpose tool like code execution where arbitrary code with arbitrary side effects can be executed.

[ekzhu]

This can be helped by using distributed agent runtime. Agents don't have to be in the same application process. The agent that uses tools can be running from a separate machine.

We can make agentchat to run on distributed runtime.

[victordibia]

I agree that the challenge above can be helped by using a distributed runtime.
We will need to create a design doc/discussion around how to support distributed runtimes in AgentChat.

Beyond that, any other comments on the proposed implementation (I added a how section.) ?

[ekzhu]

I am wondering if the dependencies & secret should be provided and resolved at start of a runtime in the backend rather than managed through some UI. It is very hard to secure the application environment against arbitrary code execution (i.e., user defined tools).

[victordibia]

I agree that end users should not be defining tools. However - AGS is a developer tool. For debugging. Fair to assume similar exposure with e.g., a notebook? My feeling is that it should be treated sort of like an IDE? Open to thoughts here.