Allow adding tools at runtime/dynamically

[schitcrafter]

Current state

Currently, all tools are fetched from beans defined in the @RegisterAiService annotation. This means that it is impossible to add new tools after building, as they need to be actual functions annotated with @Tool.

Proposal

Make it possible to use a more flexible approach, i.e. some method of adding/removing tools at runtime. This would open up many possibilities, like being able to add more tools when new information becomes available.

Contributing

I would be able to work on this feature, if need be. If there are any questions that I could answer, go ahead and ask them

[geoand]

Definitely interesting, but I am wondering how you would add / remove tools at runtime.

[schitcrafter]

Do you mean on the langchain4j or the user-facing side?

[geoand]

The latter

[maxandersen]

if you do this why not just use the langchain models directly?

curious about the usecase as your messages/content would also need to adjust and be aware of the tools to be possible to guide the actions safely or at least in a relevant manner.

[schitcrafter]

The reason why we don't just use the langchain models directly is because then we would lose everything that quarkus-langchain4j does, like specifying prompts with annotations, collecting/building tools from beans, etc.

The usecase here is to have different tools available for different instances of our service, because different vendors have different information available.

[geoand]

The reason why we don't just use the langchain models directly is because then we would lose everything that quarkus-langchain4j does, like specifying prompts with annotations, collecting/building tools from beans, etc.

I completely agree, this makes perfect sense

The usecase here is to have different tools available for different instances of our service, because different vendors have different information available.

Can you show some pseudo-code of how you expect to use this?

[schitcrafter]

With the architecture that we have thought of, we would get a REST call by some service registering itself as a new tool, after which we would add this tool to the AI together with a handler that internally makes a REST call back to the service. When the AI decides that it would like to use this tool, it calls the provided function (which, as far as I know, is how langchain4j does it internally).

For how the second part could be implemented, probably new fields on the @RegisterAiService annotation, either providing a way for one of the beans in the tools field to act as a supplier, or adding a new toolSupplier field. These suppliers (made by the user) could then return all dynamically added tools. I'm not sure if there could be a way to modify the internally created AiService, based on the current API, as one usually just injects the interface from which the AiService is made.

[geoand]

So assume some of the proposals covered in https://github.com/quarkiverse/quarkus-langchain4j/discussions/246#discussioncomment-8589272 could help?

[schitcrafter]

I'm not sure those proposals would help us with this specific problem - they would be interesting for other stuff, though. I can see us cutting down on tools every time we call a function on the interface, so a user that can't get information from a tool for permissions reasons shouldn't even see that tool as being available.

As far as I've understood it, @ToolBox allows us to cut our tools down to a subset of the available tools/specify which function on the interface has which tools available, which are nice features, but not what we're looking for. The tools inside a ToolBox would still need to be rigidly defined as functions (maybe even with the @Tool annotation), which makes it impossible for us to change them at runtime.

Maybe the @ToolBox would be a nice place to put a programmatic approach to tools, like making a ToolBox be able to implement an interface which defines method that get all available tools / call a specific one.

[geoand]

How about the toolSelector idea?

[schitcrafter]

toolSelector would only allow us to choose a subset of already existing tools, unfortunately. Going back to our usecase (different tools for different vendors), this would allow us to cut down our tools for each vendor, but would still require us to include all tools for all vendors inside the application, which we simply can't do as that would give one vendor information about the tools of another.

In general though, this toolSelector seems very useful as a way to cut down on tokens / enforce permissions.

[geoand]

I see.

So would having something like @RegisterAiService(toolsSupplier = SomeSupplierOfListOfClass.class) meet your needs?

[schitcrafter]

Yes, that would be what I'm looking for, if the toolsSupplier then gave out a list of ToolSpecification and ToolExecutor / something easy to create on the fly. If we need to return functions/beans here then we couldn't add new ones (easily, anyway. I'm sure quarkus could do this somehow with lots of reflection/bean magic but I wouldn't want that as the intended usecase)

[geoand]

then we couldn't add new ones

This is the part I'm still unclear about... How do you envision adding new ones?

[schitcrafter]

I'd say that we would make a class that we put in the toolsSupplier field, which then has acts as a supplier for a set of both ToolSpecification and ToolExecutor (for describing and executing tools, respectively). How exactly this happens doesn't really matter for us, it might be a simple implements Supplier<ToolSpecAndExecutor>, or implementing some new interface, or an annotation, or whatever. This supplier should be accessed by the library before every request to the AI, and what is returned is based on which tools have already been added. Adding a new tool would mean creating a new specification and executor, and giving it to the toolsSupplier to be returned next time the library asks for them.

Of course, this is based on what I would like to use, not on how quarkus-langchain4j / langchain4j works internally. I haven't looked at their code and don't know what would work best for their architecture.

Example code

With a tool specified like this (not strictly necessary, I've just done it to make the code simpler)

public record Tool(
        ToolSpecification spec,
        ToolExecutor executor
) {}

We could have a tool supplier like this:

public class MyToolSupplier implements Supplier<Set<Tool>> {
    // this could also be a custom interface, like a `ToolSupplier`

    private Set<Tool> tools = new HashSet<>();

    public Set<Tool> get() {
        return tools;
    }

    // The tool passed into here is created by calling a REST
    // interface, in our case. The executor of the tool just
    // makes a REST call back to the correct service,
    // with all parameters specified in the ToolSpecification.
    public void addTool(Tool tool) {
        this.tools.add(tool);
    }
}

[geoand]

This supplier should be accessed by the library before every request to the AI

Okay, this is what I was missing.

It's definitely interesting, but needs plenty of thought.

[geoand]

@langchain4j is there any new concept in upstream LangChain4j that would cover this?

Thanks

[langchain4j]

@geoand I will be looking into adding more flexible way(s) to register/select tools this/next week, I will look deeper into this proposal, but on the first glance it seems doable.

[geoand]

🙏🏼

[langchain4j]

Hi @schitcrafter, I am working on this feature. Could you please explain how do you plan to define/implement ToolExecutor? Also, what about tool parameters? Do you plan to have any? It would be great if you could provide a few examples. Thanks!

[schitcrafter]

Hey, this is a thing that came up at my workplace, and since I don't work there anymore, @d135-1r43 will be taking over this issue. Sorry for the confusion

[langchain4j]

Hi @d135-1r43, could you please check my last question?