Call code outside of langchain in routes

[adampash]

I have two related questions for routing:

1. PromptRoute always requires a chain, but this feels quite limiting. It can be convenient to use the router's outcome as the end result of a pipeline—that is, sometimes I just need to know the selected route so I can delegate to the right part of my code. As is, I have to pass a chain in. It would be very handy to be able to pass in, say, a callback function, that simply gets the name of the chosen route.
2. Related to that: In the evaluate function of the RoutingChain, the debugger helpfully logs the chosen route, but I can't access that in code, so if I want to do the above (run my own code depending on which route is chosen), I have to create dummy chains for every route, then pattern match on whichever chain is returned by the evaluate function. It's a lot of overhead when the name is right there, just out of reach. :)

Would it be possible to do any of the following:

1. Set chain to optional on PromptRoute?
2. Set a callback function on PromptRoute?
3. In RoutingChain, create a new function that evaluates but just returns the name instead of a chain?

Basically what I'm looking for is any path to delegate out to my code from the existing routing structure without a lot of overhead that's ultimately thrown away. I do, of course, see the value of the chain in this pipeline... it's just that I also know that there are a lot of times when I don't need that overhead.

Also, it's very possible I'm missing something that would allow me to accomplish exactly what I'm asking about, and I've just missed entirely! Let me know which and I'm happy to help out.

[brainlid]

Hi @adampash!

Thanks for detailing your use case and the shortcomings of the current API. That makes a lot of sense.

You are correct that the requirement for the chain is blocking your more direct usage. I'd like to fix that too.

You can use it this way for now:

routing_chain
|> RoutingChain.run()
|> LangChain.Utils.ChainResult.to_string()

This returns an {:ok, route_name} which includes the possibility of "DEFAULT" as a result if no better match is found. It could also return an {:error, reason} result.

If we set a callback function on the PromptRoute, how would you use that? Would it send a message to a process? Did you have something else in mind? It feels like the selected route name should be a more first-class response.

The idea with the evaluate function is to automatically execute the selected chain. If there is no chain. I'm currently thinking about what the behavior should be if there is no chain. It could be specified by an option passed into evaluate or perhaps a config on the RoutingChain. Hmm. :thinking:

[adampash]

You can use it this way for now:

routing_chain
|> RoutingChain.run()
|> LangChain.Utils.ChainResult.to_string()

This returns an {:ok, route_name} which includes the possibility of "DEFAULT" as a result if no better match is found. It could also return an {:error, reason} result.

Ah, that's perfect! I can definitely make due with that.

If we set a callback function on the PromptRoute, how would you use that? Would it send a message to a process? Did you have something else in mind? It feels like the selected route name should be a more first-class response.

I guess my thinking was a callback—even an MFA—as an alternative to a chain could be useful. If this route is chosen, call this function.

The idea with the evaluate function is to automatically execute the selected chain. If there is no chain. I'm currently thinking about what the behavior should be if there is no chain. It could be specified by an option passed into evaluate or perhaps a config on the RoutingChain. Hmm. 🤔

Yeah, that makes sense, and when all your logic involves processing LLM chains, works really well. I think that finding elegant patterns for delegating to your own code would be really valuable. I had even wondered if there was a pattern where a chain could be just a local MFA that implements some chain behavior...

I'm sure you've thought a lot more about this, I'm sort of spitballing! Worth saying, the router is really handy, and I can see using it a lot. :)

[brainlid]

I don't like overloading a single function to have more return types than makes sense. How about an evaluate_chain which returns the selected chain. If the chain was nil, it could return an :error tuple.

Then a separate evaluate_route_name or evaluate_route? It could return either the selected route name string or the PromptRoute struct. The one snag is the "default" case. The logic needs a default to fall back to if no specific routes match. The default route is also used in some error cases as well. I didn't want to require or expect people to provide a PromptRoute that is configured as the default. That gets messy. Did they include one? Did they include multiples? etc.

What do you think?

[brainlid]

Another option is to break the current API and instead of setting a default_chain, it would set a default_route. Then the default route name could be used in the prompt internally and a PromptRoute struct or route name could more easily be returned.

[adampash]

I don't like overloading a single function to have more return types than makes sense. How about an evaluate_chain which returns the selected chain. If the chain was nil, it could return an :error tuple.

Then a separate evaluate_route_name or evaluate_route? It could return either the selected route name string or the PromptRoute struct. The one snag is the "default" case. The logic needs a default to fall back to if no specific routes match. The default route is also used in some error cases as well. I didn't want to require or expect people to provide a PromptRoute that is configured as the default. That gets messy. Did they include one? Did they include multiples? etc.

What do you think?

From my perspective, for what I need, I think this is a really solid path. And the default_route option makes sense, too.

[brainlid]

This should be resolved now. Will publish a new release soon.

[adampash]

Thanks so much, @brainlid!