Consider making usage of "Tools" a config option

[moppman]

The idea of using LangChain Tools for special operations is pretty cool, as is your Calculator implementation.

However, using LLMs to automatically classify prompts seems to be pretty fragile, because I'm seeing quite a lot of false positive prompt classifications over several models like Llama2, Mistral and also Gemma.

The models often wrongly classify highlighted code passages as arithmetic expressions, also questions concerning logic and formal languages (e.g. "What's the difference between first order logic and higher order logic?").

So I was wondering if you would consider guarding the classifyPrompt code paths with a config option by maybe expanding the classifyPrompt condition to something like toolsEnabled && classifyPrompt, where toolsEnabled could be another checkbox next to "Disable content parsing" or a box in the Lumos options.

Thanks, and take care!

[andrewnguonly]

Fair points. I previously considered making tools optional. Here are some other thoughts and ideas:

• Tools can be enabled/disabled on an individual basis.
• Prefix triggers can be customized and can be configured as the only method for executing a tool. Basically, tool execution is configured on the classification method (trigger or LLM response).
• To reduce false positives, specifically for the calculator tool, add extra manual validation to confirm that at least 2 operators (i.e. numbers) and at least 1 operand exist in the prompt.

Any thoughts on the above points?

[moppman]

Nice, these all make a lot of sense to me.

[0] Tools can be enabled/disabled on an individual basis.

This would be cool, although it will add noise to the menu or popup especially if you continue adding tools.

[1] Prefix triggers can be customized and can be configured as the only method for executing a tool. Basically, tool execution is configured on the classification method (trigger or LLM response).

This I really like! Making prefix triggers the only method is a sensible default imo. You immediately save one classification roundtrip. The more adventurous users can still enable LLM classification. You could also streamline the triggers: e.g. "!calc" or "!img" like DDG's bangs, so customizing them wouldn't really be necessary anymore.

Making mandatory triggers a default option also means that users won't have the overhead of enabling/disabling each tool individually, so you'd get [0] for free.

[2] To reduce false positives, specifically for the calculator tool, add extra manual validation to confirm that at least 2 operators (i.e. numbers) and at least 1 operand exist in the prompt.

:+1: