search

i-am-bee / bee-agent-framework

The framework for building scalable agentic applications.
https://i-am-bee.github.io/bee-agent-framework/
Apache License 2.0
745 stars 79 forks source link

OpenMeteo tool optimization #110

Open michael-desmond opened 5 days ago

michael-desmond commented 5 days ago

Is your feature request related to a problem? Please describe. Using openMeteo with Granite 3.0 8b and I ran into a few issues that may be worth us looking into. Also touches on the general topic of tool optimization.

  1. The tool name openMeteo is not language intuitive i.e. its not obviously descriptive of its function. As such the LLM will sometimes use the tool to perform actions its not intended for.
  2. The tool responds with metadata that is probably not useful to the calling agent. https://github.com/i-am-bee/bee-agent-framework/blob/9e107d75f570032528f1b46dec231f8d23db8741/src/tools/weather/openMeteo.ts#L45C1-L52C21
  3. Inclusion of both daily and hourly weather data seems unnecessary. A model with a 4k context window has its token memory consumed by a single response in some forecasting circumstances.
  4. The selection of current, daily and hourly data are somewhat arbitrary and the naming is not ideal

Describe the solution you'd like

  1. Rename the tool to something intuitive such as GetWeatherForecast of WeatherForecast to make obvious its function. This should marginally help any LLM to more consistently utilize the tool.
  2. Minimize the tool response, removing latitude, longitude, generationtime_ms, utc_offset_seconds, timezone, timezone_abbreviation and elevation.
  3. Add a configuration option to limit response to daily, hourly or both. This way smaller models could be configured to save token memory.
  4. Do a pass over the variables selected for the various forecast ranges i.e. hourly-> Apparent Temperature, Relative Humidity (2 m), Precipitation Probability (temperature, humidity, precipitation) daily -> Maximum Apparent Temperature (2 m), Minimum Apparent Temperature (2 m), Precipitation Probability Max current -> Apparent Temperature, Relative Humidity (2 m), Precipitation
Tomas2D commented 2 days ago
  1. That is a matter of subject because OpenMeteo is a well-known service. Renaming is a challenging task because, in the future, you can have multiple weather forecast providers in the framework. How would you differentiate between them?
  2. Yes, we can probably remove those extra metadata. Feel free to do a PR. I am just worried about the timezone, as well.
  3. How would you do that?
  4. 👍🏻

What do you think @matoushavlena ?

geneknit commented 2 days ago

@Tomas2D this is a broader issue around LLM capabilities and needing language that an LLM can traverse. LLMs shouldn't be expected to generate a plan to call OpenMeteo, they should be expected to plan to get_weather_forecast. @dakshiagrawal has proposed a layer of abstractions over common tools like search_internet that sits between branded tools and the LLM. This will (a) fix the problem descreibed by @michael-desmond and (b) enable greater flexibliity to add tools (e.g. if a developer pays for a subscription weather service). Off the bat I can think of this being very useful for users to bring their own RAG solutions to a Bee agent.