parkervg / blendsql

Query language for blending SQL logic and LLM reasoning across structured + unstructured data. [Findings of ACL 2024]
https://parkervg.github.io/blendsql/
Apache License 2.0
72 stars 4 forks source link

`SmoothieMeta` only counts stats for the `default_model` #33

Open parkervg opened 1 week ago

parkervg commented 1 week ago

Describe the bug Currently, logic in the main _blend() function only considers the default_model object when aggregating stats in the SmoothieMeta object.

Any models specified via a from_args() call to an ingredient, then, are ignored.

To Reproduce

model = OpenaiLLM(
    "gpt-3.5-turbo",
    config={
        "temperature": 0.0
    },
    caching=False
)

query = """
SELECT DISTINCT venue FROM w
      WHERE city = 'sydney' AND {{
          LLMMap(
              'More than 30 total points?',
              'w::score'
          )
      }} = TRUE
"""
ingredients = {
    LLMMap.from_args(
        model=TransformersLLM(
            "HuggingFaceTB/SmolLM-135M-Instruct", caching=False, config={"chat_template": ChatMLTemplate, "device_map": "cpu"},
        )
    )
}
db = SQLite(
    fetch_from_hub("1884_New_Zealand_rugby_union_tour_of_New_South_Wales_1.db")
)

smoothie = blend(query=query, db=db, ingredients=ingredients, default_model=model, verbose=True)
# Both these are empty 
print(smoothie.meta.raw_prompts)
print(smoothie.meta.prompts)

Expected behavior We should track usages across all models invoked during a BlendSQL execution.