parkervg / blendsql

Query language for blending SQL logic and LLM reasoning across structured + unstructured data. [Findings of ACL 2024]
https://parkervg.github.io/blendsql/
Apache License 2.0
72 stars 4 forks source link

`TransformersVisionModel`, `ImageCaption` Ingredient, `default_model` behavior #26

Closed parkervg closed 4 months ago

parkervg commented 4 months ago

This PR adds the ability to pass models to specific ingredients via .from_args(model=model). This passed model will take priority, and if no ingredient-level model is specified, we will fall back to the default_model given to the blend() function.

For example:

ingredients = {ImageCaption.from_args(model=vision_model), LLMMap}
res = blend(
    query="""
    SELECT "Name",
    {{ImageCaption('parks::Image')}} as "Image Description", 
    {{
        LLMMap(
            question='Size in km2?',
            context='parks::Area'
        )
    }} as "Size in km" FROM parks
    WHERE "Location" = 'Alaska'
    ORDER BY "Size in km" DESC LIMIT 1
        """,
    db=db,
    default_model=text_model,
    ingredients=ingredients,
)

Since LLMMap is not given a model in it's initialization, it will be fed the text_model at runtime. The new ImageCaption model, however, needs a TransformersVisionModel, so we pass it via the classmethod from_args.

Benchmark Runtimes

Before: Task Average Runtime # Unique Queries
financials 0.0461312 7
rugby 0.317933 4
1966_nba_draft 0.110849 2
After: Task Average Runtime # Unique Queries
financials 0.0390639 7
rugby 0.323947 4
1966_nba_draft 0.105875 2

Closes #22