parkervg / blendsql

Query language for blending SQL logic and LLM reasoning across structured + unstructured data. [Findings of ACL 2024]
https://parkervg.github.io/blendsql/
Apache License 2.0
72 stars 4 forks source link

Attach specific `Model` objects to ingredients via `from_args`, enabling README examples #22

Closed parkervg closed 4 months ago

parkervg commented 4 months ago

We have the following example in the README:

Name Image Location Area Recreation Visitors (2022) Description
Death Valley death_valley.jpeg California, Nevada 3,408,395.63 acres (13,793.3 km2) 1,128,862 Death Valley is the hottest, lowest, and driest place in the United States, with daytime temperatures that have exceeded 130 °F (54 °C).
Everglades everglades.jpeg Alaska 7,523,897.45 acres (30,448.1 km2) 9,457 The country's northernmost park protects an expanse of pure wilderness in Alaska's Brooks Range and has no park facilities.
New River Gorge new_river_gorge.jpeg West Virgina 7,021 acres (28.4 km2) 1,593,523 The New River Gorge is the deepest river gorge east of the Mississippi River.
Katmai katmai.jpg Alaska 3,674,529.33 acres (14,870.3 km2) 33,908 This park on the Alaska Peninsula protects the Valley of Ten Thousand Smokes, an ash flow formed by the 1912 eruption of Novarupta.
SELECT {{VQA('Describe this image.', 'parks::Image')}} FROM parks
    WHERE "Location" = 'Alaska'
    ORDER BY {{
        LLMMap(
            'Size in km2?',
            'parks::Area'
        )
    }} LIMIT 1

However, currently we only take one Model object via the blender arg:

blend(
    query=query,
    db=db, 
    blender=blender,
    ingredients={LLMQA, VQA}
)

Instead, we want the following pattern:

blend(
    query=query,
    db=db, 
    blender=text_model,
    ingredients={LLMQA, VQA.from_args(model=vqa_model)}
)

Where text_model is a text-based Model object, and vqa_model is a Model object that takes an image as an input.

For reference on the VQA model logic, see https://parkervg.github.io/blendsql/reference/examples/vqa-ingredient/

Acceptance Criteria