search

parkervg / blendsql

Query language for blending SQL logic and LLM reasoning across structured + unstructured data. [Findings of ACL 2024]
https://parkervg.github.io/blendsql/
Apache License 2.0
72 stars 4 forks source link

Adding `benchmark/` directory #20

Closed parkervg closed 4 months ago

parkervg commented 4 months ago

Inspired by projects like duckdb, this PR adds a benchmark directory containing queries we can use for doing runtime tests.

You can invoke it via python -m benchmark.run. It will return a markdown table. For example, in this branch, we get the following results.

Task Average Runtime # Unique Queries
financials 0.0497038 7
rugby 0.272745 4
1966_nba_draft 0.355472 2

The idea is that all future PRs will have the before/after benchmark results included.

parkervg commented 4 months ago

Example: we can see that PR #18 by @zvs08 already has a definitive impact on runtime! Especially on the join-hungry '1966_nba_draft' task:

Task Average Runtime # Unique Queries
financials 0.0467936 7
rugby 0.267355 4
1966_nba_draft 0.113532 2