search

tconbeer / textual-fastdatatable

A performance-focused reimplementation of Textual's DataTable widget, with a pluggable data storage backend.
MIT License
15 stars 2 forks source link

Add Polars backend #112

Closed kdkavanagh closed 2 months ago

kdkavanagh commented 2 months ago

Closes #96

tconbeer commented 2 months ago

Thanks for this! I'm away from my computer for a little while but I'll give you a review when I get back

kdkavanagh commented 2 months ago

Updated - I moved the whole class definition under the try. This makes pylint/mypy a bit happier + avoids needing all the if not none checks

kdkavanagh commented 2 months ago

Benchmark results suggest it may be the fastest impl:

Records Built-In DataTable FastDataTable (Arrow from Parquet) FastDataTable (Arrow from Records) FastDataTable (Polars from Parquet)
lap_times_100.parquet 0.008s / 1.304s 0.006s / 1.406s 0.007s / 1.441s 0.023s / 1.145s
lap_times_1000.parquet 0.050s / 1.749s 0.007s / 1.933s 0.005s / 1.866s 0.005s / 1.098s
lap_times_10000.parquet 0.339s / 1.985s 0.006s / 1.909s 0.009s / 1.862s 0.005s / 1.107s
lap_times_100000.parquet 3.171s / 4.755s 0.012s / 1.895s 0.053s / 1.932s 0.013s / 1.096s
lap_times_538121.parquet 17.126s / 18.921s 0.025s / 1.933s 0.272s / 2.138s 0.033s / 1.131s
wide_10000.parquet 2.216s / 5.913s 0.009s / 2.362s 0.023s / 2.291s 0.007s / 1.086s
wide_100000.parquet 22.291s / 26.005s 0.023s / 2.300s 0.184s / 2.552s 0.011s / 1.091s
tconbeer commented 2 months ago

The benchmark results are exciting -- the optional import stuff still needs some work, though.