search

audeering / audb

Manage audio and video databases
https://audeering.github.io/audb/
Other
23 stars 1 forks source link

Benchmarks: lance file format for dependency table #425

Open hagenw opened 1 month ago

hagenw commented 1 month ago

Based on the comment in https://github.com/audeering/audb/pull/424, I benchmarked lance as a file format for storing the dependency table.

Results for writing 1000000 lines.

method csv pickle parquet lance
pd.DataFrame[object] 2.017 0.683 0.491 0.470
pd.DataFrame[string] 2.031 0.641 0.524 0.560
pd.DataFrame[pyarrow] 2.050 0.275 0.272 0.308
pd.DataFrame[object] -> pd.DataFrame[pyarrow] 2.203 0.501
pd.DataFrame[object] -> pa.Table 0.552
pd.DataFrame[string] -> pa.Table 0.603
pd.DataFrame[pyarrow] -> pa.Table 0.272
pd.DataFrame[object] -> pd.DataFrame[pyarrow] -> pa.Table 0.477
pa.Table 0.237 0.290 0.260

Results for reading 1000000 lines.

method csv pickle parquet lance
----> pd.DataFrame[object] 1.126 0.266 0.349 0.671
----> pd.DataFrame[string] 1.206 0.244 0.396 0.678
----> pd.DataFrame[pyarrow] 2.508 0.124 0.385 0.638
----> pd.DataFrame[pyarrow] -> pd.DataFrame[object] 2.714 0.289
-c--> pd.DataFrame[object] 1.165
-c--> pd.DataFrame[string] 1.133
-c--> pd.DataFrame[pyarrow] 2.572
-c--> pd.DataFrame[pyarrow] -> pd.DataFrame[object] 2.664
-pa-> pd.DataFrame[object] 0.426
-pa-> pd.DataFrame[string] 0.415
-pa-> pd.DataFrame[pyarrow] 0.542
-pa-> pd.DataFrame[pyarrow] -> pd.DataFrame[object] 0.801
----> pa.Table -> pd.DataFrame[object] 0.287 0.217
----> pa.Table -> pd.DataFrame[string] 0.305 0.320
----> pa.Table -> pd.DataFrame[pyarrow] 0.131 0.092
----> pa.Table -> pd.DataFrame[pyarrow] -> pd.DataFrame[object] 0.370
----> pa.Table 0.054 0.048 0.332
----> lance.LanceDataset 0.014

As can be seen, reading speed is only better if we read to an lance.LanceDataset object. Which means we also need to check, how the dependency table methods would behave when directly working on a lance dataset.

NOTE: results are not yet added to the README