Benchmarking

[dharhas]

Perform and communicate proper benchmarking to show comparison to other tools. Target using different IBIS backends as well as against Postgresql/PostGIS for geospatial data types. Both numerical benchmarks as well as user experience of (Altair) charting on top of the different backends should be explored. OmniSci will provide access to some large sample datasets for use in the benchmark

[xmnlab]

Added size.

[xmnlab]

Priority 1. Benchmark Pandas against Ibis with OmnisciDB backend. Priority 2. Benchmark Pandas+Dask against Ibis with OmnisciDB backend. Priority 3. Benchmark Ibis with Big Query and OmnisciDB backends.

[xmnlab]

Datasets for the benchmark

• San Fransisco bus dataset
• NYC https://data.cityofnewyork.us/Transportation/2017-Yellow-Taxi-Trip-Data/biws-g3hs

References:

• http://datashader.org/topics/nyc_taxi.html
• https://github.com/MUSA-620-Spring-2019/datashader-nyc-taxi-app
• https://towardsdatascience.com/geospatial-operations-at-scale-with-dask-and-geopandas-4d92d00eb7e8

[xmnlab]

the repo for ibis-benchmark is https://github.com/Quansight/ibis-benchmark

some updates:

• currently works with nyc-taxi dataset
• to download the dataset run: ./scripts/download.sh
• it currently works with omniscidb locally (using docker it raises an memory error message, it needs to be investigated)
• to load nyc-taxi dataset to omniscidb run: ./scripts/load_data.sh
• current it just checks very simple operations (head and tail) and store the result at /tmp/log_benchmark.json, eg:

  {
  "omniscidb": {
  "table_head": 0.03497052192687988,
  "table_tail": 0.024263644218444826
  },
  "pandas": {
  "table_head": 0.15434300899505615,
  "table_tail": 0.1464146375656128
  }
  }

• the benchmark operations are defined at ./ibis_benchmark/benchmark.py

The next step is the adding of more complex expressions for the benchmark

[xmnlab]

@dharhas should this benchmark be initially for CPU only? GPU? both?

[dharhas]

both would make sense to me.

[xmnlab]

the code is currently at https://github.com/quansight/ibis-benchmark and the report page is currently here: https://quansight.github.io/ibis-benchmark/report-nyc-taxi.html

PS: the pandas benchmark is not presented at this benchmark. that was discussed in the meeting and maybe benchmark strategy will change.