search

apache / arrow-nanoarrow

Helpers for Arrow C Data & Arrow C Stream interfaces
https://arrow.apache.org/nanoarrow
Apache License 2.0
153 stars 34 forks source link

chore(dev/benchmarks): Benchmark IPC reader #405

Closed paleolimbot closed 3 months ago

paleolimbot commented 3 months ago

This PR adds IPC reader benchmarks via the ArrowArrayStream implementation, which is the most common use. It required a little CMake shuffling to get this to work plus some code to generate IPC stream fixtures. The current benchmarks are:

Benchmark output in details:

# Benchmark Report ## Configurations These benchmarks were run with the following configurations: | preset_name | preset_description | |:------------|:-------------------------------------------------| | local | Uses the nanoarrow C sources from this checkout. | | v0.4.0 | Uses the nanoarrow C sources the 0.4.0 release. | ## Summary A quick and dirty summary of benchmark results between this checkout and the last released version. | benchmark_label | v0.4.0 | local | change | pct_change | |:--------------------------------------------------------------|---------:|---------:|---------:|-----------:| | [ArrayAppendInt16](#arrayappendint16) | 2.67ms | 2.67ms | 3.03µs | 0.1% | | [ArrayAppendInt32](#arrayappendint32) | 3.13ms | 3.14ms | 18.1µs | 0.6% | | [ArrayAppendInt64](#arrayappendint64) | 3.86ms | 3.44ms | 1ns | -10.7% | | [ArrayAppendInt8](#arrayappendint8) | 2.4ms | 2.41ms | 9.04µs | 0.4% | | [ArrayAppendNulls](#arrayappendnulls) | 12.14ms | 12.13ms | 1ns | -0.1% | | [ArrayAppendString](#arrayappendstring) | 8.44ms | 8.79ms | 345.74µs | 4.1% | | [ArrayViewGetInt16](#arrayviewgetint16) | 633.71µs | 631µs | 1ns | -0.4% | | [ArrayViewGetInt32](#arrayviewgetint32) | 627.51µs | 633.47µs | 5.96µs | 1% | | [ArrayViewGetInt64](#arrayviewgetint64) | 677.86µs | 680.66µs | 2.8µs | 0.4% | | [ArrayViewGetInt8](#arrayviewgetint8) | 945.2µs | 943.48µs | 1ns | -0.2% | | [ArrayViewGetString](#arrayviewgetstring) | 1.25ms | 1.26ms | 3.48µs | 0.3% | | [ArrayViewIsNull](#arrayviewisnull) | 1.2ms | 1.19ms | 1ns | -0.4% | | [ArrayViewIsNullNonNullable](#arrayviewisnullnonnullable) | 952.93µs | 941.49µs | 1ns | -1.2% | | [IpcReadManyBatchesFromBuffer](#ipcreadmanybatchesfrombuffer) | 6.1ms | 5.47ms | 1ns | -10.4% | | [IpcReadManyBatchesFromFile](#ipcreadmanybatchesfromfile) | 6.99ms | 6.28ms | 1ns | -10.2% | | [IpcReadManyColumnsFromFile](#ipcreadmanycolumnsfromfile) | 8.55ms | 8.67ms | 117.82µs | 1.4% | | [SchemaInitWideStruct](#schemainitwidestruct) | 1.02ms | 1.02ms | 4.79µs | 0.5% | | [SchemaViewInitWideStruct](#schemaviewinitwidestruct) | 104.11µs | 104.31µs | 198.06ns | 0.2% | ## ArrowArray-related benchmarks Benchmarks for producing ArrowArrays using the ArrowArrayXXX() functions. ### ArrayAppendString Use ArrowArrayAppendString() to build a string array. [View Source](https://github.com/paleolimbot/arrow-nanoarrow/blob/benchmark-ipc/dev/benchmarks/c/array_benchmark.cc#L289-L314) | preset_name | iterations | real_time | cpu_time | items_per_second | |:------------|-----------:|----------:|---------:|-----------------:| | local | 81 | 8.79ms | 8.76ms | 114,185,666 | | v0.4.0 | 81 | 8.44ms | 8.41ms | 118,902,224 | ### ArrayAppendInt8 Use ArrowArrayAppendInt() to build an int8 array. [View Source](https://github.com/paleolimbot/arrow-nanoarrow/blob/benchmark-ipc/dev/benchmarks/c/array_benchmark.cc#L338-L340) | preset_name | iterations | real_time | cpu_time | items_per_second | |:------------|-----------:|----------:|---------:|-----------------:| | local | 294 | 2.41ms | 2.4ms | 415,884,526 | | v0.4.0 | 291 | 2.4ms | 2.4ms | 417,592,258 | ### ArrayAppendInt16 Use ArrowArrayAppendInt() to build an int16 array. [View Source](https://github.com/paleolimbot/arrow-nanoarrow/blob/benchmark-ipc/dev/benchmarks/c/array_benchmark.cc#L343-L345) | preset_name | iterations | real_time | cpu_time | items_per_second | |:------------|-----------:|----------:|---------:|-----------------:| | local | 262 | 2.67ms | 2.67ms | 375,204,429 | | v0.4.0 | 260 | 2.67ms | 2.66ms | 375,796,399 | ### ArrayAppendInt32 Use ArrowArrayAppendInt() to build an int32 array. [View Source](https://github.com/paleolimbot/arrow-nanoarrow/blob/benchmark-ipc/dev/benchmarks/c/array_benchmark.cc#L348-L350) | preset_name | iterations | real_time | cpu_time | items_per_second | |:------------|-----------:|----------:|---------:|-----------------:| | local | 224 | 3.14ms | 3.14ms | 318,552,407 | | v0.4.0 | 223 | 3.13ms | 3.12ms | 320,376,521 | ### ArrayAppendInt64 Use ArrowArrayAppendInt() to build an int64 array. [View Source](https://github.com/paleolimbot/arrow-nanoarrow/blob/benchmark-ipc/dev/benchmarks/c/array_benchmark.cc#L353-L355) | preset_name | iterations | real_time | cpu_time | items_per_second | |:------------|-----------:|----------:|---------:|-----------------:| | local | 224 | 3.44ms | 3.43ms | 291,583,520 | | v0.4.0 | 194 | 3.86ms | 3.82ms | 261,618,420 | ### ArrayAppendNulls Use ArrowArrayAppendNulls() to build an int32 array that contains 80% null values. [View Source](https://github.com/paleolimbot/arrow-nanoarrow/blob/benchmark-ipc/dev/benchmarks/c/array_benchmark.cc#L378-L400) | preset_name | iterations | real_time | cpu_time | items_per_second | |:------------|-----------:|----------:|---------:|-----------------:| | local | 57 | 12.1ms | 12.1ms | 82,528,559 | | v0.4.0 | 57 | 12.1ms | 12.1ms | 82,489,624 | ## ArrowArrayView-related benchmarks Benchmarks for consuming ArrowArrays using the ArrowArrayViewXXX() functions. ### ArrayViewGetInt8 Use ArrowArrayViewGet() to consume an int8 array. [View Source](https://github.com/paleolimbot/arrow-nanoarrow/blob/benchmark-ipc/dev/benchmarks/c/array_benchmark.cc#L122-L124) | preset_name | iterations | real_time | cpu_time | items_per_second | |:------------|-----------:|----------:|---------:|-----------------:| | local | 742 | 943µs | 943µs | 1,060,960,927 | | v0.4.0 | 745 | 945µs | 943µs | 1,059,924,880 | ### ArrayViewGetInt16 Use ArrowArrayViewGet() to consume an int16 array. [View Source](https://github.com/paleolimbot/arrow-nanoarrow/blob/benchmark-ipc/dev/benchmarks/c/array_benchmark.cc#L127-L129) | preset_name | iterations | real_time | cpu_time | items_per_second | |:------------|-----------:|----------:|---------:|-----------------:| | local | 1115 | 631µs | 630µs | 1,586,526,901 | | v0.4.0 | 1115 | 634µs | 633µs | 1,580,582,789 | ### ArrayViewGetInt32 Use ArrowArrayViewGet() to consume an int32 array. [View Source](https://github.com/paleolimbot/arrow-nanoarrow/blob/benchmark-ipc/dev/benchmarks/c/array_benchmark.cc#L132-L134) | preset_name | iterations | real_time | cpu_time | items_per_second | |:------------|-----------:|----------:|---------:|-----------------:| | local | 1111 | 633µs | 633µs | 1,580,851,070 | | v0.4.0 | 1115 | 628µs | 627µs | 1,596,033,249 | ### ArrayViewGetInt64 Use ArrowArrayViewGet() to consume an int64 array. [View Source](https://github.com/paleolimbot/arrow-nanoarrow/blob/benchmark-ipc/dev/benchmarks/c/array_benchmark.cc#L137-L139) | preset_name | iterations | real_time | cpu_time | items_per_second | |:------------|-----------:|----------:|---------:|-----------------:| | local | 1029 | 681µs | 680µs | 1,471,227,424 | | v0.4.0 | 1034 | 678µs | 677µs | 1,476,773,664 | ### ArrayViewIsNullNonNullable Use ArrowArrayViewIsNull() to check for nulls while consuming an int32 array that does not contain a validity buffer. [View Source](https://github.com/paleolimbot/arrow-nanoarrow/blob/benchmark-ipc/dev/benchmarks/c/array_benchmark.cc#L143-L172) | preset_name | iterations | real_time | cpu_time | items_per_second | |:------------|-----------:|----------:|---------:|-----------------:| | local | 746 | 941µs | 940µs | 1,063,268,768 | | v0.4.0 | 735 | 953µs | 951µs | 1,051,845,237 | ### ArrayViewIsNull Use ArrowArrayViewIsNull() to check for nulls while consuming an int32 array that contains 20% nulls. [View Source](https://github.com/paleolimbot/arrow-nanoarrow/blob/benchmark-ipc/dev/benchmarks/c/array_benchmark.cc#L176-L215) | preset_name | iterations | real_time | cpu_time | items_per_second | |:------------|-----------:|----------:|---------:|-----------------:| | local | 589 | 1.19ms | 1.19ms | 838,600,093 | | v0.4.0 | 579 | 1.2ms | 1.2ms | 835,530,389 | ### ArrayViewGetString Use ArrowArrayViewGetStringUnsafe() to consume a string array. [View Source](https://github.com/paleolimbot/arrow-nanoarrow/blob/benchmark-ipc/dev/benchmarks/c/array_benchmark.cc#L218-L249) | preset_name | iterations | real_time | cpu_time | items_per_second | |:------------|-----------:|----------:|---------:|-----------------:| | local | 559 | 1.26ms | 1.25ms | 797,174,096 | | v0.4.0 | 558 | 1.25ms | 1.25ms | 799,731,703 | ## IPC Reader Benchmarks Benchmarks for the ArrowArrayStream IPC reader. ### IpcReadManyBatchesFromFile Use the ArrowArrayStream IPC reader to read 10,000 batches with 5 elements each from a file. [View Source](https://github.com/paleolimbot/arrow-nanoarrow/blob/benchmark-ipc/dev/benchmarks/c/ipc_benchmark.cc#L93-L113) | preset_name | iterations | real_time | cpu_time | items_per_second | |:------------|-----------:|----------:|---------:|-----------------:| | local | 110 | 6.28ms | 6.26ms | 1,596,234 | | v0.4.0 | 101 | 6.99ms | 6.98ms | 1,432,594 | ### IpcReadManyBatchesFromBuffer Use the ArrowArrayStream IPC reader to read 10,000 batches with 5 elements each from a file. [View Source](https://github.com/paleolimbot/arrow-nanoarrow/blob/benchmark-ipc/dev/benchmarks/c/ipc_benchmark.cc#L117-L147) | preset_name | iterations | real_time | cpu_time | items_per_second | |:------------|-----------:|----------:|---------:|-----------------:| | local | 127 | 5.47ms | 5.46ms | 1,831,415 | | v0.4.0 | 114 | 6.1ms | 6.1ms | 1,640,075 | ### IpcReadManyColumnsFromFile Use the ArrowArrayStream IPC reader to read 10,000 batches with 5 elements each from a file. [View Source](https://github.com/paleolimbot/arrow-nanoarrow/blob/benchmark-ipc/dev/benchmarks/c/ipc_benchmark.cc#L151-L171) | preset_name | iterations | real_time | cpu_time | items_per_second | |:------------|-----------:|----------:|---------:|-----------------:| | local | 82 | 8.67ms | 8.66ms | 14,083.85 | | v0.4.0 | 83 | 8.55ms | 8.55ms | 14,098.53 | ## Schema-related benchmarks Benchmarks for producing and consuming ArrowSchema. ### SchemaInitWideStruct Benchmark ArrowSchema creation for very wide tables. Simulates part of the process of creating a very wide table with a simple column type (integer). [View Source](https://github.com/paleolimbot/arrow-nanoarrow/blob/benchmark-ipc/dev/benchmarks/c/schema_benchmark.cc#L45-L56) | preset_name | iterations | real_time | cpu_time | items_per_second | |:------------|-----------:|----------:|---------:|-----------------:| | local | 688 | 1.02ms | 1.02ms | 9,786,782 | | v0.4.0 | 686 | 1.02ms | 1.02ms | 9,829,446 | ### SchemaViewInitWideStruct Benchmark ArrowSchema parsing for very wide tables. Simulates part of the process of consuming a very wide table. Typically the ArrowSchemaViewInit() is done by ArrowArrayViewInit() but uses a similar pattern. [View Source](https://github.com/paleolimbot/arrow-nanoarrow/blob/benchmark-ipc/dev/benchmarks/c/schema_benchmark.cc#L78-L91) | preset_name | iterations | real_time | cpu_time | items_per_second | |:------------|-----------:|----------:|---------:|-----------------:| | local | 6666 | 104µs | 104µs | 96,339,782 | | v0.4.0 | 6733 | 104µs | 104µs | 96,151,786 |
codecov-commenter commented 3 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 88.76%. Comparing base (bd16060) to head (735d761).

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #405 +/- ## ======================================= Coverage 88.76% 88.76% ======================================= Files 82 82 Lines 14633 14633 ======================================= Hits 12989 12989 Misses 1644 1644 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

pitrou commented 3 months ago

Question: does Nanoarrow IPC support compressed buffers?

paleolimbot commented 3 months ago

Question: does Nanoarrow IPC support compressed buffers?

It does not currently support compressed buffers (nor writing)

paleolimbot commented 3 months ago

I'll wait for https://github.com/apache/arrow-nanoarrow/pull/410 to merge for this one (to make sure CI is passing).