apache / arrow

Apache Arrow is the universal columnar format and multi-language toolbox for fast data interchange and in-memory analytics
https://arrow.apache.org/
Apache License 2.0
14.48k stars 3.52k forks source link

[Format][Integration] Define how to test for delta dictionary support in the JSON integration test data format #16732

Open asfimport opened 5 years ago

asfimport commented 5 years ago

Currently the integration test JSON format assumes that dictionaries remain constant throughout the stream. It might be better to change the JSON format to more closely mimic the IPC protocol (a sequence of messages tagged with the message type)

follow on to ARROW-3144

Reporter: Wes McKinney / @wesm

Related issues:

Note: This issue was originally created as ARROW-5338. Please see the migration documentation for further details.

asfimport commented 4 years ago

Neal Richardson / @nealrichardson: Proposal: add an attribute to the existing JSON document "ipc_format": "file" to indicate that this is the IPC file format, and create a new JSON document with "ipc_format": "stream" that can appropriately specify delta dictionaries.