search

apache / arrow

Apache Arrow is the universal columnar format and multi-language toolbox for fast data interchange and in-memory analytics
https://arrow.apache.org/
Apache License 2.0
14.47k stars 3.52k forks source link

[Format] Support 32- and 64-bit decimals in IPC messages #17991

Open asfimport opened 6 years ago

asfimport commented 6 years ago

Some systems may wish to send 32- or 64-bit decimal values on the wire without any conversions. Computations engines using Arrow may choose to promote these to 128-bit right away on receipt, but it would be useful to be enable systems with native handling of the smaller-size types to be able to send data via Arrow streams / IPC without extra serialization

Reporter: Wes McKinney / @wesm

Note: This issue was originally created as ARROW-2009. Please see the migration documentation for further details.

asfimport commented 5 years ago

Robert Fox: +1 to optimize decimal storage.  I use decimal a lot to store monetary values (no tolerance for lost precision).  All of my arrow files/streams are huge!  I end up using compression more than I would like.