search

jorgecarleitao / arrow2

Transmute-free Rust library to work with the Arrow format
Apache License 2.0
1.06k stars 222 forks source link

Negative scale for Decimal and Decimal256 #1518

Open xcharleslin opened 1 year ago

xcharleslin commented 1 year ago

Analogous to https://github.com/apache/arrow-rs/issues/1785.

Currently, Arrow2 Decimal and Decimal256 have scale typed as usize, which would not support negative scale.

As a driveby, precision is also currently typed as usize but could probably be typed as u8. It seems the max value for precision on pyarrow is 76.

xcharleslin commented 1 year ago

Somewhat related, currently impl From<PrimitiveType> for DataType has this https://github.com/jorgecarleitao/arrow2/blame/8ee5ad8c774e7355218376caf73d94bce2a769a6/src/datatypes/mod.rs#L493-L494 

    Int128 => Decimal(32, 32),
    Int256 => Decimal256(32, 32),

Would Decimal(32, 32) only support 0.000000 to 0.999999?

Also, would more precision be required to support Int128 and Int256? The maximum values for these are 39 and 78 digits long respectively (although it seems PyArrow only supports 38 and 76 for some reason). 

    Int128 => Decimal(38, 0),
    Int256 => Decimal256(76, 0),