anthoRx / parquet-sql

Apache License 2.0
3 stars 1 forks source link

Decimal(20, 0) seems misencoded #7

Closed nsvir closed 1 week ago

nsvir commented 4 years ago

Currrently converted to Binary and schema Decimal(20, 0): image

Spark displays it right: image

But ParquetViewer displays it wrong: image

Needs to be confirmed with datafactory behavior

nsvir commented 4 years ago

Some SQL columns contains a BigDecimal >18: image

Which is converted to string during parquet-ization: image

Wrapped into String type in OrderLegacy object in legacy-acl: image

Then casted into IntegerType: image

nsvir commented 4 years ago

Based on this analyse, I suggest to keep this behavior to reduce rework side effects. But this library might not be copmpliant to parquet-format documentation