Open whscullin opened 2 months ago
There are issues with decimals in arrow in the current version. We started fixing it in the current main branch and I hope to make some time to fix the remaining issues before the next release.
Please use float for now.
That's not something we can really control. We may have some ability to automatically cast some numbers to float, but we don't control all the input SQL, and there's no way to programmatically force floats in the output that I've found.
Let's fix how Arrow handles decimal for sure. If you can help out, that would be great or you can follow the respective issues over on the arrow repo. I just described the plan in https://github.com/apache/arrow/issues/37920#issuecomment-2058022504.
What happens?
Numeric values from duckdb-wasm can be returned in several formats, including Arrow's
DecimalBigNum
. When a value with a decimal point is returned viaDecimalBigNum
, it loses its decimal point.To Reproduce
With this snippet, where
connection
is a DuckDB wasm connection:the values returned become:
n1
, as a constant, has lost its decimal point. It is returned as aDecimalBigNum
.n2
has the correct decimal placement, and it is returned as a Javascriptnumber
. If you do any computation withn1
, such as1.234/1
, it gets returned as anumber
and is correct.This may be an Arrow issue with a mismatch between the IPC serialization of of
DecimalBigNum
between Rust and Javascript, but I don't have an easy way to test that, yet.Browser/Environment:
node v18.16.0
Device:
MacBook Pro
DuckDB-Wasm Version:
1.28.1-dev179.0
DuckDB-Wasm Deployment:
Malloy VS Code Extension
Full Name:
Will Scullin
Affiliation:
Google