search

wader / fq

jq for binary formats - tool, language and decoders for working with binary and text formats
Other
9.72k stars 225 forks source link

[feature] add decimal floating-point number support #689

Open matthewdale opened 1 year ago

matthewdale commented 1 year ago

The BSON and Avro formats support encoding decimal floating-point numbers which cannot be exactly represented by a float64. Currently, neither the bson or avro decoder packages actually support decoding the respective decimal number value, only displaying the values as binary. There are two main things that stand in the way of decoding BSON and Avro decimal number values:

Add support for decoding various decimal floating-point binary formats. Consider using the shopspring/decimal library for the decoded value representation because it can represent almost any decimal value (except NaN, +Inf, -Inf).

wader commented 1 year ago

I wonder if we should ask the gojq author what he thinks of decimal support. He has already extended from original jq to support arbitrarily precision so maybe he has some ideas. Would be nice to not have too much difference compared to upstream.

btw any idea if big.Rat from standard library would be ok also?

matthewdale commented 1 year ago

Good idea, I opened an issue on gojq requesting support for arbitrary precision non-integer decimal values: https://github.com/itchyny/gojq/issues/216

I believe big.Rat can represent all numbers that shopspring/decimal can represent and seems to support the required arithmetic operations (except modulus, which would probably be accomplished by truncating the big.Rat to a big.Int and using that to calculate modulus). The main difficulty with big.Rat in the context of fq is that to get a decimal-with-radix representation (i.e. "123.4" instead of "1234/10") you use FloatString, which requires you to specify (and keep track of) the required number of digits of precision yourself. It's worth an experiment to validate if it's possible.