finos / kdb-studio

Studio for kdb+ / Rapid execution environment for q
Apache License 2.0
18 stars 12 forks source link

studio renders small floats inconsistently with q #144

Open adavies42 opened 1 year ago

adavies42 commented 1 year ago

studio rounds very small floats to zero, which q never does

e.g. enter 7.275958e-12 in studio and run it, and it returns 0f, while the q console would say 7.275958e-12 for the same input

gyorokpeter commented 11 months ago

Apparently q switches to exponential notation if the exponent is less than -4 or greater than 16. Do we want to mirror this behavior in studio? Or should it be configurable, like there is already a "decimal digits" setting? Using that setting it's possible to view this number as non-zero, although it still uses the full decimal notation, e.g. setting it to 20 gives 0.000000000007275958f.