"%g" formatter for floats looses a lot of precision on some use cases

OCamlPro / ocplib-json-typed

Libraries for reliable manipulation JSON objects.

Other

24 stars 19 forks source link

"%g" formatter for floats looses a lot of precision on some use cases #25

Open smondet opened 5 years ago

smondet commented 5 years ago

Here: https://github.com/OCamlPro/ocplib-json-typed/blob/master/src/json_repr.ml#L161

(* Example: Half-an-hour on time-stamps :) *)
ocaml# Format.fprintf Format.std_formatter "%g\n%!" (Unix.gettimeofday ());;
1.54543e+09
ocaml# (Unix.gettimeofday () -. 1.54543e+09) /. 60.;;
- : float = 30.1761083324750281

pp is used there:

https://gitlab.com/tezos/tezos/blob/master/src/lib_data_encoding/json.ml#L316 ( https://gitlab.com/tezos/tezos/blob/master/src/lib_stdlib_unix/lwt_utils_unix.ml#L184 )

smondet commented 5 years ago

Cf. also:

ocaml# Ezjsonm.to_string (`A [`Float (Unix.gettimeofday ())]);;
- : string = "[1545432525.245818]"