float conversion on dump/load

[rtgiskard]

pytomlpp.dumps({'a': 0.4})    # 'a = 0.40000000000000002'
tomlkit.dumps({'a': 0.4})    # 'a = 0.4\n'
toml.dumps({'a': 0.4})    # 'a = 0.4\n'

pytomlpp.loads('a = 0.40000000000000002')    # {'a': 0.4}
tomlkit.loads('a = 0.40000000000000002')    # {'a': 0.4}
toml.loads('a = 0.40000000000000002')    # {'a': 0.4}

pytomlpp.dumps({'a': 0.40000000000000001})    # 'a = 0.40000000000000002'
tomlkit.dumps({'a': 0.40000000000000001})    # 'a = 0.4\n'
toml.dumps({'a': 0.40000000000000001})    # 'a = 0.4\n'

not clear about the underlying conversion, is it possible to keep consistent with other python implementation?

[colinxu2020]

I can't reproduce this problem. Please see the screenshot I sent.

[rtgiskard]

Strange, I'm using the latest archlinux, and this is the environment of my system

here is the log:

~ $ ipython
Python 3.11.3 (main, Jun  5 2023, 09:32:32) [GCC 13.1.1 20230429]
Type 'copyright', 'credits' or 'license' for more information
IPython 8.14.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import pytomlpp

In [2]: pytomlpp.dumps({'a': 0.4})
Out[2]: 'a = 0.40000000000000002'

In [3]: pytomlpp.loads('a = 0.40000000000000002')
Out[3]: {'a': 0.4}

In [4]: pytomlpp.lib_version
Out[4]: '3.3.0'

~ $ ldd -v .local/lib/python3.11/site-packages/pytomlpp/_impl.cpython-311-x86_64-linux-gnu.so 
    linux-vdso.so.1 (0x00007ffde85f0000)
    libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00007fba44e00000)
    libm.so.6 => /usr/lib/libm.so.6 (0x00007fba45083000)
    libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007fba44ddb000)
    libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007fba4507e000)
    libc.so.6 => /usr/lib/libc.so.6 (0x00007fba44a00000)
    /usr/lib64/ld-linux-x86-64.so.2 (0x00007fba45224000)

    Version information:
    .local/lib/python3.11/site-packages/pytomlpp/_impl.cpython-311-x86_64-linux-gnu.so:
        libgcc_s.so.1 (GCC_3.0) => /usr/lib/libgcc_s.so.1
        libpthread.so.0 (GLIBC_2.2.5) => /usr/lib/libpthread.so.0
        libc.so.6 (GLIBC_2.14) => /usr/lib/libc.so.6
        libc.so.6 (GLIBC_2.4) => /usr/lib/libc.so.6
        libc.so.6 (GLIBC_2.2.5) => /usr/lib/libc.so.6
        libstdc++.so.6 (GLIBCXX_3.4.18) => /usr/lib/libstdc++.so.6
        libstdc++.so.6 (GLIBCXX_3.4.9) => /usr/lib/libstdc++.so.6
        libstdc++.so.6 (CXXABI_1.3.2) => /usr/lib/libstdc++.so.6
        libstdc++.so.6 (CXXABI_1.3.3) => /usr/lib/libstdc++.so.6
        libstdc++.so.6 (GLIBCXX_3.4.11) => /usr/lib/libstdc++.so.6
        libstdc++.so.6 (CXXABI_1.3.5) => /usr/lib/libstdc++.so.6
        libstdc++.so.6 (CXXABI_1.3) => /usr/lib/libstdc++.so.6
        libstdc++.so.6 (GLIBCXX_3.4) => /usr/lib/libstdc++.so.6
        libm.so.6 (GLIBC_2.2.5) => /usr/lib/libm.so.6
    /usr/lib/libstdc++.so.6:
        libm.so.6 (GLIBC_2.2.5) => /usr/lib/libm.so.6
        ld-linux-x86-64.so.2 (GLIBC_2.3) => /usr/lib64/ld-linux-x86-64.so.2
        libgcc_s.so.1 (GCC_4.2.0) => /usr/lib/libgcc_s.so.1
        libgcc_s.so.1 (GCC_3.4) => /usr/lib/libgcc_s.so.1
        libgcc_s.so.1 (GCC_3.3) => /usr/lib/libgcc_s.so.1
        libgcc_s.so.1 (GCC_4.3.0) => /usr/lib/libgcc_s.so.1
        libgcc_s.so.1 (GCC_3.0) => /usr/lib/libgcc_s.so.1
        libc.so.6 (GLIBC_2.14) => /usr/lib/libc.so.6
        libc.so.6 (GLIBC_2.38) => /usr/lib/libc.so.6
        libc.so.6 (GLIBC_2.6) => /usr/lib/libc.so.6
        libc.so.6 (GLIBC_2.26) => /usr/lib/libc.so.6
        libc.so.6 (GLIBC_2.33) => /usr/lib/libc.so.6
        libc.so.6 (GLIBC_2.25) => /usr/lib/libc.so.6
        libc.so.6 (GLIBC_2.18) => /usr/lib/libc.so.6
        libc.so.6 (GLIBC_2.16) => /usr/lib/libc.so.6
        libc.so.6 (GLIBC_2.32) => /usr/lib/libc.so.6
        libc.so.6 (GLIBC_2.4) => /usr/lib/libc.so.6
        libc.so.6 (GLIBC_2.7) => /usr/lib/libc.so.6
        libc.so.6 (GLIBC_2.3.4) => /usr/lib/libc.so.6
        libc.so.6 (GLIBC_2.17) => /usr/lib/libc.so.6
        libc.so.6 (GLIBC_2.3) => /usr/lib/libc.so.6
        libc.so.6 (GLIBC_2.36) => /usr/lib/libc.so.6
        libc.so.6 (GLIBC_2.3.2) => /usr/lib/libc.so.6
        libc.so.6 (GLIBC_2.34) => /usr/lib/libc.so.6
        libc.so.6 (GLIBC_2.2.5) => /usr/lib/libc.so.6
    /usr/lib/libm.so.6:
        ld-linux-x86-64.so.2 (GLIBC_PRIVATE) => /usr/lib64/ld-linux-x86-64.so.2
        libc.so.6 (GLIBC_ABI_DT_RELR) => /usr/lib/libc.so.6
        libc.so.6 (GLIBC_2.35) => /usr/lib/libc.so.6
        libc.so.6 (GLIBC_2.14) => /usr/lib/libc.so.6
        libc.so.6 (GLIBC_2.34) => /usr/lib/libc.so.6
        libc.so.6 (GLIBC_2.4) => /usr/lib/libc.so.6
        libc.so.6 (GLIBC_2.3.2) => /usr/lib/libc.so.6
        libc.so.6 (GLIBC_PRIVATE) => /usr/lib/libc.so.6
        libc.so.6 (GLIBC_2.2.5) => /usr/lib/libc.so.6
    /usr/lib/libgcc_s.so.1:
        libc.so.6 (GLIBC_2.35) => /usr/lib/libc.so.6
        libc.so.6 (GLIBC_2.14) => /usr/lib/libc.so.6
        libc.so.6 (GLIBC_2.34) => /usr/lib/libc.so.6
        libc.so.6 (GLIBC_2.3.2) => /usr/lib/libc.so.6
        libc.so.6 (GLIBC_2.2.5) => /usr/lib/libc.so.6
    /usr/lib/libpthread.so.0:
        libc.so.6 (GLIBC_ABI_DT_RELR) => /usr/lib/libc.so.6
        libc.so.6 (GLIBC_2.2.5) => /usr/lib/libc.so.6
    /usr/lib/libc.so.6:
        ld-linux-x86-64.so.2 (GLIBC_2.2.5) => /usr/lib64/ld-linux-x86-64.so.2
        ld-linux-x86-64.so.2 (GLIBC_2.3) => /usr/lib64/ld-linux-x86-64.so.2
        ld-linux-x86-64.so.2 (GLIBC_PRIVATE) => /usr/lib64/ld-linux-x86-64.so.2

Any advice on how to debug the issue, or any other info that I should provide?

[marzer]

pytomlpp is backed by a C++ library, so this would be down to differences in how the C++ implementation handles floats, versus what python does.

For more context: I'm the author of the backing C++ library, toml++, and the way that I've written float handling is to exactly round-trip floats, regardless of how small the decimal component is. I suspect what's happening here is that there's some precision loss or rounding happening on in the pure-python libraries, giving you 0.4 instead of the (actual) value of 0.40000000000000002.

Personally I don't see this as a problem worth solving; even things as simple as compiler floating-point optimizations can break this from one build to the next so it's a bit of a crapshoot.

[rtgiskard]

Agreed, so the point is that one should avoid expecting float number to be serialized with exactly the same literal string as its wrote manually.

[marzer]

so the point is that one should avoid expecting float number to be serialized with exactly the same literal string as its wrote manually.

Yep, with decimal representations, at least. This is a pretty common problem across the serialization space, and is usually solved via hexfloats, which TOML doesn't currently support (there's a proposal in discussion: https://github.com/toml-lang/toml/issues/562)

[bobfang1992]

Thanks @marzer for clarification. I will close this for now. Looking forward for the hexfloat support and then I will update the underlying.