rapidsai / cudf

cuDF - GPU DataFrame Library
https://docs.rapids.ai/api/cudf/stable/
Apache License 2.0
8.24k stars 884 forks source link

[BUG] casting from `float32` to `Decimal64Dtype` is resulting in incorrect values #14169

Closed galipremsagar closed 2 months ago

galipremsagar commented 11 months ago

Describe the bug When there is a type-cast from float32 to Decimal64Dtype, the decimal values yield incorrect results. This issue was surfaced when testing arrow-13. arrow-12 has the same bug and it was fixed in arrow-13, we have similar issue in our libcudf too.

Steps/Code to reproduce bug

IIn [20]: import pyarrow as pa # This is pyarrow-13, not 12

In [21]: import cudf

In [22]: data = [14.12302, 97938.2, None, 0.0, -8.302014, None, 94.31304, -112.2314, 0.3333333]

In [23]: s = cudf.Series(data, dtype='float32')

In [24]: s
Out[24]: 
0     14.12302017
1     97938.20313
2            <NA>
3             0.0
4    -8.302014351
5            <NA>
6     94.31304169
7    -112.2313995
8     0.333333313
dtype: float32

In [25]: parray = pa.array(data, type=pa.float32())

In [26]: parray
Out[26]: 
<pyarrow.lib.FloatArray object at 0x7ff00e3bf700>
[
  14.12302,
  97938.2,
  null,
  0,
  -8.302014,
  null,
  94.31304,
  -112.2314,
  0.3333333
]

In [27]: decimal_dtype = cudf.Decimal64Dtype(11, 4)

In [28]: s.astype(decimal_dtype)
Out[28]: 
0       14.1230
1    97938.2016      # Incorrect, it should be 97938.2031
2          None
3        0.0000
4       -8.3020
5          None
6       94.3130
7     -112.2314
8        0.3333
dtype: decimal64

In [29]: parray.cast(decimal_dtype.to_arrow())
Out[29]: 
<pyarrow.lib.Decimal128Array object at 0x7ff00e1bebc0>
[
  14.1230,
  97938.2031,
  null,
  0.0000,
  -8.3020,
  null,
  94.3130,
  -112.2314,
  0.3333
]

Expected behavior

In [28]: s.astype(decimal_dtype)
Out[28]: 
0       14.1230
1    97938.2031      # Incorrect, it should be 97938.2031
2          None
3        0.0000
4       -8.3020
5          None
6       94.3130
7     -112.2314
8        0.3333
dtype: decimal64

Environment overview (please complete the following information)

Environment details Please run and paste the output of the cudf/print_env.sh script here, to gather any other relevant environment details

Click here to see environment details

     **git***
     commit 4b2a8e2ccdeff36ff58d28d80c8742a474e1d69a (HEAD -> arrow_13)
     Author: galipremsagar 
     Date:   Thu Sep 21 23:13:31 2023 -0700

     Upgrade to arrow 13
     **git submodules***

     ***OS Information***
     DISTRIB_ID=Ubuntu
     DISTRIB_RELEASE=18.04
     DISTRIB_CODENAME=bionic
     DISTRIB_DESCRIPTION="Ubuntu 18.04.4 LTS"
     NAME="Ubuntu"
     VERSION="18.04.4 LTS (Bionic Beaver)"
     ID=ubuntu
     ID_LIKE=debian
     PRETTY_NAME="Ubuntu 18.04.4 LTS"
     VERSION_ID="18.04"
     HOME_URL="https://www.ubuntu.com/"
     SUPPORT_URL="https://help.ubuntu.com/"
     BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
     PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
     VERSION_CODENAME=bionic
     UBUNTU_CODENAME=bionic
     Linux dt07 4.15.0-76-generic #86-Ubuntu SMP Fri Jan 17 17:24:28 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

     ***GPU Information***
     Fri Sep 22 03:31:47 2023
     +---------------------------------------------------------------------------------------+
     | NVIDIA-SMI 530.30.02              Driver Version: 530.30.02    CUDA Version: 12.1     |
     |-----------------------------------------+----------------------+----------------------+
     | GPU  Name                  Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
     | Fan  Temp  Perf            Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
     |                                         |                      |               MIG M. |
     |=========================================+======================+======================|
     |   0  Tesla T4                        On | 00000000:3B:00.0 Off |                    0 |
     | N/A   46C    P0               27W /  70W|   1348MiB / 15360MiB |      0%      Default |
     |                                         |                      |                  N/A |
     +-----------------------------------------+----------------------+----------------------+
     |   1  Tesla T4                        On | 00000000:5E:00.0 Off |                    0 |
     | N/A   34C    P8                9W /  70W|      4MiB / 15360MiB |      0%      Default |
     |                                         |                      |                  N/A |
     +-----------------------------------------+----------------------+----------------------+
     |   2  Tesla T4                        On | 00000000:AF:00.0 Off |                    0 |
     | N/A   29C    P8               10W /  70W|      4MiB / 15360MiB |      0%      Default |
     |                                         |                      |                  N/A |
     +-----------------------------------------+----------------------+----------------------+
     |   3  Tesla T4                        On | 00000000:D8:00.0 Off |                    0 |
     | N/A   28C    P8               10W /  70W|      4MiB / 15360MiB |      0%      Default |
     |                                         |                      |                  N/A |
     +-----------------------------------------+----------------------+----------------------+

     +---------------------------------------------------------------------------------------+
     | Processes:                                                                            |
     |  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
     |        ID   ID                                                             Usage      |
     |=======================================================================================|
     +---------------------------------------------------------------------------------------+

     ***CPU***
     Architecture:        x86_64
     CPU op-mode(s):      32-bit, 64-bit
     Byte Order:          Little Endian
     CPU(s):              64
     On-line CPU(s) list: 0-63
     Thread(s) per core:  2
     Core(s) per socket:  16
     Socket(s):           2
     NUMA node(s):        2
     Vendor ID:           GenuineIntel
     CPU family:          6
     Model:               85
     Model name:          Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz
     Stepping:            4
     CPU MHz:             1194.971
     BogoMIPS:            4200.00
     Virtualization:      VT-x
     L1d cache:           32K
     L1i cache:           32K
     L2 cache:            1024K
     L3 cache:            22528K
     NUMA node0 CPU(s):   0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54,56,58,60,62
     NUMA node1 CPU(s):   1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55,57,59,61,63
     Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts pku ospke md_clear flush_l1d arch_capabilities

     ***CMake***
     /nvme/0/pgali/envs/cudfdev/bin/cmake
     cmake version 3.27.6

     CMake suite maintained and supported by Kitware (kitware.com/cmake).

     ***g++***
     /nvme/0/pgali/envs/cudfdev/bin/g++
     g++ (conda-forge gcc 11.4.0-2) 11.4.0
     Copyright (C) 2021 Free Software Foundation, Inc.
     This is free software; see the source for copying conditions.  There is NO
     warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

     ***nvcc***
     /nvme/0/pgali/envs/cudfdev/bin/nvcc
     nvcc: NVIDIA (R) Cuda compiler driver
     Copyright (c) 2005-2022 NVIDIA Corporation
     Built on Wed_Sep_21_10:33:58_PDT_2022
     Cuda compilation tools, release 11.8, V11.8.89
     Build cuda_11.8.r11.8/compiler.31833905_0

     ***Python***
     /nvme/0/pgali/envs/cudfdev/bin/python
     Python 3.10.12

     ***Environment Variables***
     PATH                            : /nvme/0/pgali/envs/cudfdev/bin:/nvme/0/pgali/envs/cudfdev/bin:/nvme/0/pgali/anaconda3/bin:/nvme/0/pgali/envs/cudfdev/bin:/nvme/0/pgali/.cargo/bin:/home/nfs/pgali/.vscode-server/bin/abd2f3db4bdb28f9e95536dfa84d8479f1eb312d/bin/remote-cli:/nvme/0/pgali/.cargo/bin:/nvme/0/pgali/anaconda3/bin:/nvme/0/pgali/anaconda3/condabin:/nvme/0/pgali/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/local/cuda/bin:/usr/local/cuda/bin
     LD_LIBRARY_PATH                 : /usr/local/cuda/lib64:/usr/local/cuda/lib64::/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64:/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64
     NUMBAPRO_NVVM                   :
     NUMBAPRO_LIBDEVICE              :
     CONDA_PREFIX                    : /nvme/0/pgali/envs/cudfdev
     PYTHON_PATH                     :

     ***conda packages***
     /nvme/0/pgali/anaconda3/bin/conda
     # packages in environment at /nvme/0/pgali/envs/cudfdev:
     #
     # Name                    Version                   Build  Channel
     _libgcc_mutex             0.1                 conda_forge    conda-forge
     _openmp_mutex             4.5                       2_gnu    conda-forge
     _sysroot_linux-64_curr_repodata_hack 3                   h69a702a_13    conda-forge
     accessible-pygments       0.0.4              pyhd8ed1ab_0    conda-forge
     aiobotocore               2.5.4              pyhd8ed1ab_0    conda-forge
     aiohttp                   3.8.5           py310h2372a71_0    conda-forge
     aioitertools              0.11.0             pyhd8ed1ab_0    conda-forge
     aiosignal                 1.3.1              pyhd8ed1ab_0    conda-forge
     alabaster                 0.7.13             pyhd8ed1ab_0    conda-forge
     anyio                     4.0.0              pyhd8ed1ab_0    conda-forge
     argon2-cffi               23.1.0             pyhd8ed1ab_0    conda-forge
     argon2-cffi-bindings      21.2.0          py310h5764c6d_3    conda-forge
     arrow                     1.2.3              pyhd8ed1ab_0    conda-forge
     asttokens                 2.4.0              pyhd8ed1ab_0    conda-forge
     async-lru                 2.0.4              pyhd8ed1ab_0    conda-forge
     async-timeout             4.0.3              pyhd8ed1ab_0    conda-forge
     attrs                     23.1.0             pyh71513ae_1    conda-forge
     aws-c-auth                0.7.3                he2921ad_3    conda-forge
     aws-c-cal                 0.6.2                hc309b26_1    conda-forge
     aws-c-common              0.9.0                hd590300_0    conda-forge
     aws-c-compression         0.2.17               h4d4d85c_2    conda-forge
     aws-c-event-stream        0.3.2                h2e3709c_0    conda-forge
     aws-c-http                0.7.12               hc865f51_1    conda-forge
     aws-c-io                  0.13.32              h1a03231_3    conda-forge
     aws-c-mqtt                0.9.6                h3a0376c_0    conda-forge
     aws-c-s3                  0.3.17               h1678ad6_0    conda-forge
     aws-c-sdkutils            0.1.12               h4d4d85c_1    conda-forge
     aws-checksums             0.1.17               h4d4d85c_1    conda-forge
     aws-crt-cpp               0.23.1               hf7d0843_2    conda-forge
     aws-sam-translator        1.75.0             pyhd8ed1ab_0    conda-forge
     aws-sdk-cpp               1.11.156             he6c2984_2    conda-forge
     aws-xray-sdk              2.12.0             pyhd8ed1ab_0    conda-forge
     babel                     2.12.1             pyhd8ed1ab_1    conda-forge
     backcall                  0.2.0              pyh9f0ad1d_0    conda-forge
     backports                 1.0                pyhd8ed1ab_3    conda-forge
     backports.functools_lru_cache 1.6.5              pyhd8ed1ab_0    conda-forge
     backports.zoneinfo        0.2.1           py310hff52083_7    conda-forge
     bcrypt                    4.0.1           py310hcb5633a_1    conda-forge
     beautifulsoup4            4.12.2             pyha770c72_0    conda-forge
     benchmark                 1.8.0                h59595ed_0    conda-forge
     binutils                  2.40                 hdd6e379_0    conda-forge
     binutils_impl_linux-64    2.40                 hf600244_0    conda-forge
     binutils_linux-64         2.40                 hbdbef99_2    conda-forge
     blas                      1.0                         mkl    conda-forge
     bleach                    6.0.0              pyhd8ed1ab_0    conda-forge
     blinker                   1.6.2              pyhd8ed1ab_0    conda-forge
     bokeh                     3.2.2              pyhd8ed1ab_0    conda-forge
     boto3                     1.28.17            pyhd8ed1ab_0    conda-forge
     botocore                  1.31.17            pyhd8ed1ab_3    conda-forge
     brotlipy                  0.7.0           py310h5764c6d_1005    conda-forge
     bzip2                     1.0.8                h7f98852_4    conda-forge
     c-ares                    1.19.1               hd590300_0    conda-forge
     c-compiler                1.5.2                h0b41bf4_0    conda-forge
     ca-certificates           2023.7.22            hbcca054_0    conda-forge
     cached-property           1.5.2                hd8ed1ab_1    conda-forge
     cached_property           1.5.2              pyha770c72_1    conda-forge
     cachetools                5.3.1              pyhd8ed1ab_0    conda-forge
     certifi                   2023.7.22          pyhd8ed1ab_0    conda-forge
     cffi                      1.15.1          py310h2fee648_5    conda-forge
     cfgv                      3.3.1              pyhd8ed1ab_0    conda-forge
     cfn-lint                  0.80.2             pyhd8ed1ab_0    conda-forge
     charset-normalizer        3.2.0              pyhd8ed1ab_0    conda-forge
     click                     8.1.7           unix_pyh707e725_0    conda-forge
     cloudpickle               2.2.1              pyhd8ed1ab_0    conda-forge
     cmake                     3.27.6               hcfe8598_0    conda-forge
     colorama                  0.4.6              pyhd8ed1ab_0    conda-forge
     comm                      0.1.4              pyhd8ed1ab_0    conda-forge
     commonmark                0.9.1                      py_0    conda-forge
     contourpy                 1.1.1           py310hd41b1e2_0    conda-forge
     coverage                  7.3.1           py310h2372a71_1    conda-forge
     cryptography              41.0.4          py310h75e40e8_0    conda-forge
     cubinlinker               0.3.0           py310hfdf336d_0    rapidsai
     cuda-nvtx                 11.8.86                       0    nvidia
     cuda-python               11.8.2          py310h01a121a_0    conda-forge
     cuda-sanitizer-api        11.8.86                       0    nvidia
     cuda-version              11.8                 h70ddcb2_2    conda-forge
     cudatoolkit               11.8.0              h4ba93d1_12    conda-forge
     cudf                      23.10.0                  pypi_0    pypi
     cupy                      12.2.0          py310hbb1d8f0_1    conda-forge
     cxx-compiler              1.5.2                hf52228f_0    conda-forge
     cyrus-sasl                2.1.27               h54b06d7_7    conda-forge
     cython                    3.0.2           py310hc6cd4ac_2    conda-forge
     cytoolz                   0.12.2          py310h2372a71_0    conda-forge
     dask                      2023.9.2           pyhd8ed1ab_0    conda-forge
     dask-core                 2023.9.3a230920 py_gda256320e_1    dask/label/dev
     dask-cudf                 23.10.0                  pypi_0    pypi
     dataclasses               0.8                pyhc8e2a94_3    conda-forge
     datasets                  2.14.4             pyhd8ed1ab_0    conda-forge
     debugpy                   1.8.0           py310hc6cd4ac_0    conda-forge
     decopatch                 1.4.10             pyhd8ed1ab_0    conda-forge
     decorator                 5.1.1              pyhd8ed1ab_0    conda-forge
     defusedxml                0.7.1              pyhd8ed1ab_0    conda-forge
     dill                      0.3.7              pyhd8ed1ab_0    conda-forge
     distlib                   0.3.7              pyhd8ed1ab_0    conda-forge
     distributed               2023.9.2           pyhd8ed1ab_0    conda-forge
     distro                    1.8.0              pyhd8ed1ab_0    conda-forge
     dlpack                    0.5                  h9c3ff4c_0    conda-forge
     docker-py                 6.1.3              pyhd8ed1ab_0    conda-forge
     docutils                  0.19            py310hff52083_1    conda-forge
     doxygen                   1.9.1                hb166930_1    conda-forge
     ecdsa                     0.18.0             pyhd8ed1ab_1    conda-forge
     entrypoints               0.4                pyhd8ed1ab_0    conda-forge
     exceptiongroup            1.1.3              pyhd8ed1ab_0    conda-forge
     execnet                   2.0.2              pyhd8ed1ab_0    conda-forge
     executing                 1.2.0              pyhd8ed1ab_0    conda-forge
     fastavro                  1.8.3           py310h2372a71_0    conda-forge
     fastrlock                 0.8.2           py310hc6cd4ac_0    conda-forge
     filelock                  3.12.4             pyhd8ed1ab_0    conda-forge
     flask                     2.3.3              pyhd8ed1ab_0    conda-forge
     flask_cors                3.0.10             pyhd3deb0d_0    conda-forge
     fmt                       9.1.0                h924138e_0    conda-forge
     fqdn                      1.5.1              pyhd8ed1ab_0    conda-forge
     freetype                  2.12.1               h267a509_2    conda-forge
     frozenlist                1.4.0           py310h2372a71_0    conda-forge
     fsspec                    2023.9.1           pyh1a96a4e_0    conda-forge
     future                    0.18.3             pyhd8ed1ab_0    conda-forge
     gcc                       11.4.0               h7baecda_2    conda-forge
     gcc_impl_linux-64         11.4.0               h7aa1c59_2    conda-forge
     gcc_linux-64              11.4.0               hfd045f2_2    conda-forge
     gflags                    2.2.2             he1b5a44_1004    conda-forge
     glog                      0.6.0                h6f12383_0    conda-forge
     gmock                     1.14.0               ha770c72_1    conda-forge
     gmp                       6.2.1                h58526e2_0    conda-forge
     gmpy2                     2.1.2           py310h3ec546c_1    conda-forge
     graphql-core              3.2.3              pyhd8ed1ab_0    conda-forge
     greenlet                  2.0.2           py310hc6cd4ac_1    conda-forge
     gtest                     1.14.0               h00ab1b0_1    conda-forge
     gxx                       11.4.0               h7baecda_2    conda-forge
     gxx_impl_linux-64         11.4.0               h7aa1c59_2    conda-forge
     gxx_linux-64              11.4.0               hfc1ae95_2    conda-forge
     huggingface_hub           0.17.2             pyhd8ed1ab_0    conda-forge
     hypothesis                6.86.2             pyha770c72_0    conda-forge
     identify                  2.5.29             pyhd8ed1ab_0    conda-forge
     idna                      3.4                pyhd8ed1ab_0    conda-forge
     imagesize                 1.4.1              pyhd8ed1ab_0    conda-forge
     importlib-metadata        6.8.0              pyha770c72_0    conda-forge
     importlib_metadata        6.8.0                hd8ed1ab_0    conda-forge
     importlib_resources       5.13.0             pyhd8ed1ab_0    conda-forge
     iniconfig                 2.0.0              pyhd8ed1ab_0    conda-forge
     intel-openmp              2022.1.0          h9e868ea_3769
     ipykernel                 6.25.2             pyh2140261_0    conda-forge
     ipython                   8.15.0             pyh0d859eb_0    conda-forge
     isoduration               20.11.0            pyhd8ed1ab_0    conda-forge
     itsdangerous              2.1.2              pyhd8ed1ab_0    conda-forge
     jedi                      0.19.0             pyhd8ed1ab_0    conda-forge
     jinja2                    3.1.2              pyhd8ed1ab_1    conda-forge
     jmespath                  1.0.1              pyhd8ed1ab_0    conda-forge
     joblib                    1.3.2              pyhd8ed1ab_0    conda-forge
     jschema-to-python         1.2.3              pyhd8ed1ab_0    conda-forge
     json5                     0.9.14             pyhd8ed1ab_0    conda-forge
     jsondiff                  2.0.0              pyhd8ed1ab_0    conda-forge
     jsonpatch                 1.32               pyhd8ed1ab_0    conda-forge
     jsonpickle                2.2.0              pyhd8ed1ab_0    conda-forge
     jsonpointer               2.4             py310hff52083_2    conda-forge
     jsonschema                4.19.1             pyhd8ed1ab_0    conda-forge
     jsonschema-spec           0.2.4              pyhd8ed1ab_0    conda-forge
     jsonschema-specifications 2023.7.1           pyhd8ed1ab_0    conda-forge
     jsonschema-with-format-nongpl 4.19.1             pyhd8ed1ab_0    conda-forge
     junit-xml                 1.9                pyh9f0ad1d_0    conda-forge
     jupyter-cache             0.6.1              pyhd8ed1ab_0    conda-forge
     jupyter-lsp               2.2.0              pyhd8ed1ab_0    conda-forge
     jupyter_client            8.3.1              pyhd8ed1ab_0    conda-forge
     jupyter_core              5.3.1           py310hff52083_0    conda-forge
     jupyter_events            0.7.0              pyhd8ed1ab_2    conda-forge
     jupyter_server            2.7.3              pyhd8ed1ab_0    conda-forge
     jupyter_server_terminals  0.4.4              pyhd8ed1ab_1    conda-forge
     jupyterlab                4.0.6              pyhd8ed1ab_0    conda-forge
     jupyterlab_pygments       0.2.2              pyhd8ed1ab_0    conda-forge
     jupyterlab_server         2.25.0             pyhd8ed1ab_0    conda-forge
     kernel-headers_linux-64   3.10.0              h4a8ded7_13    conda-forge
     keyutils                  1.6.1                h166bdaf_0    conda-forge
     krb5                      1.21.2               h659d440_0    conda-forge
     lazy-object-proxy         1.9.0           py310h1fa729e_0    conda-forge
     lcms2                     2.15                 h7f713cb_2    conda-forge
     ld_impl_linux-64          2.40                 h41732ed_0    conda-forge
     lerc                      4.0.0                h27087fc_0    conda-forge
     libabseil                 20230802.1      cxx17_h59595ed_0    conda-forge
     libarrow                  13.0.0           h1935d02_4_cpu    conda-forge
     libblas                   3.9.0            16_linux64_mkl    conda-forge
     libbrotlicommon           1.1.0                hd590300_0    conda-forge
     libbrotlidec              1.1.0                hd590300_0    conda-forge
     libbrotlienc              1.1.0                hd590300_0    conda-forge
     libcblas                  3.9.0            16_linux64_mkl    conda-forge
     libcrc32c                 1.1.2                h9c3ff4c_0    conda-forge
     libcufile                 1.4.0.31                      0    nvidia
     libcufile-dev             1.4.0.31                      0    nvidia
     libcurand                 10.3.0.86                     0    nvidia
     libcurand-dev             10.3.0.86                     0    nvidia
     libcurl                   8.3.0                hca28451_0    conda-forge
     libdeflate                1.19                 hd590300_0    conda-forge
     libedit                   3.1.20191231         he28a2e2_2    conda-forge
     libev                     4.33                 h516909a_1    conda-forge
     libevent                  2.1.12               hf998b51_1    conda-forge
     libexpat                  2.5.0                hcb278e6_1    conda-forge
     libffi                    3.4.2                h7f98852_5    conda-forge
     libgcc-devel_linux-64     11.4.0               h922705a_2    conda-forge
     libgcc-ng                 13.2.0               h807b86a_2    conda-forge
     libgfortran-ng            13.2.0               h69a702a_2    conda-forge
     libgfortran5              13.2.0               ha4646dd_2    conda-forge
     libgomp                   13.2.0               h807b86a_2    conda-forge
     libgoogle-cloud           2.12.0               h8d7e28b_2    conda-forge
     libgrpc                   1.57.0               ha4d0f93_1    conda-forge
     libiconv                  1.17                 h166bdaf_0    conda-forge
     libjpeg-turbo             2.1.5.1              hd590300_1    conda-forge
     libkvikio                 23.10.00a       cuda11_230922_g8db93d0_23    rapidsai-nightly
     liblapack                 3.9.0            16_linux64_mkl    conda-forge
     libllvm14                 14.0.6               hcd5def8_4    conda-forge
     libnghttp2                1.52.0               h61bc06f_0    conda-forge
     libnsl                    2.0.0                h7f98852_0    conda-forge
     libntlm                   1.4               h7f98852_1002    conda-forge
     libnuma                   2.0.16               h0b41bf4_1    conda-forge
     libpng                    1.6.39               h753d276_0    conda-forge
     libprotobuf               4.23.4               hf27288f_6    conda-forge
     librdkafka                1.9.2                ha5a0de0_2    conda-forge
     librmm                    23.10.00a       cuda11_230922_gcd76d414_24    rapidsai-nightly
     libsanitizer              11.4.0               h4dcbe23_2    conda-forge
     libsodium                 1.0.18               h36c2ea0_1    conda-forge
     libsqlite                 3.43.0               h2797004_0    conda-forge
     libssh2                   1.11.0               h0841786_0    conda-forge
     libstdcxx-devel_linux-64  11.4.0               h922705a_2    conda-forge
     libstdcxx-ng              13.2.0               h7e041cc_2    conda-forge
     libthrift                 0.19.0               h8fd135c_0    conda-forge
     libtiff                   4.6.0                h29866fb_1    conda-forge
     libutf8proc               2.8.0                h166bdaf_0    conda-forge
     libuuid                   2.38.1               h0b41bf4_0    conda-forge
     libuv                     1.46.0               hd590300_0    conda-forge
     libwebp-base              1.3.2                hd590300_0    conda-forge
     libxcb                    1.15                 h0b41bf4_0    conda-forge
     libzlib                   1.2.13               hd590300_5    conda-forge
     livereload                2.6.3              pyh9f0ad1d_0    conda-forge
     llvmlite                  0.40.1          py310h1b8f574_0    conda-forge
     locket                    1.0.0              pyhd8ed1ab_0    conda-forge
     lz4                       4.3.2           py310h0cfdcf0_0    conda-forge
     lz4-c                     1.9.4                hcb278e6_0    conda-forge
     make                      4.3                  hd18ef5c_1    conda-forge
     makefun                   1.15.1             pyhd8ed1ab_0    conda-forge
     markdown                  3.4.4              pyhd8ed1ab_0    conda-forge
     markdown-it-py            2.2.0              pyhd8ed1ab_0    conda-forge
     markupsafe                2.1.3           py310h2372a71_1    conda-forge
     matplotlib-inline         0.1.6              pyhd8ed1ab_0    conda-forge
     mdit-py-plugins           0.4.0              pyhd8ed1ab_0    conda-forge
     mdurl                     0.1.0              pyhd8ed1ab_0    conda-forge
     mimesis                   11.1.0             pyhd8ed1ab_0    conda-forge
     mistune                   3.0.1              pyhd8ed1ab_0    conda-forge
     mkl                       2022.1.0           hc2b9512_224
     moto                      4.2.3              pyhd8ed1ab_0    conda-forge
     mpc                       1.3.1                hfe3b2da_0    conda-forge
     mpfr                      4.2.0                hb012696_0    conda-forge
     mpmath                    1.3.0              pyhd8ed1ab_0    conda-forge
     msgpack-python            1.0.5           py310hdf3cbec_0    conda-forge
     multidict                 6.0.4           py310h1fa729e_0    conda-forge
     multiprocess              0.70.15         py310h2372a71_0    conda-forge
     myst-nb                   0.17.2             pyhd8ed1ab_0    conda-forge
     myst-parser               0.18.1             pyhd8ed1ab_0    conda-forge
     nbclient                  0.7.4              pyhd8ed1ab_0    conda-forge
     nbconvert                 7.8.0              pyhd8ed1ab_0    conda-forge
     nbconvert-core            7.8.0              pyhd8ed1ab_0    conda-forge
     nbconvert-pandoc          7.8.0              pyhd8ed1ab_0    conda-forge
     nbformat                  5.9.2              pyhd8ed1ab_0    conda-forge
     nbsphinx                  0.9.3              pyhd8ed1ab_0    conda-forge
     ncurses                   6.4                  hcb278e6_0    conda-forge
     nest-asyncio              1.5.6              pyhd8ed1ab_0    conda-forge
     networkx                  3.1                pyhd8ed1ab_0    conda-forge
     ninja                     1.11.1               h924138e_0    conda-forge
     nodeenv                   1.8.0              pyhd8ed1ab_0    conda-forge
     notebook                  7.0.4              pyhd8ed1ab_0    conda-forge
     notebook-shim             0.2.3              pyhd8ed1ab_0    conda-forge
     numba                     0.57.1          py310h0f6aa51_0    conda-forge
     numpy                     1.24.4          py310ha4c1d20_0    conda-forge
     numpydoc                  1.5.0              pyhd8ed1ab_0    conda-forge
     nvcc_linux-64             11.8                h667003e_22    conda-forge
     nvcomp                    2.6.1                h0800d71_2    conda-forge
     nvtx                      0.2.8           py310h2372a71_0    conda-forge
     openapi-schema-validator  0.6.1              pyhd8ed1ab_0    conda-forge
     openapi-spec-validator    0.6.0              pyhd8ed1ab_0    conda-forge
     openjpeg                  2.5.0                h488ebb8_3    conda-forge
     openssl                   3.1.3                hd590300_0    conda-forge
     orc                       1.9.0                h52d3b3c_2    conda-forge
     overrides                 7.4.0              pyhd8ed1ab_0    conda-forge
     packaging                 23.1               pyhd8ed1ab_0    conda-forge
     pandas                    1.5.3           py310h9b08913_1    conda-forge
     pandoc                    3.1.3                h32600fe_0    conda-forge
     pandocfilters             1.5.0              pyhd8ed1ab_0    conda-forge
     paramiko                  3.3.1              pyhd8ed1ab_0    conda-forge
     parso                     0.8.3              pyhd8ed1ab_0    conda-forge
     partd                     1.4.0              pyhd8ed1ab_1    conda-forge
     pathable                  0.4.3              pyhd8ed1ab_0    conda-forge
     pbr                       5.11.1             pyhd8ed1ab_0    conda-forge
     pexpect                   4.8.0              pyh1a96a4e_2    conda-forge
     pickleshare               0.7.5                   py_1003    conda-forge
     pillow                    10.0.1          py310h29da1c1_1    conda-forge
     pip                       23.2.1             pyhd8ed1ab_0    conda-forge
     pkgutil-resolve-name      1.3.10             pyhd8ed1ab_1    conda-forge
     platformdirs              3.10.0             pyhd8ed1ab_0    conda-forge
     pluggy                    1.3.0              pyhd8ed1ab_0    conda-forge
     pre-commit                3.4.0              pyha770c72_1    conda-forge
     prometheus_client         0.17.1             pyhd8ed1ab_0    conda-forge
     prompt-toolkit            3.0.39             pyha770c72_0    conda-forge
     prompt_toolkit            3.0.39               hd8ed1ab_0    conda-forge
     protobuf                  4.23.4          py310h620c231_2    conda-forge
     psutil                    5.9.5           py310h2372a71_1    conda-forge
     pthread-stubs             0.4               h36c2ea0_1001    conda-forge
     ptxcompiler               0.8.1           py310h01a121a_0    conda-forge
     ptyprocess                0.7.0              pyhd3deb0d_0    conda-forge
     pure_eval                 0.2.2              pyhd8ed1ab_0    conda-forge
     py-cpuinfo                9.0.0              pyhd8ed1ab_0    conda-forge
     pyarrow                   13.0.0          py310hf9e7431_4_cpu    conda-forge
     pyasn1                    0.5.0              pyhd8ed1ab_0    conda-forge
     pycparser                 2.21               pyhd8ed1ab_0    conda-forge
     pydantic                  1.10.12         py310h2372a71_1    conda-forge
     pydata-sphinx-theme       0.14.1             pyhd8ed1ab_0    conda-forge
     pygments                  2.16.1             pyhd8ed1ab_0    conda-forge
     pynacl                    1.5.0           py310h5764c6d_2    conda-forge
     pyopenssl                 23.2.0             pyhd8ed1ab_1    conda-forge
     pyparsing                 3.1.1              pyhd8ed1ab_0    conda-forge
     pysocks                   1.7.1              pyha2e5f31_6    conda-forge
     pytest                    7.4.2              pyhd8ed1ab_0    conda-forge
     pytest-benchmark          4.0.0              pyhd8ed1ab_0    conda-forge
     pytest-cases              3.6.14             pyhd8ed1ab_0    conda-forge
     pytest-cov                4.1.0              pyhd8ed1ab_0    conda-forge
     pytest-xdist              3.3.1              pyhd8ed1ab_0    conda-forge
     python                    3.10.12         hd12c33a_0_cpython    conda-forge
     python-confluent-kafka    1.9.2           py310h5764c6d_2    conda-forge
     python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
     python-fastjsonschema     2.18.0             pyhd8ed1ab_0    conda-forge
     python-jose               3.3.0              pyh6c4a22f_1    conda-forge
     python-json-logger        2.0.7              pyhd8ed1ab_0    conda-forge
     python-snappy             0.6.1           py310hcee4d7c_0    conda-forge
     python-xxhash             3.3.0           py310h2372a71_0    conda-forge
     python_abi                3.10                    4_cp310    conda-forge
     pytorch                   1.11.0             py3.10_cpu_0    pytorch
     pytorch-mutex             1.0                         cpu    pytorch
     pytz                      2023.3.post1       pyhd8ed1ab_0    conda-forge
     pywin32-on-windows        0.1.0              pyh1179c8e_3    conda-forge
     pyyaml                    6.0.1           py310h2372a71_1    conda-forge
     pyzmq                     25.1.1          py310h5bbb5d0_0    conda-forge
     rdma-core                 28.9                 h59595ed_1    conda-forge
     re2                       2023.03.02           h8c504da_0    conda-forge
     readline                  8.2                  h8228510_1    conda-forge
     recommonmark              0.7.1              pyhd8ed1ab_0    conda-forge
     referencing               0.30.2             pyhd8ed1ab_0    conda-forge
     regex                     2023.8.8        py310h2372a71_0    conda-forge
     requests                  2.31.0             pyhd8ed1ab_0    conda-forge
     responses                 0.23.1             pyhd8ed1ab_0    conda-forge
     rfc3339-validator         0.1.4              pyhd8ed1ab_0    conda-forge
     rfc3986-validator         0.1.1              pyh9f0ad1d_0    conda-forge
     rhash                     1.4.4                hd590300_0    conda-forge
     rmm                       23.10.00a       cuda11_py310_230922_gcd76d414_24    rapidsai-nightly
     rpds-py                   0.10.3          py310hcb5633a_0    conda-forge
     rsa                       4.9                pyhd8ed1ab_0    conda-forge
     s2n                       1.3.51               h06160fa_0    conda-forge
     s3fs                      2023.9.1           pyhd8ed1ab_0    conda-forge
     s3transfer                0.6.2              pyhd8ed1ab_0    conda-forge
     sacremoses                0.0.53             pyhd8ed1ab_0    conda-forge
     sarif-om                  1.0.4              pyhd8ed1ab_0    conda-forge
     scikit-build              0.17.6             pyh4af843d_0    conda-forge
     scipy                     1.11.2          py310hb13e2d6_1    conda-forge
     sed                       4.8                  he412f7d_0    conda-forge
     send2trash                1.8.2              pyh41d4057_0    conda-forge
     setuptools                68.2.2             pyhd8ed1ab_0    conda-forge
     six                       1.16.0             pyh6c4a22f_0    conda-forge
     snappy                    1.1.10               h9fff704_0    conda-forge
     sniffio                   1.3.0              pyhd8ed1ab_0    conda-forge
     snowballstemmer           2.2.0              pyhd8ed1ab_0    conda-forge
     sortedcontainers          2.4.0              pyhd8ed1ab_0    conda-forge
     soupsieve                 2.5                pyhd8ed1ab_1    conda-forge
     spdlog                    1.11.0               h9b3ece8_1    conda-forge
     sphinx                    5.3.0              pyhd8ed1ab_0    conda-forge
     sphinx-autobuild          2021.3.14          pyhd8ed1ab_0    conda-forge
     sphinx-copybutton         0.5.2              pyhd8ed1ab_0    conda-forge
     sphinx-markdown-tables    0.0.17             pyh6c4a22f_0    conda-forge
     sphinxcontrib-applehelp   1.0.7              pyhd8ed1ab_0    conda-forge
     sphinxcontrib-devhelp     1.0.5              pyhd8ed1ab_0    conda-forge
     sphinxcontrib-htmlhelp    2.0.4              pyhd8ed1ab_0    conda-forge
     sphinxcontrib-jsmath      1.0.1              pyhd8ed1ab_0    conda-forge
     sphinxcontrib-qthelp      1.0.6              pyhd8ed1ab_0    conda-forge
     sphinxcontrib-serializinghtml 1.1.9              pyhd8ed1ab_0    conda-forge
     sphinxcontrib-websupport  1.2.6              pyhd8ed1ab_0    conda-forge
     sqlalchemy                2.0.21          py310h2372a71_0    conda-forge
     sshpubkeys                3.3.1              pyhd8ed1ab_0    conda-forge
     stack_data                0.6.2              pyhd8ed1ab_0    conda-forge
     streamz                   0.6.4              pyh6c4a22f_0    conda-forge
     sympy                     1.12            pypyh9d50eac_103    conda-forge
     sysroot_linux-64          2.17                h4a8ded7_13    conda-forge
     tabulate                  0.9.0              pyhd8ed1ab_1    conda-forge
     tblib                     2.0.0              pyhd8ed1ab_0    conda-forge
     terminado                 0.17.1             pyh41d4057_0    conda-forge
     tinycss2                  1.2.1              pyhd8ed1ab_0    conda-forge
     tk                        8.6.12               h27826a3_0    conda-forge
     tokenizers                0.13.1          py310h633acb5_2    conda-forge
     toml                      0.10.2             pyhd8ed1ab_0    conda-forge
     tomli                     2.0.1              pyhd8ed1ab_0    conda-forge
     toolz                     0.12.0             pyhd8ed1ab_0    conda-forge
     tornado                   6.3.3           py310h2372a71_1    conda-forge
     tqdm                      4.66.1             pyhd8ed1ab_0    conda-forge
     traitlets                 5.10.0             pyhd8ed1ab_0    conda-forge
     transformers              4.24.0             pyhd8ed1ab_0    conda-forge
     types-pyyaml              6.0.12.11          pyhd8ed1ab_0    conda-forge
     typing-extensions         4.8.0                hd8ed1ab_0    conda-forge
     typing_extensions         4.8.0              pyha770c72_0    conda-forge
     typing_utils              0.1.0              pyhd8ed1ab_0    conda-forge
     tzdata                    2023c                h71feb2d_0    conda-forge
     ucx                       1.14.1               h64cca9d_5    conda-forge
     ukkonen                   1.0.1           py310hbf28c38_3    conda-forge
     uri-template              1.3.0              pyhd8ed1ab_0    conda-forge
     urllib3                   1.26.15            pyhd8ed1ab_0    conda-forge
     virtualenv                20.24.4            pyhd8ed1ab_0    conda-forge
     wcwidth                   0.2.6              pyhd8ed1ab_0    conda-forge
     webcolors                 1.13               pyhd8ed1ab_0    conda-forge
     webencodings              0.5.1              pyhd8ed1ab_2    conda-forge
     websocket-client          1.6.3              pyhd8ed1ab_0    conda-forge
     werkzeug                  2.3.7              pyhd8ed1ab_0    conda-forge
     wheel                     0.41.2             pyhd8ed1ab_0    conda-forge
     wrapt                     1.15.0          py310h1fa729e_0    conda-forge
     xmltodict                 0.13.0             pyhd8ed1ab_0    conda-forge
     xorg-libxau               1.0.11               hd590300_0    conda-forge
     xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge
     xxhash                    0.8.2                hd590300_0    conda-forge
     xyzservices               2023.7.0           pyhd8ed1ab_0    conda-forge
     xz                        5.2.6                h166bdaf_0    conda-forge
     yaml                      0.2.5                h7f98852_2    conda-forge
     yarl                      1.9.2           py310h2372a71_0    conda-forge
     zeromq                    4.3.4                h9c3ff4c_1    conda-forge
     zict                      3.0.0              pyhd8ed1ab_0    conda-forge
     zipp                      3.17.0             pyhd8ed1ab_0    conda-forge
     zlib                      1.2.13               hd590300_5    conda-forge
     zstd                      1.5.5                hfc55251_0    conda-forge

Additional context Add any other context about the problem here.

galipremsagar commented 10 months ago

@bdice I found the PR that was responsible for resolving this issue in the arrow project: https://github.com/apache/arrow/pull/35997

Would you be able to take a look at this in libcudf?

bdice commented 9 months ago

I'm hoping to work on this in 24.02 but not sure if I'll have time in this release. It's on my list of tasks.

pmattione-nvidia commented 7 months ago

I'm working on a set of changes to try to address this.

pmattione-nvidia commented 3 months ago

Update: This work has been broken into many separate PRs, 4 of which have been merged so far:

Explicit conversion PR Benchmarking PR Powers-of-10 PR Utilities PR

The primary PR will be submitted very soon. I'm putting some finishing touches on it based off of conversations with @ttnghia on handling some of the edge cases.

pmattione-nvidia commented 2 months ago

There will always be discrepancies between cuDF and pyarrow, because cuDF has chosen to truncate its results, whereas pyarrow has chosen to round them.

pmattione-nvidia commented 2 months ago

The new decimal <--> floating conversion PR has been merged.

galipremsagar commented 2 months ago

Thanks @pmattione-nvidia ! Closing this issue, was able to verify that the issue is fixed.