search

rapidsai / cudf

cuDF - GPU DataFrame Library
https://docs.rapids.ai/api/cudf/stable/
Apache License 2.0
8.42k stars 900 forks source link

[BUG] Cast string to date returns wrong result for single digit month or day #5126

Closed andygrove closed 3 years ago

andygrove commented 4 years ago

Describe the bug

Calling ColumnVector.asTimestampDays("%Y-%m-%d") on a string column works correctly for values in YYYY-MM-DD format but not for YYYY-MM-D, YYYY-M-DD, or YYYY-M-D.

Steps/Code to reproduce bug

  @Test
  void testCastStringToTimestamp() {
    String str = "2020-5-5";
    String format = "%Y-%m-%d";
    try (ColumnVector dates = ColumnVector.fromStrings(str);
         ColumnVector cv = dates.asTimestampDays(format);
         HostColumnVector copyToHost = cv.copyToHost();) {

      assertEquals(18387, copyToHost.getInt(0));
    }
  }

Expected behavior The value returned should be 1837, but zero is returned. It works fine for 2020-05-05.

Environment overview (please complete the following information)

Environment details


     **git***
     commit e1ec4995aa1df35591c7cd60a0d7863e8202bd87 (HEAD -> branch-0.14, rapids/branch-0.14)
     Merge: b9577a3e6 416ef8ddd
     Author: Devavret Makkar <devavret@users.noreply.github.com>
     Date:   Tue May 5 02:22:18 2020 +0530

     Merge pull request #5022 from hummingtree/hotfix/transform_header

     Add timestamp header to transform
     **git submodules***
     b165e1fb11eeea64ccf95053e40f2424312599cc ../thirdparty/cub (v1.7.1)
     e3f867027c1d9603b5a677795900465b9fac9cb8 ../thirdparty/jitify (heads/cudf)
     cdcda484d0c7db114ea29c3b33429de5756ecfd8 ../thirdparty/libcudacxx (0.8.1-99-gcdcda48)
     a97a7380c76346c22bb67b93695bed19592afad2 ../thirdparty/libcudacxx/libcxx (heads/rapidsai-interop)

     ***OS Information***
     DISTRIB_ID=Ubuntu
     DISTRIB_RELEASE=18.04
     DISTRIB_CODENAME=bionic
     DISTRIB_DESCRIPTION="Ubuntu 18.04.3 LTS"
     NAME="Ubuntu"
     VERSION="18.04.3 LTS (Bionic Beaver)"
     ID=ubuntu
     ID_LIKE=debian
     PRETTY_NAME="Ubuntu 18.04.3 LTS"
     VERSION_ID="18.04"
     HOME_URL="https://www.ubuntu.com/"
     SUPPORT_URL="https://help.ubuntu.com/"
     BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
     PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
     VERSION_CODENAME=bionic
     UBUNTU_CODENAME=bionic
     Linux andygrove-dt 5.3.0-51-generic #44~18.04.2-Ubuntu SMP Thu Apr 23 14:27:18 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

     ***GPU Information***
     Thu May  7 10:34:12 2020
     +-----------------------------------------------------------------------------+
     | NVIDIA-SMI 440.64.00    Driver Version: 440.64.00    CUDA Version: 10.2     |
     |-------------------------------+----------------------+----------------------+
     | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
     | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
     |===============================+======================+======================|
     |   0  Quadro RTX 6000     On   | 00000000:17:00.0 Off |                  Off |
     | 33%   31C    P8     3W / 260W |      1MiB / 24220MiB |      0%      Default |
     +-------------------------------+----------------------+----------------------+
     |   1  GeForce GT 710      On   | 00000000:65:00.0 N/A |                  N/A |
     | 50%   49C    P0    N/A /  N/A |    632MiB /  2000MiB |     N/A      Default |
     +-------------------------------+----------------------+----------------------+

     +-----------------------------------------------------------------------------+
     | Processes:                                                       GPU Memory |
     |  GPU       PID   Type   Process name                             Usage      |
     |=============================================================================|
     |    1                    Not Supported                                       |
     +-----------------------------------------------------------------------------+

     ***CPU***
     Architecture:        x86_64
     CPU op-mode(s):      32-bit, 64-bit
     Byte Order:          Little Endian
     CPU(s):              12
     On-line CPU(s) list: 0-11
     Thread(s) per core:  2
     Core(s) per socket:  6
     Socket(s):           1
     NUMA node(s):        1
     Vendor ID:           GenuineIntel
     CPU family:          6
     Model:               85
     Model name:          Intel(R) Core(TM) i7-7800X CPU @ 3.50GHz
     Stepping:            4
     CPU MHz:             1200.052
     CPU max MHz:         4000.0000
     CPU min MHz:         1200.0000
     BogoMIPS:            6999.82
     Virtualization:      VT-x
     L1d cache:           32K
     L1i cache:           32K
     L2 cache:            1024K
     L3 cache:            8448K
     NUMA node0 CPU(s):   0-11
     Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req md_clear flush_l1d

     ***CMake***
     /home/andygrove/miniconda2/envs/cudf_dev/bin/cmake
     cmake version 3.17.0

     CMake suite maintained and supported by Kitware (kitware.com/cmake).

     ***g++***
     /usr/bin/g++
     g++ (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
     Copyright (C) 2017 Free Software Foundation, Inc.
     This is free software; see the source for copying conditions.  There is NO
     warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

     ***nvcc***

     ***Python***
     /home/andygrove/miniconda2/envs/cudf_dev/bin/python
     Python 3.7.6

     ***Environment Variables***
     PATH                            : /opt/apache-maven-3.6.3/bin:/home/andygrove/miniconda2/envs/cudf_dev/bin:/home/andygrove/miniconda2/condabin:/home/andygrove/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
     LD_LIBRARY_PATH                 :
     NUMBAPRO_NVVM                   :
     NUMBAPRO_LIBDEVICE              :
     CONDA_PREFIX                    : /home/andygrove/miniconda2/envs/cudf_dev
     PYTHON_PATH                     :

     ***conda packages***
     /home/andygrove/miniconda2/condabin/conda
     # packages in environment at /home/andygrove/miniconda2/envs/cudf_dev:
     #
     # Name                    Version                   Build  Channel
     _libgcc_mutex             0.1                 conda_forge    conda-forge
     _openmp_mutex             4.5                      1_llvm    conda-forge
     alabaster                 0.7.12                     py_0    conda-forge
     appdirs                   1.4.3                      py_1    conda-forge
     arrow-cpp                 0.15.0           py37h090bef1_2    conda-forge
     attrs                     19.3.0                     py_0    conda-forge
     babel                     2.8.0                      py_0    conda-forge
     backcall                  0.1.0                      py_0    conda-forge
     black                     19.10b0                  py37_0    conda-forge
     bleach                    3.1.4              pyh9f0ad1d_0    conda-forge
     bokeh                     2.0.1            py37hc8dfbb8_0    conda-forge
     boost-cpp                 1.70.0               h8e57a91_2    conda-forge
     brotli                    1.0.7             he1b5a44_1001    conda-forge
     brotlipy                  0.7.0           py37h8f50634_1000    conda-forge
     bzip2                     1.0.8                h516909a_2    conda-forge
     c-ares                    1.15.0            h516909a_1001    conda-forge
     ca-certificates           2020.4.5.1           hecc5488_0    conda-forge
     certifi                   2020.4.5.1       py37hc8dfbb8_0    conda-forge
     cffi                      1.14.0           py37hd463f26_0    conda-forge
     cfgv                      3.1.0                      py_0    conda-forge
     chardet                   3.0.4           py37hc8dfbb8_1006    conda-forge
     click                     7.1.1              pyh8c360ce_0    conda-forge
     cloudpickle               1.3.0                      py_0    conda-forge
     cmake                     3.17.0               h28c56e5_0    conda-forge
     cmake_setuptools          0.1.3                      py_0    rapidsai
     commonmark                0.9.1                      py_0    conda-forge
     cryptography              2.8              py37hb09aad4_2    conda-forge
     cudatoolkit               10.0.130                      0    nvidia
     cudf                      0.14.0a0+3395.gf1dad0783.dirty          pypi_0    pypi
     cudnn                     7.6.0                cuda10.0_0    nvidia
     cupy                      7.3.0            py37h658377b_0    conda-forge
     cython                    0.29.16          py37h3340039_0    conda-forge
     cytoolz                   0.10.1           py37h516909a_0    conda-forge
     dask                      2.14.0+14.g03e9bc99          pypi_0    pypi
     decorator                 4.4.2                      py_0    conda-forge
     defusedxml                0.6.0                      py_0    conda-forge
     distributed               2.14.0+29.g8534e84b          pypi_0    pypi
     dlpack                    0.2                  he1b5a44_1    conda-forge
     docutils                  0.16             py37hc8dfbb8_1    conda-forge
     double-conversion         3.1.5                he1b5a44_2    conda-forge
     editdistance              0.5.3            py37h3340039_0    conda-forge
     entrypoints               0.3             py37hc8dfbb8_1001    conda-forge
     expat                     2.2.9                he1b5a44_2    conda-forge
     fastavro                  0.23.2           py37h8f50634_0    conda-forge
     fastrlock                 0.4             py37h3340039_1001    conda-forge
     flake8                    3.7.9            py37hc8dfbb8_1    conda-forge
     flatbuffers               1.12.0               he1b5a44_0    conda-forge
     freetype                  2.10.1               he06d7ca_0    conda-forge
     fsspec                    0.7.2                      py_0    conda-forge
     future                    0.18.2           py37hc8dfbb8_1    conda-forge
     gflags                    2.2.2             he1b5a44_1002    conda-forge
     glog                      0.4.0                h49b9bf7_3    conda-forge
     gmp                       6.2.0                he1b5a44_2    conda-forge
     grpc-cpp                  1.23.0               h18db393_0    conda-forge
     heapdict                  1.0.1                      py_0    conda-forge
     hypothesis                5.10.0                     py_0    conda-forge
     icu                       64.2                 he1b5a44_1    conda-forge
     identify                  1.4.14             pyh9f0ad1d_0    conda-forge
     idna                      2.9                        py_1    conda-forge
     imagesize                 1.2.0                      py_0    conda-forge
     importlib-metadata        1.6.0            py37hc8dfbb8_0    conda-forge
     importlib_metadata        1.6.0                         0    conda-forge
     ipykernel                 5.2.0            py37h43977f1_1    conda-forge
     ipython                   7.13.0           py37hc8dfbb8_2    conda-forge
     ipython_genutils          0.2.0                      py_1    conda-forge
     isort                     4.3.21           py37hc8dfbb8_1    conda-forge
     jedi                      0.17.0           py37hc8dfbb8_0    conda-forge
     jinja2                    2.11.2             pyh9f0ad1d_0    conda-forge
     jpeg                      9c                h14c3975_1001    conda-forge
     jsonschema                3.2.0            py37hc8dfbb8_1    conda-forge
     jupyter_client            6.1.3                      py_0    conda-forge
     jupyter_core              4.6.3            py37hc8dfbb8_1    conda-forge
     krb5                      1.17.1               h2fd8d38_0    conda-forge
     ld_impl_linux-64          2.34                 h53a641e_0    conda-forge
     libblas                   3.8.0               16_openblas    conda-forge
     libcblas                  3.8.0               16_openblas    conda-forge
     libcurl                   7.69.1               hf7181ac_0    conda-forge
     libedit                   3.1.20170329      hf8c457e_1001    conda-forge
     libevent                  2.1.10               h72c5cf5_0    conda-forge
     libffi                    3.2.1             he1b5a44_1007    conda-forge
     libgcc-ng                 9.2.0                h24d8f2e_2    conda-forge
     libgfortran-ng            7.3.0                hdf63c60_5    conda-forge
     liblapack                 3.8.0               16_openblas    conda-forge
     libllvm8                  8.0.1                hc9558a2_0    conda-forge
     libopenblas               0.3.9                h5ec1e0e_0    conda-forge
     libpng                    1.6.37               hed695b0_1    conda-forge
     libprotobuf               3.8.0                h8b12597_0    conda-forge
     librmm                    0.14.0a200420      cuda10.0_258    rapidsai-nightly
     libsodium                 1.0.17               h516909a_0    conda-forge
     libssh2                   1.8.2                h22169c7_2    conda-forge
     libstdcxx-ng              9.2.0                hdf63c60_2    conda-forge
     libtiff                   4.1.0                hfc65ed5_0    conda-forge
     libuv                     1.34.0               h516909a_0    conda-forge
     llvm-openmp               10.0.0               hc9558a2_0    conda-forge
     llvmlite                  0.31.0           py37h5202443_1    conda-forge
     locket                    0.2.0                      py_2    conda-forge
     lz4-c                     1.8.3             he1b5a44_1001    conda-forge
     markdown                  3.0.1                    pypi_0    pypi
     markupsafe                1.1.1            py37h8f50634_1    conda-forge
     mccabe                    0.6.1                      py_1    conda-forge
     mistune                   0.8.4           py37h8f50634_1001    conda-forge
     more-itertools            8.2.0                      py_0    conda-forge
     msgpack-python            1.0.0            py37h99015e2_1    conda-forge
     mypy_extensions           0.4.3            py37hc8dfbb8_1    conda-forge
     nbconvert                 5.6.1            py37hc8dfbb8_1    conda-forge
     nbformat                  5.0.6                      py_0    conda-forge
     nbsphinx                  0.6.1              pyh9f0ad1d_0    conda-forge
     nccl                      2.4.6.1              cuda10.0_0    nvidia
     ncurses                   6.1               hf484d3e_1002    conda-forge
     nodeenv                   1.3.5                      py_0    conda-forge
     notebook                  6.0.3                    py37_0    conda-forge
     numba                     0.48.0           py37hb3f55d8_0    conda-forge
     numpy                     1.18.1           py37h8960a57_1    conda-forge
     numpydoc                  0.9.2                      py_0    conda-forge
     nvstrings-cudaunknown     0.0.0.dev0               pypi_0    pypi
     olefile                   0.46                       py_0    conda-forge
     openssl                   1.1.1f               h516909a_0    conda-forge
     packaging                 20.1                       py_0    conda-forge
     pandas                    0.25.3           py37hb3f55d8_0    conda-forge
     pandoc                    1.19.2                        0    conda-forge
     pandocfilters             1.4.2                      py_1    conda-forge
     parquet-cpp               1.5.1                         2    conda-forge
     parso                     0.7.0              pyh9f0ad1d_0    conda-forge
     partd                     1.1.0                      py_0    conda-forge
     pathspec                  0.8.0              pyh9f0ad1d_0    conda-forge
     pexpect                   4.8.0            py37hc8dfbb8_1    conda-forge
     pickleshare               0.7.5           py37hc8dfbb8_1001    conda-forge
     pillow                    7.1.1            py37h718be6c_0    conda-forge
     pip                       20.0.2                     py_2    conda-forge
     pluggy                    0.13.0                   py37_0    conda-forge
     pre-commit                2.2.0            py37hc8dfbb8_1    conda-forge
     pre_commit                2.2.0                         1    conda-forge
     prometheus_client         0.7.1                      py_0    conda-forge
     prompt-toolkit            3.0.5                      py_0    conda-forge
     psutil                    5.7.0            py37h8f50634_1    conda-forge
     ptyprocess                0.6.0                   py_1001    conda-forge
     py                        1.8.1                      py_0    conda-forge
     pyarrow                   0.15.0           py37h8b68381_1    conda-forge
     pycodestyle               2.5.0                      py_0    conda-forge
     pycparser                 2.20                       py_0    conda-forge
     pyflakes                  2.1.1                      py_0    conda-forge
     pygments                  2.6.1                      py_0    conda-forge
     pyopenssl                 19.1.0                     py_1    conda-forge
     pyparsing                 2.4.7              pyh9f0ad1d_0    conda-forge
     pyrsistent                0.16.0           py37h8f50634_0    conda-forge
     pysocks                   1.7.1            py37hc8dfbb8_1    conda-forge
     pytest                    5.4.1            py37hc8dfbb8_0    conda-forge
     python                    3.7.6           h8356626_5_cpython    conda-forge
     python-dateutil           2.8.1                      py_0    conda-forge
     python_abi                3.7                     1_cp37m    conda-forge
     pytz                      2019.3                     py_0    conda-forge
     pyyaml                    5.3.1            py37h8f50634_0    conda-forge
     pyzmq                     19.0.0           py37hac76be4_1    conda-forge
     rapidjson                 1.1.0             he1b5a44_1002    conda-forge
     re2                       2020.04.01           he1b5a44_0    conda-forge
     readline                  8.0                  hf8c457e_0    conda-forge
     recommonmark              0.6.0                      py_0    conda-forge
     regex                     2020.4.4         py37h8f50634_0    conda-forge
     requests                  2.23.0             pyh8c360ce_2    conda-forge
     rhash                     1.3.6             h14c3975_1001    conda-forge
     rmm                       0.14.0a200420          py37_258    rapidsai-nightly
     send2trash                1.5.0                      py_0    conda-forge
     setuptools                46.1.3           py37hc8dfbb8_0    conda-forge
     six                       1.14.0                     py_1    conda-forge
     snappy                    1.1.8                he1b5a44_1    conda-forge
     snowballstemmer           2.0.0                      py_0    conda-forge
     sortedcontainers          2.1.0                      py_0    conda-forge
     sphinx                    3.0.2                      py_0    conda-forge
     sphinx-markdown-tables    0.0.12                   pypi_0    pypi
     sphinx_rtd_theme          0.4.3                      py_0    conda-forge
     sphinxcontrib-applehelp   1.0.2                      py_0    conda-forge
     sphinxcontrib-devhelp     1.0.2                      py_0    conda-forge
     sphinxcontrib-htmlhelp    1.0.3                      py_0    conda-forge
     sphinxcontrib-jsmath      1.0.1                      py_0    conda-forge
     sphinxcontrib-qthelp      1.0.3                      py_0    conda-forge
     sphinxcontrib-serializinghtml 1.1.4                      py_0    conda-forge
     sphinxcontrib-websupport  1.2.1              pyh9f0ad1d_0    conda-forge
     sqlite                    3.30.1               hcee41ef_0    conda-forge
     streamz                   0.5.3                    pypi_0    pypi
     tblib                     1.6.0                      py_0    conda-forge
     terminado                 0.8.3            py37hc8dfbb8_1    conda-forge
     testpath                  0.4.4                      py_0    conda-forge
     thrift-cpp                0.12.0            hf3afdfd_1004    conda-forge
     tk                        8.6.10               hed695b0_0    conda-forge
     toml                      0.10.0                     py_0    conda-forge
     toolz                     0.10.0                     py_0    conda-forge
     tornado                   6.0.4            py37h8f50634_1    conda-forge
     traitlets                 4.3.3            py37hc8dfbb8_1    conda-forge
     typed-ast                 1.4.1            py37h516909a_0    conda-forge
     typing_extensions         3.7.4.1          py37hc8dfbb8_3    conda-forge
     uriparser                 0.9.3                he1b5a44_1    conda-forge
     urllib3                   1.25.9                     py_0    conda-forge
     virtualenv                16.7.5                     py_0    conda-forge
     wcwidth                   0.1.9              pyh9f0ad1d_0    conda-forge
     webencodings              0.5.1                      py_1    conda-forge
     wheel                     0.34.2                     py_1    conda-forge
     xz                        5.2.5                h516909a_0    conda-forge
     yaml                      0.2.4                h516909a_0    conda-forge
     zeromq                    4.3.2                he1b5a44_2    conda-forge
     zict                      2.0.0                      py_0    conda-forge
     zipp                      3.1.0                      py_0    conda-forge
     zlib                      1.2.11            h516909a_1006    conda-forge
     zstd                      1.4.3                h3b9ef0a_0    conda-forge

Additional context None

davidwendt commented 4 years ago

This is working as intended as defined by the documentation for the API here: https://docs.rapids.ai/api/libcudf/nightly/group__strings__convert.html#ga9c8dd7fe42bcd7e0f615e730dfe0593a Since the format may contain any or no character delimiters there is no practical way to support dynamic length fields. Would 2020111 be January 11 or November 1 for valid format %Y%m%d ?

github-actions[bot] commented 3 years ago

This issue has been labeled inactive-90d due to no recent activity in the past 90 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed.

github-actions[bot] commented 3 years ago

This issue has been labeled inactive-30d due to no recent activity in the past 30 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. This issue will be labeled inactive-90d if there is no activity in the next 60 days.

beckernick commented 3 years ago

Are we okay to close this issue @andygrove @davidwendt ? Was this resolved implicitly by https://github.com/rapidsai/cudf/pull/6460 ?

andygrove commented 3 years ago

@beckernick yes this can be closed now. This was resolved with a combination of is_timestamp from #6460 and some additional processing in the Spark plugin.

beckernick commented 3 years ago

Thanks! Closing