rapidsai / cudf

cuDF - GPU DataFrame Library
https://docs.rapids.ai/api/cudf/stable/
Apache License 2.0
8.34k stars 888 forks source link

[BUG] Incorrect parquet file written for sliced string data #5858

Closed rjzamora closed 4 years ago

rjzamora commented 4 years ago

Describe the bug The data written to disk during to_parquet is incorrect if the method is called on a sliced DataFrame (e.g df.iloc[10:]) containg string data. For now, the only workaround is to explicity copy any data that needs to be written to disk.

Steps/Code to reproduce bug

import string
import cudf
from cudf.tests.utils import assert_eq

def _check(full_copy=False, use_str=False):
    path = "./test.parquet"
    df = cudf.DataFrame(
        {
            "a": list(string.ascii_lowercase) if use_str else range(26),
            "index": range(26),
        }
    )
    df = df.set_index("index")
    split = len(df)//2

    if full_copy:
        df.iloc[split:].copy().to_parquet(path)
    else:
        df.iloc[split:].to_parquet(path)

    result = cudf.read_parquet(path)
    assert_eq(df.iloc[split:].copy(), result)

_check(full_copy=True, use_str=False)  # PASSES
_check(full_copy=False, use_str=False)  # PASSES
_check(full_copy=True, use_str=True)  # PASSES
_check(full_copy=False, use_str=True)  # FAILS

Trace:

---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
<ipython-input-8-31a6bea48a7b> in <module>
----> 1 _check(full_copy=False, use_str=True)

<ipython-input-4-f015e4c354f8> in _check(full_copy, use_str)
     24     result = cudf.read_parquet(path)
     25     #import pdb; pdb.set_trace()
---> 26     assert_eq(df.iloc[split:], result)

    [... skipping hidden 3 frame]

pandas/_libs/testing.pyx in pandas._libs.testing.assert_almost_equal()

pandas/_libs/testing.pyx in pandas._libs.testing.assert_almost_equal()

    [... skipping hidden 1 frame]

AssertionError: DataFrame.iloc[:, 0] (column name="a") are different

DataFrame.iloc[:, 0] (column name="a") values are different (100.0 %)
[left]:  [n, o, p, q, r, s, t, u, v, w, x, y, z]
[right]: [a, b, c, d, e, f, g, h, i, j, k, l, m]

Expected behavior Ideally, the data view should be written to disk as expected.

Environment overview (please complete the following information)

Environment details

Click here to see environment details

     **git***
     commit fc1bbdbf042c4fc84e2d562a9927fed3404aeee5 (HEAD -> fix-multi-file-io, origin/fix-multi-file-io)
     Author: rjzamora 
     Date:   Wed Aug 5 10:29:08 2020 -0700

     remove pdb statement
     **git submodules***

     ***OS Information***
     DGX_NAME="DGX Server"
     DGX_PRETTY_NAME="NVIDIA DGX Server"
     DGX_SWBUILD_DATE="2020-03-04"
     DGX_SWBUILD_VERSION="4.4.0"
     DGX_COMMIT_ID="ee09ebc"
     DGX_PLATFORM="DGX Server for DGX-1"
     DGX_SERIAL_NUMBER="QTFCOU8220024"
     DISTRIB_ID=Ubuntu
     DISTRIB_RELEASE=18.04
     DISTRIB_CODENAME=bionic
     DISTRIB_DESCRIPTION="Ubuntu 18.04.4 LTS"
     NAME="Ubuntu"
     VERSION="18.04.4 LTS (Bionic Beaver)"
     ID=ubuntu
     ID_LIKE=debian
     PRETTY_NAME="Ubuntu 18.04.4 LTS"
     VERSION_ID="18.04"
     HOME_URL="https://www.ubuntu.com/"
     SUPPORT_URL="https://help.ubuntu.com/"
     BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
     PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
     VERSION_CODENAME=bionic
     UBUNTU_CODENAME=bionic
     Linux dgx14 4.15.0-76-generic #86-Ubuntu SMP Fri Jan 17 17:24:28 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

     ***GPU Information***
     Wed Aug  5 12:54:12 2020
     +-----------------------------------------------------------------------------+
     | NVIDIA-SMI 440.64.00    Driver Version: 440.64.00    CUDA Version: 10.2     |
     |-------------------------------+----------------------+----------------------+
     | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
     | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
     |===============================+======================+======================|
     |   0  Tesla V100-SXM2...  On   | 00000000:06:00.0 Off |                    0 |
     | N/A   31C    P0    56W / 300W |    691MiB / 32510MiB |      0%      Default |
     +-------------------------------+----------------------+----------------------+
     |   1  Tesla V100-SXM2...  On   | 00000000:07:00.0 Off |                    0 |
     | N/A   32C    P0    41W / 300W |     12MiB / 32510MiB |      0%      Default |
     +-------------------------------+----------------------+----------------------+
     |   2  Tesla V100-SXM2...  On   | 00000000:0A:00.0 Off |                    0 |
     | N/A   30C    P0    43W / 300W |     12MiB / 32510MiB |      0%      Default |
     +-------------------------------+----------------------+----------------------+
     |   3  Tesla V100-SXM2...  On   | 00000000:0B:00.0 Off |                    0 |
     | N/A   29C    P0    41W / 300W |     12MiB / 32510MiB |      0%      Default |
     +-------------------------------+----------------------+----------------------+
     |   4  Tesla V100-SXM2...  On   | 00000000:85:00.0 Off |                    0 |
     | N/A   30C    P0    43W / 300W |     12MiB / 32510MiB |      0%      Default |
     +-------------------------------+----------------------+----------------------+
     |   5  Tesla V100-SXM2...  On   | 00000000:86:00.0 Off |                    0 |
     | N/A   30C    P0    41W / 300W |     12MiB / 32510MiB |      0%      Default |
     +-------------------------------+----------------------+----------------------+
     |   6  Tesla V100-SXM2...  On   | 00000000:89:00.0 Off |                    0 |
     | N/A   32C    P0    43W / 300W |     12MiB / 32510MiB |      0%      Default |
     +-------------------------------+----------------------+----------------------+
     |   7  Tesla V100-SXM2...  On   | 00000000:8A:00.0 Off |                    0 |
     | N/A   28C    P0    42W / 300W |     12MiB / 32510MiB |      0%      Default |
     +-------------------------------+----------------------+----------------------+

     +-----------------------------------------------------------------------------+
     | Processes:                                                       GPU Memory |
     |  GPU       PID   Type   Process name                             Usage      |
     |=============================================================================|
     |    0     70083      C   ...ora/miniconda3/envs/cudf_dev/bin/python   679MiB |
     +-----------------------------------------------------------------------------+

     ***CPU***
     Architecture:        x86_64
     CPU op-mode(s):      32-bit, 64-bit
     Byte Order:          Little Endian
     CPU(s):              80
     On-line CPU(s) list: 0-79
     Thread(s) per core:  2
     Core(s) per socket:  20
     Socket(s):           2
     NUMA node(s):        2
     Vendor ID:           GenuineIntel
     CPU family:          6
     Model:               79
     Model name:          Intel(R) Xeon(R) CPU E5-2698 v4 @ 2.20GHz
     Stepping:            1
     CPU MHz:             3064.612
     CPU max MHz:         3600.0000
     CPU min MHz:         1200.0000
     BogoMIPS:            4390.04
     Virtualization:      VT-x
     L1d cache:           32K
     L1i cache:           32K
     L2 cache:            256K
     L3 cache:            51200K
     NUMA node0 CPU(s):   0-19,40-59
     NUMA node1 CPU(s):   20-39,60-79
     Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdt_a rdseed adx smap intel_pt xsaveopt cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts md_clear flush_l1d

     ***CMake***
     /datasets/rzamora/miniconda3/envs/cudf_dev/bin/cmake
     cmake version 3.17.0

     CMake suite maintained and supported by Kitware (kitware.com/cmake).

     ***g++***
     /usr/bin/g++
     g++ (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
     Copyright (C) 2017 Free Software Foundation, Inc.
     This is free software; see the source for copying conditions.  There is NO
     warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

     ***nvcc***
     /usr/local/cuda/bin/nvcc
     nvcc: NVIDIA (R) Cuda compiler driver
     Copyright (c) 2005-2019 NVIDIA Corporation
     Built on Wed_Oct_23_19:24:38_PDT_2019
     Cuda compilation tools, release 10.2, V10.2.89

     ***Python***
     /datasets/rzamora/miniconda3/envs/cudf_dev/bin/python
     Python 3.7.6

     ***Environment Variables***
     PATH                            : /home/nfs/rzamora/.vscode-server/bin/cd9ea6488829f560dc949a8b2fb789f3cdc05f5d/bin:/home/nfs/rzamora/bin:/home/nfs/rzamora/.local/bin:/datasets/rzamora/miniconda3/envs/cudf_dev/bin:/datasets/rzamora/miniconda3/condabin:/usr/local/cuda/bin:/opt/bin:/home/nfs/rzamora/.vscode-server/bin/cd9ea6488829f560dc949a8b2fb789f3cdc05f5d/bin:/home/nfs/rzamora/bin:/home/nfs/rzamora/.local/bin:/usr/local/cuda/bin:/opt/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
     LD_LIBRARY_PATH                 :
     NUMBAPRO_NVVM                   :
     NUMBAPRO_LIBDEVICE              :
     CONDA_PREFIX                    : /datasets/rzamora/miniconda3/envs/cudf_dev
     PYTHON_PATH                     :

     ***conda packages***
     /datasets/rzamora/miniconda3/condabin/conda
     # packages in environment at /datasets/rzamora/miniconda3/envs/cudf_dev:
     #
     # Name                    Version                   Build  Channel
     _libgcc_mutex             0.1                        main
     abseil-cpp                20200225.2           he1b5a44_0    conda-forge
     absl-py                   0.9.0                    pypi_0    pypi
     aiohttp                   3.6.2            py37h7b6447c_0
     alabaster                 0.7.12                   py37_0
     appdirs                   1.4.3            py37h28b3542_0
     arrow-cpp                 0.17.1          py37h1234567_10_cuda    conda-forge
     arrow-cpp-proc            1.0.0                      cuda    conda-forge
     astor                     0.8.1                    pypi_0    pypi
     async-timeout             3.0.1                    py37_0
     attrs                     19.3.0                     py_0
     aws-sdk-cpp               1.7.164              hc831370_1    conda-forge
     babel                     2.8.0                      py_0
     backcall                  0.2.0                      py_0
     beautifulsoup4            4.9.1                    pypi_0    pypi
     black                     19.10b0                  py37_0    conda-forge
     blas                      1.0                         mkl
     bleach                    3.1.5                      py_0
     blis                      0.4.1                    pypi_0    pypi
     bokeh                     2.1.1                    py37_0
     boost-cpp                 1.72.0               h7b93d67_2    conda-forge
     bottleneck                1.3.2                    pypi_0    pypi
     brotli                    1.0.7                he6710b0_0
     brotlipy                  0.7.0           py37h7b6447c_1000
     bzip2                     1.0.8                h7b6447c_0
     c-ares                    1.15.0            h7b6447c_1001
     ca-certificates           2020.6.24                     0    anaconda
     cachetools                4.1.1                    pypi_0    pypi
     cairo                     1.16.0            h3fc0475_1005    conda-forge
     catalogue                 1.0.0                    pypi_0    pypi
     certifi                   2020.6.20                py37_0    anaconda
     cffi                      1.14.0           py37h2e261b9_0
     cfgv                      3.1.0                      py_0    conda-forge
     chardet                   3.0.4                 py37_1003
     clang                     8.0.1                hc9558a2_2    conda-forge
     clang-tools               8.0.1                hc9558a2_2    conda-forge
     clangxx                   8.0.1                         2    conda-forge
     click                     7.1.2                      py_0
     cloudpickle               1.5.0                      py_0
     cmake                     3.17.0               h28c56e5_0    conda-forge
     cmake_setuptools          0.1.3                      py_0    rapidsai
     commonmark                0.9.1                      py_0
     cryptography              2.9.2            py37h1ba5d50_0
     cudatoolkit               10.2.89              hfd86e86_1
     cudf                      0.15.0a0+3829.ga128c1b32           dev_0    
     cudnn                     7.6.5                cuda10.2_0
     cupy                      7.6.0            py37h940342b_0    conda-forge
     curl                      7.69.1               hbc83047_0
     cycler                    0.10.0                   pypi_0    pypi
     cymem                     2.0.3                    pypi_0    pypi
     cython                    0.29.21          py37h3340039_0    conda-forge
     cytoolz                   0.10.1           py37h7b6447c_0
     dask                      2.21.0+63.gfdc4d252           dev_0    
     dask-cuda                 0.14.0b0+95.gb24567d           dev_0    
     dask-cudf                 0.15.0a0+3829.ga128c1b32           dev_0    
     dask-labextension         2.0.2                      py_0    conda-forge
     decorator                 4.4.2                      py_0
     defusedxml                0.6.0                      py_0
     distlib                   0.3.1              pyh9f0ad1d_0    conda-forge
     distributed               2.20.0+8.ga8a75865           dev_0    
     dlpack                    0.3                  he1b5a44_1    conda-forge
     docutils                  0.16                     py37_1
     double-conversion         3.1.5                he1b5a44_2    conda-forge
     editdistance              0.5.3            py37h3340039_0    conda-forge
     entrypoints               0.3                      py37_0
     expat                     2.2.9                he6710b0_2
     fastai                    1.0.61                   pypi_0    pypi
     fastavro                  0.23.6           py37h8f50634_0    conda-forge
     fastprogress              0.2.3                    pypi_0    pypi
     fastrlock                 0.4              py37he6710b0_0
     filelock                  3.0.12                     py_0
     flake8                    3.8.3                      py_1    conda-forge
     flatbuffers               1.12.0               he1b5a44_0    conda-forge
     fontconfig                2.13.1            h1056068_1002    conda-forge
     freetype                  2.10.2               h5ab3b9f_0
     fribidi                   1.0.9                h7b6447c_0
     fsspec                    0.7.4                      py_0    conda-forge
     future                    0.18.2                   py37_1
     gast                      0.2.2                    pypi_0    pypi
     gettext                   0.19.8.1             hd7bead4_3
     gflags                    2.2.2                he6710b0_0
     glib                      2.63.1               h5a9c865_0
     glog                      0.4.0                he6710b0_0
     gmp                       6.1.2                h6c8ec71_1
     google-auth               1.19.2                   pypi_0    pypi
     google-auth-oauthlib      0.4.1                    pypi_0    pypi
     google-pasta              0.2.0                    pypi_0    pypi
     graphite2                 1.3.14               h23475e2_0
     graphviz                  2.42.3               h0511662_0    conda-forge
     grpc-cpp                  1.30.1               h9ea6770_0    conda-forge
     grpcio                    1.24.3                   pypi_0    pypi
     h5py                      2.10.0                   pypi_0    pypi
     harfbuzz                  2.4.0                hee91db6_5    conda-forge
     heapdict                  1.0.1                      py_0
     hypothesis                5.19.2                     py_0    conda-forge
     icu                       67.1                 he1b5a44_0    conda-forge
     identify                  1.4.23             pyh9f0ad1d_0    conda-forge
     idna                      2.10                       py_0
     imagesize                 1.2.0                      py_0
     importlib-metadata        1.7.0                    py37_0
     importlib_metadata        1.7.0                         0
     iniconfig                 1.0.0              pyh9f0ad1d_0    conda-forge
     intel-openmp              2020.1                      217
     ipykernel                 5.3.2            py37h5ca1d4c_0
     ipython                   7.16.1           py37h43977f1_0    conda-forge
     ipython_genutils          0.2.0                    py37_0
     isort                     5.0.9            py37hc8dfbb8_0    conda-forge
     jedi                      0.17.1                   py37_0
     jinja2                    2.11.2                     py_0
     jpeg                      9d                   h516909a_0    conda-forge
     json5                     0.9.5                      py_0
     jsonschema                3.2.0                    py37_1
     jupyter-server-proxy      1.5.0                      py_0    conda-forge
     jupyter_client            6.1.6                      py_0
     jupyter_core              4.6.3                    py37_0
     jupyterlab                2.1.5                      py_0
     jupyterlab-nvdashboard    0.3.0                     dev_0    
     jupyterlab_server         1.2.0                      py_0
     keras-applications        1.0.8                    pypi_0    pypi
     keras-preprocessing       1.1.2                    pypi_0    pypi
     kiwisolver                1.2.0                    pypi_0    pypi
     krb5                      1.17.1               h173b8e3_0
     lcms2                     2.11                 h396b838_0
     ld_impl_linux-64          2.33.1               h53a641e_7
     libcurl                   7.69.1               h20c2e04_0
     libedit                   3.1.20181209         hc058e9b_0
     libevent                  2.1.10               hcdb4288_1    conda-forge
     libffi                    3.2.1                hd88cf55_4
     libgcc-ng                 9.1.0                hdf63c60_0
     libgfortran-ng            7.3.0                hdf63c60_0
     libiconv                  1.15                 h63c8f33_5
     libllvm8                  8.0.1                hc9558a2_0    conda-forge
     libllvm9                  9.0.1                h4a3c616_1
     libopenblas               0.3.10               h5a2b251_0
     libpng                    1.6.37               hbc83047_0
     libprotobuf               3.12.3               hd408876_0
     librmm                    0.15.0a200713   cuda10.2_gf9e9c55_323    rapidsai-nightly
     libsodium                 1.0.18               h7b6447c_0
     libssh2                   1.9.0                h1ba5d50_1
     libstdcxx-ng              9.1.0                hdf63c60_0
     libtiff                   4.1.0                h2733197_1
     libtool                   2.4.6                h7b6447c_5
     libuuid                   2.32.1            h14c3975_1000    conda-forge
     libuv                     1.38.0               h516909a_0    conda-forge
     libxcb                    1.14                 h7b6447c_0
     libxml2                   2.9.10               h72b56ed_1    conda-forge
     llvmlite                  0.33.0           py37hc6ec683_1
     locket                    0.2.0                    py37_1
     lz4-c                     1.9.2                he6710b0_0
     markdown                  3.1.1                    py37_0
     markupsafe                1.1.1            py37h14c3975_1
     matplotlib                3.3.0                    pypi_0    pypi
     mccabe                    0.6.1                    py37_1
     mistune                   0.8.4           py37h14c3975_1001
     mkl                       2020.1                      217
     mkl-service               2.3.0            py37he904b0f_0
     mkl_fft                   1.1.0            py37h23d657b_0
     mkl_random                1.1.1            py37h0573a6f_0
     more-itertools            8.4.0                      py_0
     msgpack-python            1.0.0            py37hfd86e86_1
     multidict                 4.7.6                    pypi_0    pypi
     murmurhash                1.0.2                    pypi_0    pypi
     mypy_extensions           0.4.3                    py37_0
     nbconvert                 5.6.1                    py37_1
     nbformat                  5.0.7                      py_0
     nbsphinx                  0.7.1              pyh9f0ad1d_0    conda-forge
     nccl                      2.7.6.1              hc6a2c23_0    conda-forge
     ncurses                   6.1                  he6710b0_1
     ninja                     1.9.0            py37hfd86e86_0
     nodeenv                   1.4.0              pyh9f0ad1d_0    conda-forge
     nodejs                    14.5.0               h568c755_0    conda-forge
     notebook                  6.0.3            py37hc8dfbb8_1    conda-forge
     numba                     0.50.1           py37h0da4684_0    conda-forge
     numexpr                   2.7.1                    pypi_0    pypi
     numpy                     1.18.5           py37ha1c710e_0
     numpy-base                1.18.5           py37hde5b4d6_0
     numpydoc                  1.1.0              pyh9f0ad1d_0    conda-forge
     nvidia-ml-py3             7.352.0                  pypi_0    pypi
     nvtabular                 0.1.1                     dev_0    
     oauthlib                  3.1.0                    pypi_0    pypi
     olefile                   0.46                     py37_0
     openssl                   1.1.1g               h7b6447c_0    anaconda
     opt-einsum                3.3.0                    pypi_0    pypi
     packaging                 20.4                       py_0
     pandas                    1.0.5            py37h0da4684_0    conda-forge
     pandoc                    1.19.2                        0    conda-forge
     pandocfilters             1.4.2                    py37_1
     pango                     1.42.4               h7062337_4    conda-forge
     parquet-cpp               1.5.1                         2    conda-forge
     parso                     0.7.0                      py_0
     partd                     1.1.0                      py_0    conda-forge
     pathspec                  0.7.0                      py_0
     pcre                      8.44                 he6710b0_0
     pexpect                   4.8.0                    py37_1
     pickleshare               0.7.5                 py37_1001
     pillow                    7.2.0            py37hb39fc2d_0
     pip                       20.1.1                   py37_1    anaconda
     pixman                    0.38.0               h7b6447c_0
     plac                      1.1.3                    pypi_0    pypi
     pluggy                    0.13.1                   py37_0
     pre-commit                2.6.0            py37hc8dfbb8_0    conda-forge
     pre_commit                2.6.0                         0    conda-forge
     preshed                   3.0.2                    pypi_0    pypi
     prometheus_client         0.8.0                      py_0
     prompt-toolkit            3.0.5                      py_0
     protobuf                  3.12.2                   pypi_0    pypi
     psutil                    5.7.0            py37h7b6447c_0
     ptyprocess                0.6.0                    py37_0
     py                        1.9.0                      py_0
     pyarrow                   0.17.1          py37h1234567_10_cuda    conda-forge
     pyasn1                    0.4.8                    pypi_0    pypi
     pyasn1-modules            0.2.8                    pypi_0    pypi
     pycodestyle               2.6.0                      py_0
     pycparser                 2.20                       py_2
     pyflakes                  2.2.0                      py_0
     pygments                  2.6.1                      py_0
     pynvml                    8.0.4+4.g7c78212           dev_0    
     pyopenssl                 19.1.0                     py_1
     pyparsing                 2.4.7                      py_0
     pyrsistent                0.16.0           py37h7b6447c_0
     pysocks                   1.7.1                    py37_1
     pytest                    6.0.0rc1         py37hc8dfbb8_0    conda-forge
     python                    3.7.6           cpython_h8356626_6    conda-forge
     python-dateutil           2.8.1                      py_0
     python-graphviz           0.14.1             pyh9f0ad1d_0    conda-forge
     python_abi                3.7                     1_cp37m    conda-forge
     pytorch                   1.5.1           py3.7_cuda10.2.89_cudnn7.6.5_0    pytorch
     pytz                      2020.1                     py_0
     pyyaml                    5.3.1            py37h7b6447c_1
     pyzmq                     19.0.1           py37he6710b0_1
     rapidjson                 1.1.0             he1b5a44_1002    conda-forge
     re2                       2020.07.06           he1b5a44_1    conda-forge
     readline                  8.0                  h7b6447c_0
     recommonmark              0.6.0                      py_0    conda-forge
     regex                     2020.6.8         py37h7b6447c_0
     requests                  2.24.0                     py_0
     requests-oauthlib         1.3.0                    pypi_0    pypi
     rhash                     1.3.8                h1ba5d50_0
     rmm                       0.15.0a200713   py37_gf9e9c55_323    rapidsai-nightly
     rsa                       4.6                      pypi_0    pypi
     scipy                     1.4.1                    pypi_0    pypi
     send2trash                1.5.0                    py37_0
     setuptools                49.2.0                   py37_0
     simpervisor               0.3                        py_1    conda-forge
     six                       1.15.0                     py_0
     snappy                    1.1.8                he6710b0_0
     snowballstemmer           2.0.0                      py_0
     sortedcontainers          2.2.2                      py_0
     soupsieve                 2.0.1                    pypi_0    pypi
     spacy                     2.3.2                    pypi_0    pypi
     spdlog                    1.7.0                hc9558a2_0    conda-forge
     sphinx                    3.1.2                      py_0    conda-forge
     sphinx-copybutton         0.2.12             pyh9f0ad1d_0    conda-forge
     sphinx-markdown-tables    0.0.14             pyh9f0ad1d_1    conda-forge
     sphinx_rtd_theme          0.5.0              pyh9f0ad1d_0    conda-forge
     sphinxcontrib-applehelp   1.0.2                      py_0
     sphinxcontrib-devhelp     1.0.2                      py_0
     sphinxcontrib-htmlhelp    1.0.3                      py_0
     sphinxcontrib-jsmath      1.0.1                      py_0
     sphinxcontrib-qthelp      1.0.3                      py_0
     sphinxcontrib-serializinghtml 1.1.4                      py_0
     sphinxcontrib-websupport  1.2.3              pyh9f0ad1d_0    conda-forge
     sqlite                    3.31.1               h7b6447c_0
     srsly                     1.0.2                    pypi_0    pypi
     streamz                   0.5.3                    pypi_0    pypi
     tblib                     1.6.0                      py_0
     tensorboard               2.1.1                    pypi_0    pypi
     tensorflow-estimator      2.1.0                    pypi_0    pypi
     tensorflow-gpu            2.1.0                    pypi_0    pypi
     termcolor                 1.1.0                    pypi_0    pypi
     terminado                 0.8.3                    py37_0
     testpath                  0.4.4                      py_0
     tfdlpack-gpu              0.1.3                    pypi_0    pypi
     thinc                     7.4.1                    pypi_0    pypi
     thrift-cpp                0.13.0               h62aa4f2_2    conda-forge
     tk                        8.6.10               hbc83047_0
     toml                      0.10.1                     py_0
     toolz                     0.10.0                     py_0
     torchvision               0.2.2                      py_3    pytorch
     tornado                   6.0.4            py37h7b6447c_1
     tqdm                      4.48.0                   pypi_0    pypi
     traitlets                 4.3.3                    py37_0
     typed-ast                 1.4.1            py37h7b6447c_0
     typing_extensions         3.7.4.2                    py_0
     urllib3                   1.25.9                     py_0
     virtualenv                20.0.20          py37hc8dfbb8_1    conda-forge
     wasabi                    0.7.1                    pypi_0    pypi
     wcwidth                   0.2.5                      py_0
     webencodings              0.5.1                    py37_1
     werkzeug                  1.0.1                    pypi_0    pypi
     wheel                     0.34.2                   py37_0
     wrapt                     1.12.1                   pypi_0    pypi
     xorg-kbproto              1.0.7             h14c3975_1002    conda-forge
     xorg-libice               1.0.10               h516909a_0    conda-forge
     xorg-libsm                1.2.3             h84519dc_1000    conda-forge
     xorg-libx11               1.6.9                h516909a_0    conda-forge
     xorg-libxext              1.3.4                h516909a_0    conda-forge
     xorg-libxpm               3.5.13               h516909a_0    conda-forge
     xorg-libxrender           0.9.10            h516909a_1002    conda-forge
     xorg-libxt                1.1.5             h516909a_1003    conda-forge
     xorg-renderproto          0.11.1            h14c3975_1002    conda-forge
     xorg-xextproto            7.3.0             h14c3975_1002    conda-forge
     xorg-xproto               7.0.31            h14c3975_1007    conda-forge
     xz                        5.2.5                h7b6447c_0
     yaml                      0.2.5                h7b6447c_0
     yarl                      1.4.2            py37h7b6447c_0
     zeromq                    4.3.2                he6710b0_2
     zict                      2.0.0                      py_0
     zipp                      3.1.0                      py_0
     zlib                      1.2.11               h7b6447c_3
     zstd                      1.4.4                h0b5b093_3

Additional context The need for an explicit copy of the data view may affect NVTabular perfomance cc @benfred

kkraus14 commented 4 years ago

Closing as duplicate of #5615

rjzamora commented 4 years ago

Closing as duplicate of #5615

Ah - Completely forgot about that.