beckernick
commented
4 years ago Maybe this could be resolved by using the align_indexes private method during concatenation
Closed rjzamora closed 4 years ago
beckernick
commented
4 years ago Maybe this could be resolved by using the align_indexes private method during concatenation
Describe the bug When using
axis=1to concatinate two cudf dataframes, the output does not match the pandas output when the indices do not match. More specifically, if both dataframes have N rows, and none of the indices match, the output of pandas.concat has length 2N, while that of cudf.concat has length N. This leads to inconsistent behavior between pandas- and cudf-based Dask dataframes.Steps/Code to reproduce bug The problem will occur when random float values are used to define the indices of both dataframes:
Expected behavior The output of
pandas.concatis correct in this case (the output dataframe must have more rows than the input dataframes if the indeces do not match).Environment overview (please complete the following information)
Environment details
Click here to see environment details
**git*** commit 3d9da9e5da2e731d3d63a066a6d44fe3465cbb5a (HEAD -> branch-0.11, upstream/branch-0.11, origin/branch-0.11, origin/HEAD) Merge: c27db8bcf 566771cab Author: Jake Hemstad
Date: Mon Dec 2 09:08:58 2019 -0600
Merge pull request #3466 from kaatish/bug-gather-nullmask
[REVIEW] Fix gather null mask allocation for invalid index
**git submodules***
b165e1fb11eeea64ccf95053e40f2424312599cc thirdparty/cub (v1.7.1)
63f644be44201467e3938d59ed9d89cc8725c35d thirdparty/jitify (remotes/origin/feature/api_v2_v0.10)
39125e0e476b960c2001f1ec76a3441335ff91b2 thirdparty/libcudacxx (0.8.1-94-g39125e0)
08bc464bd8f4d779e4294305aa7dadebcebcc507 thirdparty/libcudacxx/libcxx (heads/rapidsai-interop)
***OS Information***
DGX_NAME="DGX Server"
DGX_PRETTY_NAME="NVIDIA DGX Server"
DGX_SWBUILD_DATE="2019-04-24"
DGX_SWBUILD_VERSION="4.1.0"
DGX_COMMIT_ID="2c52306"
DGX_PLATFORM="DGX Server for DGX-1"
DGX_SERIAL_NUMBER="QTFCOU8220024"
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.2 LTS"
NAME="Ubuntu"
VERSION="18.04.2 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.2 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic
Linux dgx14 4.15.0-47-generic #50-Ubuntu SMP Wed Mar 13 10:44:52 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
***GPU Information***
Mon Dec 2 08:29:48 2019
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.87.00 Driver Version: 418.87.00 CUDA Version: 10.1 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla V100-SXM2... On | 00000000:06:00.0 Off | 0 |
| N/A 34C P0 57W / 300W | 504MiB / 32480MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 1 Tesla V100-SXM2... On | 00000000:07:00.0 Off | 0 |
| N/A 34C P0 41W / 300W | 11MiB / 32480MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 2 Tesla V100-SXM2... On | 00000000:0A:00.0 Off | 0 |
| N/A 32C P0 43W / 300W | 11MiB / 32480MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 3 Tesla V100-SXM2... On | 00000000:0B:00.0 Off | 0 |
| N/A 31C P0 41W / 300W | 11MiB / 32480MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 4 Tesla V100-SXM2... On | 00000000:85:00.0 Off | 0 |
| N/A 32C P0 44W / 300W | 11MiB / 32480MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 5 Tesla V100-SXM2... On | 00000000:86:00.0 Off | 0 |
| N/A 32C P0 41W / 300W | 11MiB / 32480MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 6 Tesla V100-SXM2... On | 00000000:89:00.0 Off | 0 |
| N/A 34C P0 43W / 300W | 11MiB / 32480MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 7 Tesla V100-SXM2... On | 00000000:8A:00.0 Off | 0 |
| N/A 30C P0 43W / 300W | 11MiB / 32480MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 49383 C .../rzamora/miniconda3/envs/ucx/bin/python 493MiB |
+-----------------------------------------------------------------------------+
***CPU***
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 80
On-line CPU(s) list: 0-79
Thread(s) per core: 2
Core(s) per socket: 20
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 79
Model name: Intel(R) Xeon(R) CPU E5-2698 v4 @ 2.20GHz
Stepping: 1
CPU MHz: 3230.360
CPU max MHz: 3600.0000
CPU min MHz: 1200.0000
BogoMIPS: 4390.02
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 51200K
NUMA node0 CPU(s): 0-19,40-59
NUMA node1 CPU(s): 20-39,60-79
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdt_a rdseed adx smap intel_pt xsaveopt cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts flush_l1d
***CMake***
/home/nfs/rzamora/miniconda3/envs/ucx/bin/cmake
cmake version 3.15.5
CMake suite maintained and supported by Kitware (kitware.com/cmake).
***g++***
/usr/bin/g++
g++ (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
***nvcc***
***Python***
/home/nfs/rzamora/miniconda3/envs/ucx/bin/python
Python 3.7.3
***Environment Variables***
PATH : /home/nfs/rzamora/.vscode-server/bin/f359dd69833dd8800b54d458f6d37ab7c78df520/bin:/home/nfs/rzamora/bin:/home/nfs/rzamora/.local/bin:/home/nfs/rzamora/miniconda3/envs/ucx/bin:/home/nfs/rzamora/miniconda3/condabin:/home/nfs/rzamora/.vscode-server/bin/f359dd69833dd8800b54d458f6d37ab7c78df520/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
LD_LIBRARY_PATH :
NUMBAPRO_NVVM : /usr/local/cuda-10.1/nvvm/lib64/libnvvm.so
NUMBAPRO_LIBDEVICE : /usr/local/cuda-10.1/nvvm/libdevice
CONDA_PREFIX : /home/nfs/rzamora/miniconda3/envs/ucx
PYTHON_PATH :
***conda packages***
/home/nfs/rzamora/miniconda3/condabin/conda
# packages in environment at /home/nfs/rzamora/miniconda3/envs/ucx:
#
# Name Version Build Channel
_libgcc_mutex 0.1 main
aiohttp 3.6.2 pypi_0 pypi
alabaster 0.7.12 py_0 conda-forge
appdirs 1.4.3 py_1 conda-forge
arrow-cpp 0.15.0 py37h090bef1_2 conda-forge
aspy.yaml 1.3.0 py_0 conda-forge
async-timeout 3.0.1 pypi_0 pypi
atomicwrites 1.3.0 py_0 conda-forge
attrs 19.3.0 py_0 conda-forge
autoconf 2.69 pl526h14c3975_9 conda-forge
automake 1.16.1 pl526_1003 conda-forge
babel 2.7.0 py_0 conda-forge
backcall 0.1.0 py_0 conda-forge
beautifulsoup4 4.8.1 pypi_0 pypi
black 19.10b0 py37_0 conda-forge
bleach 3.1.0 py_0 conda-forge
blis 0.4.1 pypi_0 pypi
bokeh 1.4.0 py37_0 conda-forge
boost-cpp 1.70.0 h8e57a91_2 conda-forge
bottleneck 1.3.1 pypi_0 pypi
brotli 1.0.7 he1b5a44_1000 conda-forge
bzip2 1.0.8 h516909a_1 conda-forge
c-ares 1.15.0 h516909a_1001 conda-forge
ca-certificates 2019.10.16 0
cached-property 1.5.1 py_0 conda-forge
catalogue 0.0.8 pypi_0 pypi
certifi 2019.9.11 py37_0
cffi 1.13.2 py37h8022711_0 conda-forge
cfgv 2.0.1 py_0 conda-forge
chardet 3.0.4 py37_1003 conda-forge
click 7.0 py_0 conda-forge
cloudpickle 1.2.2 py_0 conda-forge
cmake 3.15.5 hf94ab9c_0 conda-forge
cmake_setuptools 0.1.3 py_0 rapidsai-nightly
commonmark 0.9.1 py_0 conda-forge
cryptography 2.8 py37h72c5cf5_0 conda-forge
cudatoolkit 10.1.168 0 nvidia
cudf 0.11.0a0+3390.g7f5e054fe dev_0
cudnn 7.6.0 cuda10.1_0 nvidia
cupy 7.0.0rc1 dev_0
curl 7.65.3 hf8cf82a_0 conda-forge
cycler 0.10.0 pypi_0 pypi
cymem 2.0.3 pypi_0 pypi
cython 0.29.14 py37he1b5a44_0 conda-forge
cytoolz 0.10.1 py37h516909a_0 conda-forge
dask 2.8.1 dev_0
dask-cuda 0.10.0a1+27.g953b21b.dirty dev_0
dask-cudf 0.11.0a0+3390.g7f5e054fe dev_0
dask-labextension 1.0.3 pypi_0 pypi
decorator 4.4.1 py_0 conda-forge
defusedxml 0.6.0 py_0 conda-forge
distributed 2.8.1+8.g9dc8da37 dev_0
dlpack 0.2 he1b5a44_1 conda-forge
docutils 0.15.2 py37_0 conda-forge
double-conversion 3.1.5 he1b5a44_2 conda-forge
ds-itr 0.1.dev0 dev_0
editdistance 0.5.3 py37he1b5a44_0 conda-forge
entrypoints 0.3 py37_1000 conda-forge
expat 2.2.5 he1b5a44_1004 conda-forge
fastai 1.0.59 pypi_0 pypi
fastavro 0.22.7 py37h516909a_0 conda-forge
fastparquet 0.3.2 py37hdd07704_0
fastprogress 0.1.22 pypi_0 pypi
fastrlock 0.4 py37he1b5a44_1000 conda-forge
flake8 3.7.9 py37_0 conda-forge
flatbuffers 1.11.0 he1b5a44_0 conda-forge
freetype 2.10.0 he983fc9_1 conda-forge
fsspec 0.5.2 py_0 conda-forge
future 0.18.2 py37_0 conda-forge
gflags 2.2.2 he1b5a44_1002 conda-forge
glog 0.4.0 he1b5a44_1 conda-forge
gmp 6.1.2 hf484d3e_1000 conda-forge
grpc-cpp 1.23.0 h18db393_0 conda-forge
heapdict 1.0.1 py_0 conda-forge
hypothesis 4.44.2 py37_0 conda-forge
icu 64.2 he1b5a44_1 conda-forge
identify 1.4.7 py_0 conda-forge
idna 2.8 py37_1000 conda-forge
imagesize 1.1.0 py_0 conda-forge
importlib_metadata 0.23 py37_0 conda-forge
ipykernel 5.1.3 py37h5ca1d4c_0 conda-forge
ipython 7.9.0 py37h5ca1d4c_0 conda-forge
ipython_genutils 0.2.0 py_1 conda-forge
isort 4.3.21 py37_0 conda-forge
jedi 0.15.1 py37_0 conda-forge
jinja2 2.10.3 py_0 conda-forge
jpeg 9c h14c3975_1001 conda-forge
json5 0.8.5 py_0
jsonschema 3.1.1 py37_0 conda-forge
jupyter-server-proxy 1.1.0 pypi_0 pypi
jupyter_client 5.3.3 py37_1 conda-forge
jupyter_core 4.5.0 py_0 conda-forge
jupyterlab 1.1.4 pyhf63ae98_0
jupyterlab-nvdashboard 0.1.11 pypi_0 pypi
jupyterlab_server 1.0.6 py_0
kiwisolver 1.1.0 pypi_0 pypi
krb5 1.16.3 h05b26f9_1001 conda-forge
libblas 3.8.0 14_openblas conda-forge
libcblas 3.8.0 14_openblas conda-forge
libcurl 7.65.3 hda55be3_0 conda-forge
libedit 3.1.20170329 hf8c457e_1001 conda-forge
libevent 2.1.10 h72c5cf5_0 conda-forge
libffi 3.2.1 he1b5a44_1006 conda-forge
libgcc-ng 9.1.0 hdf63c60_0
libgfortran-ng 7.3.0 hdf63c60_2 conda-forge
liblapack 3.8.0 14_openblas conda-forge
libllvm8 8.0.1 hc9558a2_0 conda-forge
libopenblas 0.3.7 h6e990d7_3 conda-forge
libpng 1.6.37 hed695b0_0 conda-forge
libprotobuf 3.8.0 h8b12597_0 conda-forge
librmm 0.11.0b191112 cuda10.1_2 rapidsai-nightly
libsodium 1.0.17 h516909a_0 conda-forge
libssh2 1.8.2 h22169c7_2 conda-forge
libstdcxx-ng 9.1.0 hdf63c60_0
libtiff 4.1.0 hfc65ed5_0 conda-forge
libtool 2.4.6 h14c3975_1002 conda-forge
libuv 1.33.1 h516909a_0 conda-forge
llvmlite 0.30.0 py37h8b12597_1 conda-forge
locket 0.2.0 py_2 conda-forge
lz4-c 1.8.3 he1b5a44_1001 conda-forge
m4 1.4.18 h14c3975_1001 conda-forge
make 4.2.1 h14c3975_2004 conda-forge
markdown 3.0.1 pypi_0 pypi
markupsafe 1.1.1 py37h516909a_0 conda-forge
matplotlib 3.1.2 pypi_0 pypi
mccabe 0.6.1 py_1 conda-forge
mistune 0.8.4 py37h516909a_1000 conda-forge
more-itertools 7.2.0 py_0 conda-forge
msgpack-python 0.6.2 py37hc9558a2_0 conda-forge
multidict 4.5.2 pypi_0 pypi
murmurhash 1.0.2 pypi_0 pypi
mypy_extensions 0.4.3 py37_0 conda-forge
nbconvert 5.6.1 py37_0 conda-forge
nbformat 4.4.0 py_1 conda-forge
nbsphinx 0.4.3 py_0 conda-forge
nccl 2.4.6.1 cuda10.1_0 nvidia
ncurses 6.1 hf484d3e_1002 conda-forge
nodeenv 1.3.3 py_0 conda-forge
nodejs 10.13.0 he6710b0_0
notebook 6.0.1 py37_0 conda-forge
numba 0.46.0 py37hb3f55d8_1 conda-forge
numexpr 2.7.0 pypi_0 pypi
numpy 1.17.3 py37h95a1406_0 conda-forge
numpydoc 0.9.1 py_0 conda-forge
nvidia-ml-py3 7.352.0 pypi_0 pypi
nvstrings-cudaunknown 0.0.0.dev0 pypi_0 pypi
olefile 0.46 py_0 conda-forge
openssl 1.1.1d h7b6447c_3
packaging 19.2 py_0 conda-forge
pandas 0.24.2 py37hb3f55d8_0 conda-forge
pandoc 1.19.2 0 conda-forge
pandocfilters 1.4.2 py_1 conda-forge
parquet-cpp 1.5.1 2 conda-forge
parso 0.5.1 py_0 conda-forge
partd 1.0.0 py_0 conda-forge
pathspec 0.6.0 py_0 conda-forge
perl 5.26.2 h516909a_1006 conda-forge
pexpect 4.7.0 py37_0 conda-forge
pickleshare 0.7.5 py37_1000 conda-forge
pillow 6.2.1 py37h6b7be26_0 conda-forge
pip 19.3.1 py37_0 conda-forge
pkg-config 0.29.2 h14c3975_1005 conda-forge
plac 1.1.3 pypi_0 pypi
pluggy 0.13.0 py37_0 conda-forge
pre_commit 1.18.1 py37_0 conda-forge
preshed 3.0.2 pypi_0 pypi
prometheus_client 0.7.1 py_0 conda-forge
prompt_toolkit 2.0.10 py_0 conda-forge
psutil 5.6.5 py37h516909a_0 conda-forge
ptyprocess 0.6.0 py_1001 conda-forge
py 1.8.0 py_0 conda-forge
pyarrow 0.15.0 py37h8b68381_1 conda-forge
pycodestyle 2.5.0 py_0 conda-forge
pycparser 2.19 py37_1 conda-forge
pyflakes 2.1.1 py_0 conda-forge
pygments 2.4.2 py_0 conda-forge
pynvml 8.0.3 pypi_0 pypi
pyopenssl 19.0.0 py37_0 conda-forge
pyparsing 2.4.5 py_0 conda-forge
pyrsistent 0.15.5 py37h516909a_0 conda-forge
pysocks 1.7.1 py37_0 conda-forge
pytest 5.2.2 py37_0 conda-forge
pytest-asyncio 0.10.0 py37_1000 conda-forge
python 3.7.3 h33d41f4_1 conda-forge
python-dateutil 2.8.1 py_0 conda-forge
python-snappy 0.5.4 py37hee44bf9_1 conda-forge
pytz 2019.3 py_0 conda-forge
pyyaml 5.1.2 py37h516909a_0 conda-forge
pyzmq 18.1.0 py37h1768529_0 conda-forge
rapidjson 1.1.0 he1b5a44_1002 conda-forge
re2 2019.11.01 he1b5a44_0 conda-forge
readline 8.0 hf8c457e_0 conda-forge
recommonmark 0.6.0 py_0 conda-forge
regex 2019.11.1 py37h516909a_0 conda-forge
requests 2.22.0 py37_1 conda-forge
rhash 1.3.6 h14c3975_1001 conda-forge
rmm 0.11.0b191125 py37_40 rapidsai-nightly
scipy 1.3.3 pypi_0 pypi
send2trash 1.5.0 py_0 conda-forge
setuptools 41.6.0 py37_1 conda-forge
simpervisor 0.3 pypi_0 pypi
six 1.13.0 py37_0 conda-forge
snappy 1.1.7 he1b5a44_1002 conda-forge
snowballstemmer 2.0.0 py_0 conda-forge
sortedcontainers 2.1.0 py_0 conda-forge
soupsieve 1.9.5 pypi_0 pypi
spacy 2.2.3 pypi_0 pypi
sphinx 2.2.1 py_0 conda-forge
sphinx-markdown-tables 0.0.10 pypi_0 pypi
sphinx_rtd_theme 0.4.3 py_0 conda-forge
sphinxcontrib-applehelp 1.0.1 py_0 conda-forge
sphinxcontrib-devhelp 1.0.1 py_0 conda-forge
sphinxcontrib-htmlhelp 1.0.2 py_0 conda-forge
sphinxcontrib-jsmath 1.0.1 py_0 conda-forge
sphinxcontrib-qthelp 1.0.2 py_0 conda-forge
sphinxcontrib-serializinghtml 1.1.3 py_0 conda-forge
sphinxcontrib-websupport 1.1.2 py_0 conda-forge
sqlite 3.30.1 hcee41ef_0 conda-forge
srsly 0.2.0 pypi_0 pypi
streamz 0.5.2 pypi_0 pypi
tblib 1.4.0 py_0 conda-forge
terminado 0.8.3 py37_0 conda-forge
testpath 0.4.4 py_0 conda-forge
thinc 7.3.1 pypi_0 pypi
thrift 0.11.0 py37hf484d3e_0
thrift-cpp 0.12.0 hf3afdfd_1004 conda-forge
tk 8.6.9 hed695b0_1003 conda-forge
toml 0.10.0 py_0 conda-forge
toolz 0.10.0 py_0 conda-forge
torch 1.3.1 pypi_0 pypi
torchvision 0.4.2 pypi_0 pypi
tornado 6.0.3 py37h516909a_0 conda-forge
tqdm 4.39.0 pypi_0 pypi
traitlets 4.3.3 py37_0 conda-forge
typed-ast 1.4.0 py37h516909a_0 conda-forge
typing_extensions 3.7.4.1 py37_0 conda-forge
ucxpy 0.3 dev_0
uriparser 0.9.3 he1b5a44_1 conda-forge
urllib3 1.25.7 py37_0 conda-forge
virtualenv 16.7.5 py_0 conda-forge
wasabi 0.4.0 pypi_0 pypi
wcwidth 0.1.7 py_1 conda-forge
webencodings 0.5.1 py_1 conda-forge
wheel 0.33.6 py37_0 conda-forge
xz 5.2.4 h14c3975_1001 conda-forge
yaml 0.1.7 h14c3975_1001 conda-forge
yarl 1.3.0 pypi_0 pypi
zeromq 4.3.2 he1b5a44_2 conda-forge
zict 1.0.0 py_0 conda-forge
zipp 0.6.0 py_0 conda-forge
zlib 1.2.11 h516909a_1006 conda-forge
zstd 1.4.3 h3b9ef0a_0 conda-forge
Additional context Note that this comment from #5643 is related