Open shaneding opened 3 years ago
This came up while @shaneding was working on #7997 , as this presents when ~accessing~ returning list.leaves
Thinking more, the monotonically increasing index in the original example is due to how this is constructed here (from a cuDF object). I think this may simply be fine as is. The computed index is just the collection of the the underlying indexes.
I think specifically that the character_tokenize
function resets the index because there isn't a clean way to maintain it, so when the function is run against each dask partition, each partition gets a reset index.
This issue has been labeled inactive-90d
due to no recent activity in the past 90 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed.
@beckernick do you think there's anything to do here, or should we close this issue as expected behavior?
Hmm the current results here seem to have gotten worse. @beckernick WDYT of this? I'm not familiar with the API or what it's meant to do, but this seems quite wrong.
In [105]: import cudf
...: import dask_cudf
...: s = cudf.Series(["ab", "cd", "ef", "gh", "ij"])
...: ds = dask_cudf.from_cudf(s, 2)
...: print(ds.compute())
...: print(ds.map_partitions(lambda x: x.str.character_tokenize(), meta=ds._meta).compute())
0 ab
1 cd
2 ef
3 gh
4 ij
dtype: object
0 a
0 b
1 c
1 d
2 e
2 f
3 j
3
4
4
dtype: object
Describe the bug When using map_partitions() on a
dask_cudf
dataframe we can potentially get an unexpected index. This does not affect.compute()
by default as it reconstructs a global index.Steps/Code to reproduce bug
This prints
Expected behavior Would love to discuss what the expected output would be, but below is what I would expect.
Environment details
Click here to see environment details