Open galipremsagar opened 2 years ago
Cudf supports constructing a StructColumn
from a StructArray
.
In [14]: pa_struct = pa.StructArray.from_arrays([pa.array([1, 2, 3]), pa.array(['a', 'b', 'c'])], names=['num', 'str'])
In [16]: cudf.Series.from_arrow(pa_struct)
Out[16]:
0 {'num': 1, 'str': 'a'}
1 {'num': 2, 'str': 'b'}
2 {'num': 3, 'str': 'c'}
dtype: struct
I believe the error you see here has to do with having DictionaryArray
as one of the fields in the StructArray
.
In [5]: r = cudf.Series.from_arrow(pa.StructArray.from_arrays([s.to_arrow(), f], names=['a', 'd']))
In [6]: type(r)
Out[6]: cudf.core.series.Series
In [8]: r._column.children
Out[8]:
(<cudf.core.column.categorical.CategoricalColumn object at 0x7f6459ceda70>
-- dictionary:
[
1,
2,
3
]
-- indices:
[
0,
1,
2
]
dtype: category,
<cudf.core.column.numerical.NumericalColumn object at 0x7f645a5a53b0>
[
1,
2,
3
]
dtype: int64)
The result is well formed. But cannot convert the column back to arrow:
In [11]: r.to_arrow()
---------------------------------------------------------------------------
NotImplementedError Traceback (most recent call last)
<ipython-input-11-77c2df0309ec> in <module>
----> 1 r.to_arrow()
~/cudf/python/cudf/cudf/core/single_column_frame.py in to_arrow(self)
213 ]
214 """
--> 215 return self._column.to_arrow()
216
217 @property
~/cudf/python/cudf/cudf/core/column/struct.py in to_arrow(self)
68 {
69 field: child.type
---> 70 for field, child in zip(self.dtype.fields, children)
71 }
72 )
~/cudf/python/cudf/cudf/core/dtypes.py in fields(self)
289 return {
290 field.name: cudf.utils.dtypes.cudf_dtype_from_pa_type(field.type)
--> 291 for field in self._typ
292 }
293
~/cudf/python/cudf/cudf/core/dtypes.py in <dictcomp>(.0)
289 return {
290 field.name: cudf.utils.dtypes.cudf_dtype_from_pa_type(field.type)
--> 291 for field in self._typ
292 }
293
~/cudf/python/cudf/cudf/utils/dtypes.py in cudf_dtype_from_pa_type(typ)
215 return cudf.core.dtypes.Decimal64Dtype.from_arrow(typ)
216 else:
--> 217 return cudf.api.types.pandas_dtype(typ.to_pandas_dtype())
218
219
~/compose/etc/conda/cuda_11.2/envs/rapids/lib/python3.7/site-packages/pyarrow/types.pxi in pyarrow.lib.DataType.to_pandas_dtype()
NotImplementedError: dictionary<values=int64, indices=int8, ordered=0>
This issue has been labeled inactive-30d
due to no recent activity in the past 30 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. This issue will be labeled inactive-90d
if there is no activity in the next 60 days.
This issue has been labeled inactive-90d
due to no recent activity in the past 90 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed.
Describe the bug We currently support converting a
category
column incudf
to aDictionaryArray
inpyarrow
:But we don't seem to be supporting the construction of
StructDtype
&StructColumn
from our constructors:Steps/Code to reproduce bug Follow this guide http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports to craft a minimal bug report. This helps us reproduce the issue you're having and resolve the issue more quickly.
Environment overview (please complete the following information)
Environment details Please run and paste the output of the
cudf/print_env.sh
script here, to gather any other relevant environment detailsClick here to see environment details
Additional context Add any other context about the problem here.