search

data-apis / dataframe-interchange-tests

Test suite for the dataframe interchange protocol
MIT License
3 stars 2 forks source link

Erroneous 0 row behaviour with upstream modin and pandas versions #8

Open honno opened 2 years ago

honno commented 2 years ago

I wrote this initially for the modin issue tracker, but realised this is probably not appropriate (yet) for modin. This could actually be an upstream pandas issue too. Dumping this issue here to track for now.

Describe the problem

When a (modin) dataframe containing 0 rows, it cannot be interchanged to a library supporting the interchange protocol. This includes interchanging modin dataframes to modin.

Source code / logs

Note you have to monkey patch pandas.errors.DataError to pandas.core.base.DataError for modin to work with upstream pandas.

>>> from modin import pandas as mpd
>>> df = mpd.DataFrame({"foo_col": mpd.Series([], dtype="int64")})  # dtype can be anything
>>> from modin.pandas.utils import from_dataframe as modin_from_dataframe
>>> modin_from_dataframe(df)
NotImplementedError: Non-string object dtypes are not supported yet
Full traceback ```python >>> modin_from_dataframe(df) .../modin/modin/pandas/utils.py:123, in from_dataframe(df) 120 from modin.core.execution.dispatching.factories.dispatcher import FactoryDispatcher 121 from .dataframe import DataFrame --> 123 return DataFrame(query_compiler=FactoryDispatcher.from_dataframe(df)) .../modin/modin/core/execution/dispatching/factories/dispatcher.py:175, in FactoryDispatcher.from_dataframe(cls, *args, **kwargs) 172 @classmethod 173 @_inherit_docstrings(factories.BaseFactory._from_dataframe) 174 def from_dataframe(cls, *args, **kwargs): --> 175 return cls.__factory._from_dataframe(*args, **kwargs) .../modin/modin/core/execution/dispatching/factories/factories.py:197, in BaseFactory._from_dataframe(cls, *args, **kwargs) 189 @classmethod 190 @doc( 191 _doc_io_method_template, (...) 195 ) 196 def _from_dataframe(cls, *args, **kwargs): --> 197 return cls.io_cls.from_dataframe(*args, **kwargs) .../modin/modin/core/io/io.py:120, in BaseIO.from_dataframe(cls, df) 105 @classmethod 106 def from_dataframe(cls, df): 107 """ 108 Create a Modin QueryCompiler from a DataFrame supporting the DataFrame exchange protocol `__dataframe__()`. 109 (...) 118 QueryCompiler containing data from the DataFrame. 119 """ --> 120 return cls.query_compiler_cls.from_dataframe(df, cls.frame_cls) .../modin/modin/core/storage_formats/pandas/query_compiler.py:279, in PandasQueryCompiler.from_dataframe(cls, df, data_cls) 277 @classmethod 278 def from_dataframe(cls, df, data_cls): --> 279 return cls(data_cls.from_dataframe(df)) .../modin/modin/core/dataframe/pandas/dataframe/dataframe.py:2968, in PandasDataframe.from_dataframe(cls, df) 2963 from modin.core.dataframe.pandas.exchange.dataframe_protocol.from_dataframe import ( 2964 from_dataframe_to_pandas, 2965 ) 2967 ErrorMessage.default_to_pandas(message="`from_dataframe`") -> 2968 pandas_df = from_dataframe_to_pandas(df) 2969 return cls.from_pandas(pandas_df) .../modin/modin/core/dataframe/pandas/exchange/dataframe_protocol/from_dataframe.py:68, in from_dataframe_to_pandas(df, n_chunks) 66 pandas_dfs = [] 67 for chunk in df.get_chunks(n_chunks): ---> 68 pandas_df = protocol_df_chunk_to_pandas(chunk) 69 pandas_dfs.append(pandas_df) 71 pandas_df = pandas.concat(pandas_dfs, axis=0, ignore_index=True) .../modin/modin/core/dataframe/pandas/exchange/dataframe_protocol/from_dataframe.py:102, in protocol_df_chunk_to_pandas(df) 100 raise ValueError(f"Column {name} is not unique") 101 col = df.get_column_by_name(name) --> 102 dtype = col.dtype[0] 103 if dtype in ( 104 DTypeKind.INT, 105 DTypeKind.UINT, 106 DTypeKind.FLOAT, 107 DTypeKind.BOOL, 108 ): 109 columns[name], buf = primitive_column_to_ndarray(col) .../pandas/pandas/_libs/properties.pyx:36, in pandas._libs.properties.CachedProperty.__get__() .../pandas/pandas/core/exchange/column.py:125, in PandasColumn.dtype(self) 118 if infer_dtype(self._col) == "string": 119 return ( 120 DtypeKind.STRING, 121 8, 122 dtype_to_arrow_c_fmt(dtype), 123 Endianness.NATIVE, 124 ) --> 125 raise NotImplementedError("Non-string object dtypes are not supported yet") 126 else: 127 return self._dtype_from_pandasdtype(dtype) NotImplementedError: Non-string object dtypes are not supported yet ```

This goes for interchanging a modin dataframe to pandas,

>>> from pandas.api.exchange import from_dataframe as pandas_from_dataframe
>>> pandas_from_dataframe(df)
NotImplementedError: Non-string object dtypes are not supported yet

but interchanging a pandas dataframe to modin works just fine.

>>> df2 = pd.DataFrame({"foo": pd.Series([], dtype="int64")})
>>> modin_from_dataframe(df)
Empty DataFrame
Columns: [foo]
Index: []

System information

honno commented 2 years ago

With https://github.com/modin-project/modin/issues/4652 I also get an inscrutable error when interchanging modin dataframes with categoricals with modin itself, that I don't get using normal pandas. Traceback is too large for GitHub lol, will document properly if it persists.