wesm / feather

Feather: fast, interoperable binary data frame storage for Python, R, and more powered by Apache Arrow
Apache License 2.0
2.75k stars 167 forks source link

pyarrow.lib.ArrowTypeError: Expected bytes, got a 'dict' object #397

Open ivanpugachtd opened 2 years ago

ivanpugachtd commented 2 years ago

Hi! there is a problem when trying to load using pandas-gbq which using pyarrow a column of the list (array) or dictionary (json) type into the table, while the GBQ documentation says that structure types such as array or json are supported,

df = pd.DataFrame(
                {
                    "my_string": ["a", "b", "c"],
                    "my_int64": [1, 2, 3],
                    "my_float64": [4.0, 5.0, 6.0],
                    "my_bool1": [True, False, True],
                    "my_bool2": [False, True, False],
                    "my_struct": [{"test":"str1"},{"test":"str2"},{"test":"str3"}],
                }
            )
pandas_gbq.to_gbq(df, **gbq_params)

as a result, a stacktrace error occurs: in bq_to_arrow_array return pyarrow.Array.from_pandas(series, type=arrow_type) File "pyarrow/array.pxi", line 913, in pyarrow.lib.Array.from_pandas File "pyarrow/array.pxi", line 311, in pyarrow.lib.array File "pyarrow/array.pxi", line 83, in pyarrow.lib._ndarray_to_array File "pyarrow/error.pxi", line 122, in pyarrow.lib.check_status pyarrow.lib.ArrowTypeError: Expected bytes, got a 'dict' object

Can anyone help with it?