arrow-py / arrow

🏹 Better dates & times for Python
https://arrow.readthedocs.io
Apache License 2.0
8.63k stars 669 forks source link

Cannot append nullable string columns to table #1151

Closed agagrins closed 1 year ago

agagrins commented 1 year ago

Issue Description

It seems not possible to be able to append a nullable column to a table.

Start off with a table. This works fine.

$ python
>>> import pyarrow
>>> table = pyarrow.Table.from_pylist([{"a": None}], pyarrow.schema([pyarrow.field("a", pyarrow.string(), nullable=True)]))

Then try to add another similar column. This does work.

>>> table = table.append_column(pyarrow.field("b", pyarrow.string(), nullable=True), [["x"]])

Then try to add another similar column. This does not work.

>>> table = table.append_column(pyarrow.field("c", pyarrow.string(), nullable=True), [[None]])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "pyarrow/table.pxi", line 4495, in pyarrow.lib.Table.append_column
  File "pyarrow/table.pxi", line 4452, in pyarrow.lib.Table.add_column
  File "pyarrow/error.pxi", line 144, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 100, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Field type did not match data type

System Info

This is on Windows 11 under WSL2

>>> pyarrow.show_versions()
pyarrow version info
--------------------
Package kind              : python-wheel-manylinux2014
Arrow C++ library version : 11.0.0
Arrow C++ compiler        : GNU 10.2.1
Arrow C++ compiler flags  :  -fdiagnostics-color=always
Arrow C++ git revision    :
Arrow C++ git description :
Arrow C++ build type      : release