Closed jtmiclat closed 1 year ago
@jtmiclat Thanks for the report!
In general, we don't yet support the pandas nullable dtypes such as boolean
.
As long as there are no missing values, adding boolean
to the DTYPE_OGR_FIELD_TYPES
mapping might be sufficient, but for missing values we will certainly need to add support for recognizing pd.NA as missing value. It might also be more efficient to add a support for having field data as both values + mask array.
@jorisvandenbossche I did some initial testing and adding boolean
to DTYPE_OGR_FIELD_TYPES
does address my issue but fails when there is a pd.NA in the column. The error message isn't super clear for the user
> OGR_F_SetFieldInteger(ogr_feature, field_idx, field_value)
E TypeError: an integer is required
pyogrio/_io.pyx:1631: TypeError
I think it is best to wait for support for recognizing pd.NA. Thanks!
I imagine that supporting writing dataframes with dtype="string"
falls into a similar category? - as that is also nullable I've been introducing pyogrio to some colleagues who are super impressed at the speed difference compared to fiona for reading large networks, and we came across the this behaviour difference with fiona.
Seems like dtypes string
and analogous (string[python]
, string[pyarrow]
as well as category
don't work out of the box, and need to be casted to object
.
Maybe we should change the title of this issue to indicate that it is a broader issue, or open another one ?
Some documentation in that regard would be welcome too.
@Oreilles renamed the issue to an ENH request to support nullable fields. I think category
and other custom dtype support would be a separate issue!
Hi, I was getting an error when using the
write_dataframe()
when using geodataframes containing booleans fromgoogle-cloud-bigquery
After some digging I figured out that the returned dtype for bool_field was a
pandas.BooleanDtype
/boolean
instead ofbool
and was able to replicated it without bigquery. It seems to be able to work fine with fiona but breaks with pyogrioMy hunch is to add
boolean
to https://github.com/geopandas/pyogrio/blob/75e8f13940fea6e30554115760275c7da978058c/pyogrio/_io.pyx#L60-L84 Thanks for the wonderful work!