ibis-project / ibis

the portable Python dataframe library

https://ibis-project.org

Apache License 2.0

4.3k stars 537 forks source link

fix: replace NaNs with None in some backends when loading from pandas dataframe #9094

Closed chloeh13q closed 1 week ago

chloeh13q commented 2 weeks ago

Description of changes

Examples were broken on MySQL backend and PySpark backend when there are null values in a numeric column.

Druid, PySpark, RW don't support examples.

Exasol - did not test
Flink - broken
Impala - did not test
MSSQL - broken #9095
MySQL - fixed
Oracle - did not test
PostgreSQL - fixed
Snowflake - did not test

Issues closed

8792

chloeh13q commented 2 weeks ago

For postgres, we should be a little more specific in the conversion.

Sounds good.

I also think this could work for mssql and a few other backends that currently can't load a large chunk of the examples.

I think mssql examples are broken for a different reason, at least on my local; I filed a ticket #9095 yesterday and I'm getting Incorrect syntax on all of the examples. Do you have other backends in mind? Or you just meant this in a generic sense

chloeh13q commented 2 weeks ago

I'm having trouble spinning up some of the backends so I'm not able to test whether the examples work in these backends. But I can confirm that the examples are now fixed in MySQL and postgres with this PR!

gforsyth commented 2 weeks ago

xref #9110

cpcloud commented 1 week ago

Do we have some tests for this? How do we know this won't regress?

gforsyth commented 1 week ago

We should enable some of the examples tests for postgres and mysql (I don't think we need to run all of them)