Open severinh opened 9 months ago
Or option 3)
Infer Schema from first 100 rows but if it finds different datatypes after that it should raise/error (instead of in this case setting all new values to null which is as you mentioned suuuuuupeeeer dangerous; lucky you even noticed this silent error!)
Got burnt by this as well. All my tests passed but started causing errors in production. Created sample test cases to try to understand and could not replicate as they were under 100 values. Can be a dangerous footgun currently!
Checks
[X] I have checked that this issue has not already been reported.
[X] I have confirmed this bug exists on the latest version of Polars.
Reproducible example
Log output
Issue description
In our system, we read a column with a JSON value from a database into a DataFrame, and then decode it using Polars'
str.json_decode()
.We had a production incident where the data in the column was mostly (but not entirely)
null
(sparse data). In particular, the first 100 elements werenull
. The default behavior ofjson_decode
is to infer the data type from the first 100 values. So in this case, the inferred data type becamenull
, and as a result, the entire column becamenull
. The few sparse values were dropped on the floor, resulting in wrong calculations downstream.We solved the problem with an emergency release that set
infer_schema_length
toNone
, hereby checking the entire column.Expected behavior
I believe this is a dangerous default behavior in the presence of such sparse data.
I can think of two solutions:
infer_schema_length
toNone
, since that's the safest option. I realize though that this may not be desirable, since this could potentially be very expensive.Installed versions