Open jcmuel opened 3 months ago
In this issue (#18389), I combined the examples from #18119 and #18230, to show that assert_frame_equal
and assert_series_equal
are inconsistent in the way how they are treating Null
structs and structs with all Null
fields.
In #18230, @cmdlineluser suggested that the new behavior of list.drop_nulls()
should now be considered to be correct, and that function treats Null
and {Null}
as two different things.
But in #18119, it seems like the conclusion was that assert_series_equal
should treat Null
and {Null}
as the same.
This creates inconsistencies and this bug report identifies one of them.
Checks
Reproducible example
Log output
Issue description
The test first creates two DataFrames. Each of them contains a single column of datatype list of struct and a single row. One list contains a
Null
element whereas the other one contains a struct with allNull
values. All other elements are the same.assert_frame_not_equal
correctly confirms that the two DataFrames differ.assert_series_not_equal
falsely assumes that the Series that where used to build the DataFrame are the same.This is a contradiction.
Expected behavior
The most consistent expected behavior seems to be to treat
Null
and a struct with allNull
-fields as two different things, in the same way asassert_frame_not_equal
is already doing it.The test should pass.
Installed versions