This is problematic in the case of streaming mode or even partitioning a dataframe and treating each partition in batch mode it would yield two different schemas in the case that column in partition X ends up with mainly null values.
Expected behavior
I think list.max() and other accessors should always return the list dtype, e.g list[str] should always return str dtype.
Checks
Reproducible example
Log output
No response
Issue description
This is problematic in the case of streaming mode or even partitioning a dataframe and treating each partition in batch mode it would yield two different schemas in the case that column in partition X ends up with mainly null values.
Expected behavior
I think list.max() and other accessors should always return the list dtype, e.g list[str] should always return str dtype.
The behaviour is not consistent.
Installed versions