[x] I have read through the quick start and installation sections of the README.
Info
Info
Value
Operating System
MacOS
Geni Version
0.3.8
JDK
1.8
Spark Version
3.0.2
Problem / Steps to reproduce
It seems like it is impossible to create a boolean column from all false values using records->dataset because they get recognized as null columns. Here is a failing tests.
FAIL On records->dataset - should work for bool columns at (dataset_creation_test.clj:143)
Expected:
#<org.apache.spark.sql.types.StructType@2e83f3f5 StructType(StructField(i,LongType,true), StructField(s,StringType,true), StructField(b,BooleanType,true))>
Actual:
#<org.apache.spark.sql.types.StructType@67b8b180 StructType(StructField(i,LongType,true), StructField(s,StringType,true), StructField(b,NullType,true))>
FAIL On records->dataset - should work for bool columns at (dataset_creation_test.clj:146)
Expected:
[[0 "A" false] [1 "B" false] [2 "C" false]]
Actual:
([0 "A" nil] [1 "B" nil] [2 "C" nil])
Diffs: in [0 2] expected false, was nil
in [1 2] expected false, was nil
in [2 2] expected false, was nil
The same behavior applies to map->dataset and table->dataset. If any of the booleans are true, then the schema is understood correctly.
Info
Problem / Steps to reproduce
It seems like it is impossible to create a boolean column from all
false
values usingrecords->dataset
because they get recognized as null columns. Here is a failing tests.and here is the output.
The same behavior applies to
map->dataset
andtable->dataset
. If any of the booleans aretrue
, then the schema is understood correctly.