If you use an arrow_json::ReaderBuilder to read a json file, and specify a schema that includes a map that shouldn't allow nullable values, you can still read files that have nulls in the actual json map.
To Reproduce
use std::{fs::File, io::BufReader, sync::Arc};
use arrow::datatypes::{DataType, Field, Schema};
fn main() {
let schema = Arc::new(Schema::new(vec![
Field::new("str", DataType::Utf8, false),
Field::new_map(
"map",
"entries",
Field::new("key", DataType::Utf8, false),
Field::new("value", DataType::Utf8, false), // value is not nullable
false,
false
)
]));
let file = File::open("test.json").unwrap();
let mut json = arrow_json::ReaderBuilder::new(schema).build(BufReader::new(file)).unwrap();
let batch = json.next().unwrap().unwrap();
println!("Batch: {batch:#?}");
}
Describe the bug
If you use an
arrow_json::ReaderBuilder
to read a json file, and specify a schema that includes a map that shouldn't allow nullable values, you can still read files that have nulls in the actual json map.To Reproduce
And use this json file:
Running produces:
Note I've included the
str
field so you can easily see that the right thing happens if you change your .json file toYou will get:
Expected behavior
Expect an error similar to what happens when
str
field is set to null.Additional context