search

xitongsys / parquet-go

pure golang library for reading/writing parquet file
Apache License 2.0
1.27k stars 293 forks source link

Arrow schema conversion does not regard `Nullable` #474

Closed sdressler closed 2 years ago

sdressler commented 2 years ago

When writing with an Arrow schema, the Parquet schemas fields are all "REQUIRED" regardless of whether the Arrow schema defined Nullable: true. For example:

fields := []arrow.Field{
    arrow.Field{
        Name: "foo",
        Type: arrow.PrimitiveTypes.Int64,
        Nullable: true,
    },
}
schema := arrow.NewSchema(fields, nil)

The snippet will result in a parquet schema where "foo" is required.

denbilyk commented 2 years ago

I have the same. It is very critical for me. Any assumption when will it be fixed?