schema: fix handling of optional slice value

segmentio / parquet-go

Go library to read/write Parquet files

https://pkg.go.dev/github.com/segmentio/parquet-go

Apache License 2.0

341 stars 102 forks source link

schema: fix handling of optional slice value #414

Closed kevinburkesegment closed 1 year ago

kevinburkesegment commented 1 year ago

Previously we'd encode an optional slice as Optional(Repeated(Struct)), which treated the entire slice as optional, but Parquet needs us to treat this as Repeated(Optional(struct)) where each element of the struct is optional.

The handling for this is a little bit messy, but it's also messy to pass the context down further, so we end up with a special case inside of the struct handling.

Fixes #400.

kevinburkesegment commented 1 year ago

okay @achille-roussel I think I've addressed all of your comments