search

segmentio / parquet-go

Go library to read/write Parquet files
https://pkg.go.dev/github.com/segmentio/parquet-go
Apache License 2.0
341 stars 102 forks source link

Optional *time.Time panics #497

Open comunidadio opened 1 year ago

comunidadio commented 1 year ago 
type ExampleRow struct {
    SnapshotTimestamp      *time.Time `parquet:"snapshotTimestamp,timestamp(millisecond),optional"`
}

When using GenericWriter[ExampleRow].Write, it panics with:

panic: timestamp is an invalid parquet tag: snapshotTimestamp *time.Time [timestamp]

goroutine 18 [running]:
github.com/segmentio/parquet-go.throwInvalidTag(...)
    github.com/segmentio/parquet-go@v0.0.0-20230330194643-f785677b9a75/schema.go:454

I also tried optional time.Time without pointer, assuming "zero" time (time.IsZero) would be counted as NULL, but it writes 'invalid' (zero) date to parquet instead (0001-01-01...). Perhaps handling time.IsZero as NULL is indeed a simpler/better way to handle optionality for timestamps.

comunidadio commented 1 year ago

Looks like unmerged PR at https://github.com/segmentio/parquet-go/pull/472 could allow a workaround - tho not using time.Time directly but int64.

kevinburkesegment commented 1 year ago

Apologies to make more work for you, but we've decided to move development on this project to a new organization at https://github.com/parquet-go/parquet-go to ensure its long term success. We appreciate your contribution and would appreciate if you could reopen this ticket there if it is still relevant.