When second page appeared(in my case only for single column, didn't check if multiple), because of this line my slice contains only 1674 elements.
$ parquet pages ~/my-awesome-file.parquet
Column: Column1.key_value.value.SomeStruct.SomeField
--------------------------------------------------------------------------------
page type enc count avg size size rows nulls min / max
0-0 data _ D 1674 141.38 B 231.128 kB 1664 0
0-1 data _ D 23 143.39 B 3.221 kB 23 0
Column: Column2
--------------------------------------------------------------------------------
page type enc count avg size size rows nulls min / max
0-0 data _ _ 1687 8.00 B 13.180 kB 1687 0
.
.
.
Just for testing purpose I tried to switch to this code and on second call it fetched missing 23 rows.
f, err := parquet.OpenFile(r, size)
if err != nil {
return nil, err
}
rows := make([]T, f.NumRows())
n, err := reader.Read(rows)
if err != nil {
return nil, err
}
if n != len(rows) {
n, err = reader.Read(rows[n:])
if err != nil {
return nil, err
}
}
Hi! Seems like there is a bug inside
Reader
or maybe I'm doing something wrong, details:I'm using this code to read rows from in-mem parquet buffer and it works good until column has only 1 page.
When second page appeared(in my case only for single column, didn't check if multiple), because of this line my slice contains only
1674
elements.Just for testing purpose I tried to switch to this code and on second call it fetched missing 23 rows.
Thanks in advance!