Closed RinChanNOWWW closed 1 year ago
And I have a question that how can we select rows of a nested type? For example: Struct
.
When I add selected_rows
to Struct
array, I meet such problem:
Decoding Int32 "Plain"-encoded required , index-filtered parquet pages.
I can guarantee all columns in the Struct is the same length in my use case.
cc @jorgecarleitao
This happens because it not allows selected_rows
in nested type.
// Nested Decoder
fn build_state(
&self,
page: &'a DataPage,
dict: Option<&'a Self::Dictionary>,
) -> Result<Self::State> {
let is_optional =
page.descriptor.primitive_type.field_info.repetition == Repetition::Optional;
let is_filtered = page.selected_rows().is_some();
match (page.encoding(), dict, is_optional, is_filtered) {
(Encoding::PlainDictionary | Encoding::RleDictionary, Some(dict), false, false) => {
ValuesDictionary::try_new(page, dict).map(State::RequiredDictionary)
}
(Encoding::PlainDictionary | Encoding::RleDictionary, Some(dict), true, false) => {
ValuesDictionary::try_new(page, dict).map(State::OptionalDictionary)
}
(Encoding::Plain, _, true, false) => Values::try_new::<P>(page).map(State::Optional),
(Encoding::Plain, _, false, false) => Values::try_new::<P>(page).map(State::Required),
_ => Err(utils::not_implemented(page)),
}
}
How about enabling selected_rows
in nested type and assert the length of every columns to be the same in the StructIterator
?
Ohh, that is correct - yes, we should add support for that in nested types also.
Base: 85.12% // Head: 85.09% // Decreases project coverage by -0.03%
:warning:
Coverage data is based on head (
c35aecd
) compared to base (06f0675
). Patch coverage: 0.00% of modified lines in pull request are covered.
:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.
It's useful to let user change the
selected_rows
during iteration.For example: