Closed KnightChess closed 2 weeks ago
@xushiyan cc
@xushiyan hello, Is there a problem with my test method in my local?
hello, Is there a problem with my test method in my local?
ignore checkstyle, fetch the latest commit, it work again, but can not report python test error.
Attention: Patch coverage is 90.07092%
with 14 lines
in your changes missing coverage. Please review.
Project coverage is 89.32%. Comparing base (
e23e6ed
) to head (f1ce54d
). Report is 2 commits behind head on main.
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.
@xushiyan cc:
@KnightChess awesome contribution! let me take a look. i might push some quick fixes just FYI to move faster.
@KnightChess do you think you can address the main comment in the next few days? then i can polish further if needed and land this. Trying to get this in the upcoming release within 2 weeks 🙂 (cutting RC branch with a week)
@xushiyan sorry for reply late, I will address these two days
@xushiyan Hello, I couldn't find an implementation similar to ScalarValue, and I am not very familiar with Arrow yet, and. There is a certain learning curve involved, which might delay the progress of this PR. Could you please help improve this PR?
@xushiyan cc, I try to use arrow Scalar<ArrayRef>
to replace datafusion ScalarValue
, and modified some suggestions to repair.
@xushiyan cc, I try to use arrow
Scalar<ArrayRef>
to replace datafusionScalarValue
, and modified some suggestions to repair.
@KnightChess Thanks. I was traveling. Will take a look later today.
I was wrapping up my vacation 😄 just now getting back to update this:
Changes I've made:
PartitionPruner
to handle pruning logic, taking care of hudi configs like hive style and url encoded paths.foo > 10
, bar != a
, etc, only apply AND
to them.>, >=, <, <=, =, !=
to avoid rabbit hole of implementing sql parser; IN
and NOT IN
can be achieved anyway.There are more follow up work to do on datafusion integration side, which I'll jot down in the GH issue.
@xushiyan thanks review
Description
Add filtering capabilities to table API, currently only partition fields are applicable. Multiple predicates are
AND
together.Supported operators are:
>, >=, <, <=, =, !=
.For #47
How are the changes test-covered