Closed tshauck closed 3 weeks ago
Notes from comet community meeting:
CometSparkSessionExtension
injects rules into spark session extension
QueryPlanSerde
takes spark expression an serializes via protoAdd(left, right, _)
checks if supportedDataTypes(left.dataType)
.Operator
message Expr
... need to update expr.proto
(See Add
for example)
PhysicalPlanner
is the physical planner struct
create_expr
and create_plan
are two most important partscreate_expr
or create_plan
to generate the physical planspark_expr.expr_struct
is matched on then passed to appropriate partPhysicalExpr
is implemented for Cast
.failOnError
is a common change from 3.3 to 3.4+)
CometExpressionSuite
is for testing
checkSparkAnswerAndOperator
Questions:
CAST
is recently a hotbed of activity and high priority... tests are important@tshauck video link is https://drive.google.com/file/d/1POU4lFAZfYwZR8zV1X2eoLiAmc1GDtSP/view?usp=sharing
I think this is ready for review. The first draft I wrote was pretty slanted towards scalar function expressions since that's what I did w/ unhex, but after the meeting on Wed, I made some updates so hopefully it's more general. I'm not 100% sure on the things I haven't touched yet, so certainly open to feedback on any of it.
Here's what it looks like included in the docs...
This is looking great @tshauck. Could you fix the merge conflict?
Thanks, @andygrove -- I think the conflict is resolved now.
@andygrove Conflict should actually be fixed now, thanks @kazuyukitanimura
Which issue does this PR close?
Part of #370 (haven't added anything on cast or aggregate, mainly scalar func)
Rationale for this change
Make it easier for people to contribute new expressions.
What changes are included in this PR?
A new markdown file in the contributor guide.
How are these changes tested?
~Not yet, I haven't tried building the docs. Will do before marking this PR ready.~
Built the docs and reviewed the HTML.