search

uwescience / raco

Compilation and rule-based optimization framework for relational algebra. Raco is the language, optimization, and query translation layer for the Myria project.
Other
72 stars 19 forks source link

Python UDF support in RACO #544

Closed BrandonHaynes closed 7 years ago

BrandonHaynes commented 7 years ago

@parmitam : this PR includes a merge of the lambda conversion logic and a number of other commits from master, which the new code depends on. Commit 2f0e8f6bfe1f7fdf5dcdb09c10535a017324e9fa contains all of my changes related to surfacing UDFs in MyriaL. @shrjain may be in a better position to review the small amount of logic related to lambda conversion, but it is straightforward and unlikely to break (plus there are a bunch of tests).

Note that I changed things up a little bit: instead of invoking a Python UDF as PYUDF('f', arg0, ..., argn), you just invoke it as a function: f(arg0, ..., argn). This seems like a better design, though myria-web will have to be modified to make RACO aware of the available UDFs.

coveralls commented 7 years ago

Coverage Status

Coverage decreased (-0.06%) to 91.924% when pulling 2f0e8f6bfe1f7fdf5dcdb09c10535a017324e9fa on blob_expr_UDF_unary into 55653947635ebeeed48c975f6a2c341a97b8f116 on master.

coveralls commented 7 years ago

Coverage Status

Coverage decreased (-0.04%) to 91.939% when pulling 2524e5af4e58e96ac338c5528c95f4f915ca8577 on blob_expr_UDF_unary into 55653947635ebeeed48c975f6a2c341a97b8f116 on master.

coveralls commented 7 years ago

Coverage Status

Coverage decreased (-0.04%) to 91.939% when pulling bd5c6f361e37acd46ea324e6c750fb7fe34e54d9 on blob_expr_UDF_unary into 55653947635ebeeed48c975f6a2c341a97b8f116 on master.

BrandonHaynes commented 7 years ago

Made some changes to increase compatibility with myria-python, but this PR is still blocking on that project.

coveralls commented 7 years ago

Coverage Status

Coverage decreased (-0.04%) to 91.939% when pulling 76afdf1721d235febfff67728fbf0fe48729d577 on blob_expr_UDF_unary into 55653947635ebeeed48c975f6a2c341a97b8f116 on master.

parmitam commented 7 years ago

Change Bytes_type to BLOB_type, changing the post endpoint to '/function', changing interface to not include inputSchema.

coveralls commented 7 years ago

Coverage Status

Coverage decreased (-0.04%) to 91.94% when pulling a7d9521166df0629577d9df18302e2d6cb4cce5f on blob_expr_UDF_unary into 55653947635ebeeed48c975f6a2c341a97b8f116 on master.

coveralls commented 7 years ago

Coverage Status

Coverage decreased (-0.05%) to 91.934% when pulling fe1c15792904c82e5c9e982b7a98eca05178d378 on blob_expr_UDF_unary into 55653947635ebeeed48c975f6a2c341a97b8f116 on master.

BrandonHaynes commented 7 years ago

@parmitam I've modified the Myria backend to support variadic functions, which are resolved at the point of invocation. This should work for the Python UDFs, which are now implicitly variadic. I believe that the REST API issues were already corrected.

parmitam commented 7 years ago

Looks good. Once we have the myria-python changes made, I'd like to test it end to end with myriaX.