search

entralliance / entr_runtime

ENTR Alliance is an owner-operator led initiative to create open data stack within the clean energy sector. ENTR is a distribution of existing open-source tools, frameworks, and standards, packaged together to accelerate the transition to clean energy. Join us!
http://www.entralliance.com
MIT License
6 stars 3 forks source link

Demonstrate implementation of OpenOA functions as UDFs in ENTR #24

Open ejsimley opened 2 years ago

ejsimley commented 2 years ago

We will likely want to call some OpenOA functions as UDFs from ENTR SQL. To demonstrate this, we discussed an initial implementation of the power curve fitting toolkit functions: https://github.com/entralliance/OpenOA/blob/main/operational_analysis/toolkits/power_curve/functions.py

We may also want to implement some of the filtering toolkit functions used to preprocess the SCADA data before fitting a power curve.

ejsimley commented 2 years ago

Jordan is finding that a PySpark context is needed to register Python function as UDF, so we should find a way to run UDF from dbt. We can revisit this once Jordan demonstrates the UDF example.

ejsimley commented 2 years ago

This issue is a stretch goal for Q3 deliverable. Try to create an MVP example notebook demonstrating UDFs by end of June.

lewisarmistead commented 2 years ago

Throwing Jordan's link from Slack here for reference: https://stackoverflow.com/questions/46010682/how-to-register-custom-udf-jar-in-hivethriftserver2