Move preprocessing steps from OpenOA project prepare to ENTR warehouse

[ejsimley]

Data processing steps needed to convert and QC data for OpenOA can be moved to the ENTR warehouse using dbt. Example of current steps can be found here: https://github.com/entralliance/OpenOA/blob/main/examples/project_ENGIE.py

We agreed to start with the simple "power in watts" and "calculate energy" steps starting here: https://github.com/entralliance/OpenOA/blob/89fff3a521413403d8eb349bb65810b7a498f004/examples/project_ENGIE.py#L128

[ejsimley]

We decided to have OpenOA convert power to W if they are not in W in the ENTR warehouse. But ENTR will be responsible for deriving an energy_kwh column using power and the SCADA data frequency. So OpenOA can query the energy column without having to derive it separately.

[ejsimley]

sql statements are written to calculate energy from power.: https://github.com/entralliance/entr_warehouse/blob/de721fc2cb21eedd71daebd61a88ddab241e4db4/models/marts/exposures/openoa/intermediate/int_openoa_wtg_scada__filtered.sql#L21-L31

Todo:

• We still need to write this as a macro that can be invoked during staging (or at any point)
• Need to write a macro to determine number of seconds in SCADA frequency data. This will be used to calculate energy to replace the hardcoded value

[lewisarmistead]

Thoughts on sequencing work:

• transition frequency columns into one (frequency in seconds) column
• genericize(?) power -> energy conversion steps into macro & potentially integrate into generic SCADA table or break out into another table of derived values
  • goal is to have macro emulate the convert_power_to_energy method used here
• emulate Project_ENGIE.prepare's drop_duplicates (used here in OpenOA) method as macro & incorporate into OpenOA models
• emulate Project_ENGIE.prepare's min/max filtering of temperature values (implemented here)
• Change units to those preferred by OpenOA (if different), e.g. kW to W conversion implemented here
• change range of pitch values from -180 to 180 as implemented here
• create macro for calculating wind direction from U- and V- components in reanalysis data (see here) & implement similarly to power -> energy conversion
  • also done on ERA-5 data here
• understand what's happening with the datetime normalization step here and implement in pre-processing steps - @ejsimley, I may need some help understanding what's happening there
• @ejsimley - do we need to do any other column renaming on the warehouse side, or will that all be handled by OpenOA?

Based on that plan, here are the generic macros we need:

• calculated tag macros for
  • energy from power
  • -180-180 angles from 0-360 angles
  • direction from U- and V-wind components
• unit conversion macros, e.g. kW to W - however, this may be best to leave as explicit transformations on values for specific use cases requiring units different than standard ENTR units
• flag/filter macros for
  • stale data
  • min/max threshold violation