Establish process for importing raw operational data into ENTR warehouse

entralliance / entr_runtime

ENTR Alliance is an owner-operator led initiative to create open data stack within the clean energy sector. ENTR is a distribution of existing open-source tools, frameworks, and standards, packaged together to accelerate the transition to clean energy. Join us!

http://www.entralliance.com

MIT License

6 stars 3 forks source link

Establish process for importing raw operational data into ENTR warehouse #20

Closed ejsimley closed 1 year ago

ejsimley commented 2 years ago

@lewisarmistead I wanted to create a card for this task (which uses dbt, I believe), but please update it with a better description.

ejsimley commented 2 years ago

Ideas

Use Spark to read in csv files
User specifies column mapping to ENTR column names
dbt converts wide tables to ~4 column tables in standard ENTR format