Integrate Hourly Electric Grid Monitor (EIA 930)

[grgmiller]

I wanted to request integration of the hourly electric grid monitor / EIA 930 data, found at https://www.eia.gov/beta/electricity/gridmonitor/dashboard/electric_overview/US48/US48. This site provides near real-time hourly data for all balancing authorities, I believe integrated from the OASIS data platform from each BA.

Starting in July 2015, it includes data for hourly demand, hourly day-ahead demand forecast, net generation, and net interchange with each adjacent balancing authority. Starting in July 2018, data including net generation by fuel type and subregion demand are now available.

If you navigate to "Download Data" then "Balancing authority / region files" you can download excel sheets, which seem to be updated on a daily basis. There is also an API available.

There are possibly overlaps in this dataset with other currently integrated or planned datasets, although this might represent a more aggregated / realtime source for the data.

Below is an example of the column headers located in one of these files (for CAISO): BA UTC time Date Hour Local time Time zone Generation only DF [Demand Forecast] D [Demand] NG [Net Generation] TI [Total Interchange] Imputed D Imputed NG Imputed TI Adjusted D Adjusted NG Adjusted TI NG: COL NG: NG NG: NUC NG: OIL NG: WAT NG: SUN NG: WND NG: OTH NG: UNK Imputed COL Gen Imputed NG Gen Imputed NUC Gen Imputed OIL Gen Imputed WAT Gen Imputed SUN Gen Imputed WND Gen Imputed OTH Gen Imputed UNK Gen Adjusted COL Gen Adjusted NG Gen Adjusted NUC Gen Adjusted OIL Gen Adjusted WAT Gen Adjusted SUN Gen Adjusted WND Gen Adjusted OTH Gen Adjusted UNK Gen AZPS [Net Interchange with each balancing authority v] BANC BPAT CFE IID LDWP NEVP PACW SRP TIDC WALC Subregion PGAE [Demand from each subregion v] Subregion SDGE Subregion SCE Subregion VEA

[zaneselvans]

@truggles from Carnegie is also interested in this dataset, and has an archive of some data up here: https://github.com/truggles/EIA_Cleaned_Hourly_Electricity_Demand_Data

It does look interesting! Hopefully it's something we can get integrated, if there isn't a good bulk access option available already. Thus far we really haven't been trying to go for close to real time data, and have been sticking to the final annual releases, but as the overall ETL pipeline stabilizes, hopefully some kind of automated update and publishing can be done more frequently.

[truggles]

@grgmiller, it is great to see more people are interested in this type of data. A colleague and I started using this data recently and noticed some issues with it. As of the start of September 2019, about 2.2% of the hours in the database were missing demand information. Also, there are some significant outliers in the data likely due to reporting imperfections, ~0.5% of the hours. We spent some time developing algorithms to screen the data for these anomalies and to replace the screened values and missing values with imputed values.

While our output is not "real-time" anymore, it provides a complete data sets with the extreme outliers removed making the data ready for analysis. We are in the final stretch of putting together a manuscript detailing the work. We'll get this into PUDL soon. Let me know if you want to know more.

[grgmiller]

Good to know about the data quality issues... The EIA does make it clear that it is still in beta so hopefully the data quality will improve over time. I'll start taking a look at your code.

Another option to get ISO-level data could be to use https://github.com/WattTime/pyiso to pull the data directly from each ISO/OASIS. It looks like pyiso's coverage is not as comprehensive, but I wonder how the data quality for a larger BA like CAISO compares between pyiso and EIA-930. @truggles is this something you have looked at?

[truggles]

@grgmiller, we have not compared EIA-930 values against pyiso. That would be a good check. We began with the EIA information because of its complete coverage of the contiguous US. If you make any comparisons, please let us know. :-)