singularity-energy / open-grid-emissions

Tools for producing high-quality hourly generation and emissions data for U.S. electric grids
MIT License
67 stars 4 forks source link

Fix CAISO timestamp correction #300

Closed grgmiller closed 1 year ago

grgmiller commented 1 year ago

CAISO had previously been reporting start-of-hour values to EIA-930, instead of end-of-hour, as instructed, so we had been applying a timestamp correction to shift the values by +1 hours. However, it looks like starting on June 16, 2022, CAISO started correctly reporting end-of-hour values, so we want to end the correction on this date.

Since we have not yet published 2022 OGE data, this should not affect any of our current outputs.

grgmiller commented 1 year ago

So for CAISO specifically, I was able to compare the EIA-930 data with the data that CAISO reports directly on its website. If we look at the below charts, the green line shows the "benchmark" data from CAISO, and the red line shows the cleaned EIA-930 after implementing this time shift. We can see that the data generally aligns before June 16, and then continues to align even better after the 16th (I believe that when CAISO fixed this timestamp issue, they simultaneously fixed an issue where they had been reporting instantaneous MW rather than hourly average MW, which is why the post 6/16 data looks even better.

However, one thing you notice is that CAISO seems to have "fixed" this issue on June 14, then went back to the old method on June 15, then permanently implemented the change on June 16. Maybe this suggests that to be the most accurate, we should end the correction on 6/14, restart it for one day on 6/15, then end it again on 6/16.

Natural gas: image

Solar image

grgmiller commented 1 year ago

The new code fixes this issue: image

It looks like TEPC also fixed their data, but we will need to dig into when and submit a separate PR