US-MISO has large negative 'unknown' value

[corradio]

invalid point: {'zoneKey': 'US-MISO', 'datetime': datetime.datetime(2018, 6, 28, 0, 30, tzinfo=tzfile('/usr/share/zoneinfo/America/New_York')), 'production': {'coal': 39985.0, 'gas': 16080.0, 'nuclear': 11306.0, 'wind': 1292.0, 'unknown': -313.0}, 'storage': {}, 'source': 'misoenergy.org', 'schemaVersion': 1}, reason:US-MISO: key unknown has negative value -313.0

[alixunderplatz]

@corradio Can you extract the unknown time series for the last couple of days from the database and put it in a graph? or is the negative unknown value not recorded? Unknown category looks a bit like solar during the day, but might also be pumped storage (I feel like there was some info in the MISO issue back then... but I have no time to check atm)

[corradio]

Unfortunately it's not recorded as it doesn't pass the validation :(

On Thu, Jun 28, 2018 at 9:16 AM Alex notifications@github.com wrote:

@corradio https://github.com/corradio Can you extract the unknown time series for the last couple of days from the database and put it in a graph? or is the negative unknown value not recorded? Unknown category looks a bit like solar during the day, but might also be pumped storage (I feel like there was some info in the MISO issue back then... but I have no time to check atm)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/tmrowco/electricitymap/issues/1482#issuecomment-400936586, or mute the thread https://github.com/notifications/unsubscribe-auth/ABlEKKNmRfzZD-_qrLgPVnjWYIGtvDNqks5uBILkgaJpZM4U60Ta .

[systemcatch]

Quoting the Miso site;

The category listed as “Other” is the combination of Hydro, Pumped Storage Hydro, Diesel, Demand Response Resources, External Asynchronous Resources and a varied assortment of solid waste, garbage and wood pulp burners."

So pumped hydro?

[alixunderplatz]

@systemcatch the negative part most likely comes from pumping, yeah. But then we should probably dismiss "other" as a whole, as pumping while generating will reduce the output of this category and therefore the emissions, too. Would be nice if somebody could drop a timeseries of the positive part of this category anyway

[brunolajoie]

This is the historical data we have for MISO unknown production (in MW) (mean value when available: 1000MW)

[jarek]

This is still happening, still due to the negative "other" - mostly at night, supporting the pumped storage theory.

We could write and ask if there is a separate source listing real-time pumping use, though scanning through their website I didn't see one - and https://www.misoenergy.org/markets-and-operations/RTDataAPIs/ has a quite extensive list already...

Another idea would be to try to establish usual or average mix of what is in "other", to help us make the decision on what to do with it.

If the maximum value of "other" is ~3 GW, this is in worst case around 5% of the total demand. It's probably not the worst approximation to just drop the "other" altogether. (From a CO2eq point of view, we have more inaccuracy from different kinds of coal anyway...)

[corradio]

What about we update the parser and set to 0 negative values that are below 1% of total production (instead of looking at an absolute criterion, let's look at a relative one).

[jarek]

Data point: from https://en.wikipedia.org/wiki/List_of_pumped-storage_hydroelectric_power_stations, https://en.wikipedia.org/wiki/Ludington_Pumped_Storage_Power_Plant is in MISO area (Michigan) with listed generating capacity of 2172 MW. That actually lines up kinda nicely with the values in charts in Bruno's post.

Another idea: look through MISO website for historical reports as to fuel sources per year, and compare with what is listed in "other"

[jarek]

Idea about simple possible solution: whenever "unknown"/MISO's "other" is negative, assign it entirely to hydro storage; when it's positive, assign it to "unknown".

That is, when "other" is -1700 MW, that is possibly made up of say 200 MW run-of-river hydro, 100 MW diesel, and -2000 MW of hydro storage. Instead we assign the whole -1700 MW to hydro storage.

Some data from yesterday (Friday before a long weekend): MISO peak load was just under 100 GW; in the morning once "other" returned to positive the low was around 65 GW. The highest value seen for "other" was 3.2 GW near 2pm local time when the load peaked.

Upsides:

• we have more data, including about hydro storage
• we have continuous data
• overall I think it won't hurt accuracy that much - given 65 GW low load, even a 3 GW error is less than 5%, this is less than the error in e.g. estimating CO2eq intensity of "coal"

Downsides:

• this hides some thermal generation in "other" overnight -> but it explicitly doesn't include coal nor natural gas (see systemcatch's quote above), and it's significantly less than 5% of the generation
• this under-estimates the actual energy being stored in hydro storage -> but in the current situation we don't show the hydro storage at all
• hydro storage discharge will be in "other" and thus given a carbon penalty -> but this happens currently anyway
• energy stored in hydro storage will never be explicitly consumed -> does that break anything on the backend?
• we make assumptions, which the site generally doesn't like doing

Thoughts?

[brunolajoie]

In terms of real-world physics, this looks fair enough, especially as there is less chance than average to have both storage AND other thermal (diesel, solid waste, garbage and wood pulp burners) at the same time. Storage is when prices are the lowest

I'll let oli have a say about the backend data pipeline.

[corradio]

Thank you for the extensive review @jarek. As a principle, I tend to try to have reliable data in databases. This means I strive to have data that is complete in the sense of:

• When looking at hydro data, it actually IS hydro (i.e. we know that it's hydro)
• When looking at hydro data, we know it includes all of the hydro (i.e. there's no other hydro that has been added to other categories)

I feel we're cutting too many corners here. The biggest problem is that at some point we might consider the discharge intensity to be the averaged intensity of stored energy. This might turn out to be very different than the diesel/run of river average - hence I'm afraid we're making a mistake by mixing them together.

I'm not satisfied with my own answer though, because on the other hand, simply ignoring data when it's negative doesn't seem a good idea either.

Another suggestion is to simply truncate the unknown category (set to 0 when negative). This would bias the total load amount, but would not change the mix. It would however assume that no storage is going on. Finally, if we can argue that unknown is mostly storage, we should map all unknown to hydro storage. If we can argue that unknown is mostly not storage, we should consider it as such. However, it seems like this latter assumption doesn't hold, as the value is largely negative. If none of those cases can be argued for, then we have a problem.. I'll try to think about it.

On Sun, Sep 2, 2018 at 7:10 PM Bruno Lajoie notifications@github.com wrote:

In terms of real-world physics, this looks fair enough, especially as there is little change to have both storage AND other thermal (diesel, solid waste, garbage and wood pulp burners) at the same time!

I'll let oli have a say about the backend data pipeline.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/tmrowco/electricitymap-contrib/issues/1482#issuecomment-417945032, or mute the thread https://github.com/notifications/unsubscribe-auth/ABlEKBeLiiNrZ-HSf_JTFJ0-o-s4BS6Vks5uXBD3gaJpZM4U60Ta .

[jarek]

An alternative is to dig up statistical data on what actually makes up this "unknown" category over longer periods of time. MISO might have it on their website... Unfortunately I don't have time to go searching at the moment.

Last couple of days MISO hasn't gone negative overnight, it looks like this as a sample:

[jarek]

OK, happening again, e.g. currently -421 MW at 2:45 local time

[corradio]

Closing due to inactivity. Thank you all for your inputs!