US-NW-WACM: Reporting negative gas values

[FelixDQ]

It happens for a few hours every few days.

From Kibana:

invalid point: {'datetime': datetime.datetime(2020, 10, 28, 3, 0, tzinfo=tzlocal()), 'production': {'coal': 2515, 'gas': -316, 'hydro': 717, 'solar': 0, 'wind': 142}, 'storage': {}, 'source': 'eia.gov', 'zoneKey': 'US-NW-WACM', 'schemaVersion': 3}, reason:US-NW-WACM: key gas has negative value -316

Reproduce error with:

> python test_parser.py US-NW-WACM --target_datetime '2020-02-28'
[
...
 {'datetime': datetime.datetime(2020, 2, 27, 18, 0, tzinfo=<UTC>),
  'production': {'coal': 2854, 'gas': -156, 'hydro': 764, 'solar': 73, 'wind': 731},
  'source': 'eia.gov',
  'storage': {},
  'zoneKey': 'US-NW-WACM'},
 {'datetime': datetime.datetime(2020, 2, 27, 17, 0, tzinfo=<UTC>),
  'production': {'coal': 2920, 'gas': -217, 'hydro': 843, 'solar': 71, 'wind': 729},
  'source': 'eia.gov',
  'storage': {},
  'zoneKey': 'US-NW-WACM'},
...
]

[willbeaufoy]

I think it's a problem with the source data, this is the API call that the parser makes, and it contains the negative values:

http://api.eia.gov/series/?api_key=4da724ba12182d36382ad3bc755bbc7f&series_id=EBA.WACM-ALL.NG.NG.H&start=20200227T00Z&end=20200228T00Z

The data series is called 'Net generation from natural gas' - is it possible that at certain times the gas plants are using more electricity than they are producing? Or is it more likely to be a bug in the data?

[ssmssam]

From their FAQ: https://www.eia.gov/tools/faqs/faq.php?id=101&t=3 Electricity net generation is the amount of gross electricity generation a generator produces minus the electricity used to operate the power plant. These electricity uses include fuel handling equipment, water pumps, combustion and cooling air fans, pollution control equipment, and other electricity needs.

It sounds plausible that at certain times it could be using more electricity than it is producing for example when it is idling. I've tried to look for the actual electricity generated to grid without deducting the consumption at the plant but can't see it there. If it was, we could isolate the usage of the plant and work out the true amount of electricity being consumed.

In theory, do we want the electricity output to grid + consumption at the plant to get the true amount of electricity being consumed or do we only care about the electricity output to grid?

[ssmssam]

Clicking through links on here allows you to see graphs of net generation going back to 2018 https://www.eia.gov/opendata/qb.php?category=3390184

This shows negative generation for all fuel types at some point I think. Also the typical gas value seems to have dropped from around 500 MWh in May 2019 to around 60 MWh now which seems like quite a small gas plant. It seems likely this could be idling and consuming more energy at various parts of the day.

[Kongkille]

@FelixDQ I can't recall, is this issue still relevant? I believe we discussed this some time ago.

[FelixDQ]

I don't think that we have done anything regarding this issue. However, looking at the last two months of data it doesn't seem like it's been happening recently. The historical data haven't changed though, so running the following command still results in an error.

poetry run python test_parser.py US-NW-WACM --target_datetime '2020-02-28'

This means that we still have lots of small gaps in the historical data that could be fixed by addressing this issue.