[Data Issue]: Puget Sound Energy doesn't account for imports

[KTibow]

When did this happen?

2024-11-18 5:14 PM

What zones are affected?

PSEI

What is the problem?

I might be missing something since I'm new to this, but I think Puget Sound Energy's is missing imports import logic is broken.

If you compare the metrics between Electricity Maps and EIA:		Electricity Maps	EIA
Energy being consumed	1.24 GW	3.956 GW
Electricity generation	1.24 GW	1.284 GW
Electricity imports	N/A	2.656 GW (as of a similar time 2 days ago)
Electricity exports	N/A	0.057 GW (as of a similar time 2 days ago)

it shows that most of PSE's energy actually comes from imports. Electricity Maps isn't accounting for this at all, which I believe is causing the carbon intensity estimate to be significantly off.

* While writing this I did notice that the import data isn't real time. Hopefully it can be estimated based on historical data in the typical way.

[KTibow]

I dived a little deeper into this, and my conclusion is that I can't do anything to help.

I tested the parser with `python test_parser.py "US-NW-BPAT->US-NW-PSEI" exchange`, and it definitely works (for historical data).

``` Parser result: [{'datetime': datetime.datetime(2024, 11, 15, 23, 0, tzinfo=tzutc()), 'netFlow': 2300.0, 'sortedZoneKeys': 'US-NW-BPAT->US-NW-PSEI', 'source': 'eia.gov', 'sourceType': }, {'datetime': datetime.datetime(2024, 11, 16, 0, 0, tzinfo=tzutc()), 'netFlow': 2458.0, 'sortedZoneKeys': 'US-NW-BPAT->US-NW-PSEI', 'source': 'eia.gov', 'sourceType': }, {'datetime': datetime.datetime(2024, 11, 16, 1, 0, tzinfo=tzutc()), 'netFlow': 2627.0, 'sortedZoneKeys': 'US-NW-BPAT->US-NW-PSEI', 'source': 'eia.gov', 'sourceType': }, {'datetime': datetime.datetime(2024, 11, 16, 2, 0, tzinfo=tzutc()), 'netFlow': 2579.0, 'sortedZoneKeys': 'US-NW-BPAT->US-NW-PSEI', 'source': 'eia.gov', 'sourceType': }, {'datetime': datetime.datetime(2024, 11, 16, 3, 0, tzinfo=tzutc()), 'netFlow': 2505.0, 'sortedZoneKeys': 'US-NW-BPAT->US-NW-PSEI', 'source': 'eia.gov', 'sourceType': }, {'datetime': datetime.datetime(2024, 11, 16, 4, 0, tzinfo=tzutc()), 'netFlow': 2424.0, 'sortedZoneKeys': 'US-NW-BPAT->US-NW-PSEI', 'source': 'eia.gov', 'sourceType': }, {'datetime': datetime.datetime(2024, 11, 16, 5, 0, tzinfo=tzutc()), 'netFlow': 2284.0, 'sortedZoneKeys': 'US-NW-BPAT->US-NW-PSEI', 'source': 'eia.gov', 'sourceType': }, {'datetime': datetime.datetime(2024, 11, 16, 6, 0, tzinfo=tzutc()), 'netFlow': 2158.0, 'sortedZoneKeys': 'US-NW-BPAT->US-NW-PSEI', 'source': 'eia.gov', 'sourceType': }, {'datetime': datetime.datetime(2024, 11, 16, 7, 0, tzinfo=tzutc()), 'netFlow': 2010.0, 'sortedZoneKeys': 'US-NW-BPAT->US-NW-PSEI', 'source': 'eia.gov', 'sourceType': }, {'datetime': datetime.datetime(2024, 11, 16, 8, 0, tzinfo=tzutc()), 'netFlow': 1890.0, 'sortedZoneKeys': 'US-NW-BPAT->US-NW-PSEI', 'source': 'eia.gov', 'sourceType': }, {'datetime': datetime.datetime(2024, 11, 16, 9, 0, tzinfo=tzutc()), 'netFlow': 1810.0, 'sortedZoneKeys': 'US-NW-BPAT->US-NW-PSEI', 'source': 'eia.gov', 'sourceType': }, {'datetime': datetime.datetime(2024, 11, 16, 10, 0, tzinfo=tzutc()), 'netFlow': 1766.0, 'sortedZoneKeys': 'US-NW-BPAT->US-NW-PSEI', 'source': 'eia.gov', 'sourceType': }, {'datetime': datetime.datetime(2024, 11, 16, 11, 0, tzinfo=tzutc()), 'netFlow': 1721.0, 'sortedZoneKeys': 'US-NW-BPAT->US-NW-PSEI', 'source': 'eia.gov', 'sourceType': }, {'datetime': datetime.datetime(2024, 11, 16, 12, 0, tzinfo=tzutc()), 'netFlow': 1757.0, 'sortedZoneKeys': 'US-NW-BPAT->US-NW-PSEI', 'source': 'eia.gov', 'sourceType': }, {'datetime': datetime.datetime(2024, 11, 16, 13, 0, tzinfo=tzutc()), 'netFlow': 1862.0, 'sortedZoneKeys': 'US-NW-BPAT->US-NW-PSEI', 'source': 'eia.gov', 'sourceType': }, {'datetime': datetime.datetime(2024, 11, 16, 14, 0, tzinfo=tzutc()), 'netFlow': 1988.0, 'sortedZoneKeys': 'US-NW-BPAT->US-NW-PSEI', 'source': 'eia.gov', 'sourceType': }, {'datetime': datetime.datetime(2024, 11, 16, 15, 0, tzinfo=tzutc()), 'netFlow': 2210.0, 'sortedZoneKeys': 'US-NW-BPAT->US-NW-PSEI', 'source': 'eia.gov', 'sourceType': }, {'datetime': datetime.datetime(2024, 11, 16, 16, 0, tzinfo=tzutc()), 'netFlow': 2365.0, 'sortedZoneKeys': 'US-NW-BPAT->US-NW-PSEI', 'source': 'eia.gov', 'sourceType': }, {'datetime': datetime.datetime(2024, 11, 16, 17, 0, tzinfo=tzutc()), 'netFlow': 2456.0, 'sortedZoneKeys': 'US-NW-BPAT->US-NW-PSEI', 'source': 'eia.gov', 'sourceType': }, {'datetime': datetime.datetime(2024, 11, 16, 18, 0, tzinfo=tzutc()), 'netFlow': 2483.0, 'sortedZoneKeys': 'US-NW-BPAT->US-NW-PSEI', 'source': 'eia.gov', 'sourceType': }, {'datetime': datetime.datetime(2024, 11, 16, 19, 0, tzinfo=tzutc()), 'netFlow': 2473.0, 'sortedZoneKeys': 'US-NW-BPAT->US-NW-PSEI', 'source': 'eia.gov', 'sourceType': }, {'datetime': datetime.datetime(2024, 11, 16, 20, 0, tzinfo=tzutc()), 'netFlow': 2485.0, 'sortedZoneKeys': 'US-NW-BPAT->US-NW-PSEI', 'source': 'eia.gov', 'sourceType': }, {'datetime': datetime.datetime(2024, 11, 16, 21, 0, tzinfo=tzutc()), 'netFlow': 2479.0, 'sortedZoneKeys': 'US-NW-BPAT->US-NW-PSEI', 'source': 'eia.gov', 'sourceType': }, {'datetime': datetime.datetime(2024, 11, 16, 22, 0, tzinfo=tzutc()), 'netFlow': 2456.0, 'sortedZoneKeys': 'US-NW-BPAT->US-NW-PSEI', 'source': 'eia.gov', 'sourceType': }, {'datetime': datetime.datetime(2024, 11, 16, 23, 0, tzinfo=tzutc()), 'netFlow': 2504.0, 'sortedZoneKeys': 'US-NW-BPAT->US-NW-PSEI', 'source': 'eia.gov', 'sourceType': }, {'datetime': datetime.datetime(2024, 11, 17, 0, 0, tzinfo=tzutc()), 'netFlow': 2510.0, 'sortedZoneKeys': 'US-NW-BPAT->US-NW-PSEI', 'source': 'eia.gov', 'sourceType': }, {'datetime': datetime.datetime(2024, 11, 17, 1, 0, tzinfo=tzutc()), 'netFlow': 2544.0, 'sortedZoneKeys': 'US-NW-BPAT->US-NW-PSEI', 'source': 'eia.gov', 'sourceType': }, {'datetime': datetime.datetime(2024, 11, 17, 2, 0, tzinfo=tzutc()), 'netFlow': 2437.0, 'sortedZoneKeys': 'US-NW-BPAT->US-NW-PSEI', 'source': 'eia.gov', 'sourceType': }, {'datetime': datetime.datetime(2024, 11, 17, 3, 0, tzinfo=tzutc()), 'netFlow': 2350.0, 'sortedZoneKeys': 'US-NW-BPAT->US-NW-PSEI', 'source': 'eia.gov', 'sourceType': }, {'datetime': datetime.datetime(2024, 11, 17, 4, 0, tzinfo=tzutc()), 'netFlow': 2232.0, 'sortedZoneKeys': 'US-NW-BPAT->US-NW-PSEI', 'source': 'eia.gov', 'sourceType': }, {'datetime': datetime.datetime(2024, 11, 17, 5, 0, tzinfo=tzutc()), 'netFlow': 2071.0, 'sortedZoneKeys': 'US-NW-BPAT->US-NW-PSEI', 'source': 'eia.gov', 'sourceType': }, {'datetime': datetime.datetime(2024, 11, 17, 6, 0, tzinfo=tzutc()), 'netFlow': 1892.0, 'sortedZoneKeys': 'US-NW-BPAT->US-NW-PSEI', 'source': 'eia.gov', 'sourceType': }, {'datetime': datetime.datetime(2024, 11, 17, 7, 0, tzinfo=tzutc()), 'netFlow': 1694.0, 'sortedZoneKeys': 'US-NW-BPAT->US-NW-PSEI', 'source': 'eia.gov', 'sourceType': }] ```

But the bit that refetches data doesn't seem to be working; no signs of exchange data show up even when I rewind time. I'm also not sure how to enable forecasting/estimating for exchanges.

If more stuff was open source, I would have a much better understanding of exchanges as source code is the best documentation, and I might've fixed the old data fetcher by now. However, these bits are "internal", so I can't do anything.

I hope you guys can fix this issue. Since I can only see half of the picture, you are the only people who can. I'm going to see if I can write my own estimator in the meantime.

[VIKTORVAV99]

I dived a little deeper into this, and my conclusion is that I can't do anything to help.

I tested the parser with python test_parser.py "US-NW-BPAT->US-NW-PSEI" exchange, and it definitely works (for historical data). But the bit that refetches data doesn't seem to be working; no signs of exchange data show up even when I rewind time. I'm also not sure how to enable forecasting/estimating for exchanges.

Because currently you can only go back 24 hours but the exchange data is delayed by 36 hours or so; we are working on feature that will allow you to go further back and are adding a option for 72 hour data.

If more stuff was open source, I would have a much better understanding of exchanges as source code is the best documentation, and I might've fixed the old data fetcher by now. However, these bits are "internal", so I can't do anything.

We will not be open sourcing our estimation models or estimator backend at this time. This might change in the future but it's nothing that is planned.

I hope you guys can fix this issue. Since I can only see half of the picture, you are the only people who can. I'm going to see if I can write my own estimator in the meantime.

Sadly the core of the problem is that the data has such a high delay that most basic estimation models would be very wrong. Total imports would be fairly easy to do but not the flow from the individual exchanges, and in order to not produce and show false data we have opted not to estimate these for now. We are working on more accurate alternatives though.

[jarek]

Sadly the core of the problem is that the data has such a high delay that most basic estimation models would be very wrong. Total imports would be fairly easy to do but not the flow from the individual exchanges, and in order to not produce and show false data we have opted not to estimate these for now. We are working on more accurate alternatives though.

IMO you shouldn't be reporting data for Puget Sound Energy and zones like it (where >30% or in this case >60% of energy is usually imported) at all when the imports data hasn't come in yet. The real-time data is just wrong now. Or at best, only show them in Production mode, and not in Consumption mode.

[VIKTORVAV99]

Sadly the core of the problem is that the data has such a high delay that most basic estimation models would be very wrong. Total imports would be fairly easy to do but not the flow from the individual exchanges, and in order to not produce and show false data we have opted not to estimate these for now. We are working on more accurate alternatives though.

IMO you shouldn't be reporting data for Puget Sound Energy and zones like it (where >30% or in this case >60% of energy is usually imported) at all when the imports data hasn't come in yet. The real-time data is just wrong now. Or at best, only show them in Production mode, and not in Consumption mode.

In general I agree with you and we could hide it from the app but we can unfortunately not stop reporting the data altogether via the API.

Usually when a zone is hidden or removed from the app we get complaints from the other side saying it should be available.

And there is technically no solution in place to only send data in production mode (even if that would be possible to build if we needed). I'll make sure to pass it along to the team and see what they think about it. 👍🏼