Add MISO->SPP exchange

[tmslaine]

US Southwest Power Pool has live generation mix and load data as a csv file under https://marketplace.spp.org/pages/generation-mix Timestamps seem to be UTC by default. I do not know if they have flow data somewhere.

Like ERCOT, SPP is also geoblocked.

[corradio]

Fyi I'm in the process of being cleared for access.

On Mar 4, 2018 21:07, "tmslaine" notifications@github.com wrote:

US Southwest Power Pool has live generation mix and load data as a csv file under https://marketplace.spp.org/pages/generation-mix Timestamps seem to be UTC by default. I do not know if they have flow data somewhere.

Like ERCOT, SPP is also geoblocked.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/tmrowco/electricitymap/issues/1177, or mute the thread https://github.com/notifications/unsubscribe-auth/ABlEKGOEIQjSnWZeof7eeFW5GELHwGV1ks5tbEmUgaJpZM4Sbehk .

[systemcatch]

Yup I started writing a parser for this one after I managed to get access. They allow access by ip but once you've got access to the csv you can pickle the data for easy use anywhere.

@tmslaine we should be able to add a timezone to their timestamps easy enough, data looks very good overall.

Map showing rough area SPP cover.

[systemcatch]

This is what I'm using as my data source when I'm using another ip address. You can load the pickle into pandas and see the full data.

SPPgenmix.pkl.zip

[corradio]

I don't get blocked for this content: @ ftp://pubftp.spp.org/Operational_Data/GEN_MIX/

That seems to be sufficient to build a parser/get historical data right?

[systemcatch]

More than enough! Can you get https://marketplace.spp.org/pages/interchange-trend? That shows all of the exchanges with other zones.

[corradio]

That one is blocked.. I'll reply to them.

On Wed, Mar 14, 2018 at 3:58 PM, Chris notifications@github.com wrote:

More than enough! Can you get https://marketplace.spp.org/ pages/interchange-trend? That shows all of the exchanges with other zones.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/tmrowco/electricitymap/issues/1177#issuecomment-373051489, or mute the thread https://github.com/notifications/unsubscribe-auth/ABlEKEeQGjd618mvThKK-Wzalt1aRrLMks5teTAagaJpZM4Sbehk .

[systemcatch]

There are 2 keys that make up a very small part of generation, 'Waste Disposal Services' and 'Waste Heat'. How do you want them mapped? I've not been able to find any info on what type of waste they burn.

[corradio]

I tend to put them those biomass when I don't know exactly what kind of waste it is.

On Wed, Mar 14, 2018 at 4:28 PM, Chris notifications@github.com wrote:

There are 2 keys that make up a very small part of generation, 'Waste Disposal Services' and 'Waste Heat'. How do you want them mapped? I've not been able to find any info on what type of waste they burn.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/tmrowco/electricitymap/issues/1177#issuecomment-373062789, or mute the thread https://github.com/notifications/unsubscribe-auth/ABlEKK1qLHJHjCxtgOaxcts3n5TGGc-sks5teTccgaJpZM4Sbehk .

[systemcatch]

@corradio One other interesting thing. The data source gives me timestamps in GMT (greenwich mean time) rather than US Central Time for some reason, 2 questions arise;

• Does the csv file return the same timestamp for you or is it taking into account the recipient's locale?
• How do you want to handle this, set to UTC, convert to US time, something else?

[corradio]

The CSV of the FTP should not change depending on who calls it? On our side, the parsers can return any timezone they want, as long as that information is in the datetime object.

[systemcatch]

I assume not but it was surprising to see a GMT timestamp returned.

[systemcatch]

Things left to do:

• [ ] Enable exchanges with MISO once we can verify that the data is correct.
• [ ] Find price data.

[alixunderplatz]

Regarding the price: As far as I remember, the (entire?) US wholesale electricity market works with a nodal model rather than coherent bidding zones. This means local constraints in the grid may influence the price of electricity in some areas.

This is on their front page (new OCR challenge in 3, 2, 1 ... 😸 ): You can observe the irregular distribution of the market price and even two hot-spots (with negative prices).

---

We should check if this node-model applies to the entire US and maybe open a seperate issue on the prices if it's true. Looking at the PJM price today, it looks a bit too crazy to be valid for the entire area:

[Hyric89]

Here's a list of figures for SPP generating capacity by energy source. It gives the total generating capacity and then a pie chart showing what percentage of the total pertains to which energy source. The catch is that these are 2016 figures. I'm going to crunch the numbers and round the numbers based on significant numbers.

Wind: 16.13 GW Hydro: 3.42 GW Nuclear: 2.11 GW Gas: 34.26 GW Coal: 26.06 GW Oil: 1.69 GW Other: 0.29 GW

Something interesting to note about "Other" is how the description of which reads: "Solar, Biomass, etc." On one had, it might be reasonable to give power from this generation source the same carbon intensity as biomass but on the other hand, it is absolutely dwarfed by everything else.

[systemcatch]

Nice @Hyric89! Do you want to do a pull request with those numbers? I think we would leave the other category out to avoid confusion.

[Hyric89]

Sure. Will do.

[systemcatch]

The problem with the MISO exchange is complicated. MISO's scheduled exchanges here often show flows >1GW but the flows in the data source from SPP rarely get near that, see SPPexmix.pkl.zip for example.

Tie abbreviation's are as follows.

Currently I think 'ALTW', 'AMRN', 'DPC', 'GRE', 'MEC', 'NSP', 'OTP' are part of MISO. Full list of members here

[systemcatch]

Further watching of the SPP and MISO data hasn't resolved the problem. Sometimes the SWPP actual and MISO scheduled are fairly near, other times they are 100's of MW apart. @corradio can you make a decision here? Either we wait for a better data source or use the SWPP one which is probably not that reliable.

[corradio]

Hi @systemcatch, Sorry for the delay. I'm unfortunately not in a position where I have enough information in order to take such a decision. If the MISO and SWPP differ, there's a couple of possible reasons:

1. We are not interpreting the SWPP data correctly
2. The SWPP data is not reliable
3. The MISO data displayed on their website is incorrect

This website seems to be quite "trustable" in order to break the tie: https://www.eia.gov/realtime_grid/#/data/graphs?end=20160724T00&start=20160717T00&bas=0000000000000004&regions=0

They have an API for SWPP / MISO (and all exchanges): https://www.eia.gov/opendata/qb.php?sdid=EBA.SWPP-MISO.ID.H

From the values it seems like indeed it spikes over 1GW quite often:

I think we should double check if the EIA actually measured data differs significantly from the https://marketplace.spp.org/pages/interchange-trend values. The EIA should serve as reference don't you think?

[systemcatch]

That's a good idea to use the EIA as a reference point. I can write a script to compare those csv's against the EIA data set which goes back years.

[brunolajoie]

Hey guys, do you know what is "coal market" vs "coal self"? We've got to be careful if its not SPP's own production facility, and if they just purchase it from a region physically not inside their control area. We only want to add it to the production mix of the region if if has been produced within the geographical boundary of our zone as displayed in the electricity map. (same problem than with IDAHO https://github.com/tmrowco/electricitymap/pull/1280#issuecomment-377777320

[brunolajoie]

I think that we can add both coal (market, and self) in our parser. Coal (self) seems to be self-owned SPP power plant. Coal (market) is produced by other power producers, all under the SPP control area

[systemcatch]

Luckily they are already added together!

[maxbellec]

IEA latest MISO <-> SPP exchange data is from two days ago and SPP gives data for the last few hours. I've saved that output, in 48 hours we'll be able to check if SPP and IEA match for this exchange. SPP-MISO_exchange-2018-04-04.txt

[maxbellec]

Latest EIA data can be retrieved here: http://api.eia.gov/series/?api_key=b93a583eeed9d0ed324f3b21923af444&series_id=EBA.SWPP-MISO.ID.H

[corradio]

@maxbellec any progress here?

[maxbellec]

SPP and EIA values seem quite different

[alixunderplatz]

@maxbellec stretching orange a little bit more and putting it a bit to the left seems like the behaviour of them is matching. Of course, Magnitude is still different, but the pattern is the same. Is there an offset in time?

[maxbellec]

yes that's weird, but I believe it's just by chance. It's not that similar when you looked into details. To aligne them more, you'd need to change both horizontal and vertical scales, and move the graph. I don't see how that can correspond to a bug in the data (processing).

[corradio]

Shall we rename this issue to MISO <-> SPP exchange?

[brunolajoie]

Any update on the possibility to add "MISO->SPP exchange"?

[corradio]

Closing due to inactivity. Thank you all for your inputs!