Split US

[corradio]

• [x] Obtain generation mix data for each ISO (https://pyiso.readthedocs.io/en/latest/options.html)
• [x] Obtain shapefile/GeoJSON for each ISO (DATA MISSING)
• [x] Import/exports for each ISO (DATA MISSING)

[martindaniel4]

👌

[sYnfo]

Is this something that someone new to the project could help with? If so, could you please specify in more detail what would be needed to get US on the map?

[corradio]

With pleasure! First thing to do is to use the library mentioned before to pull production mix informations from the US by writing a parser. Let's discuss in more details on http://slack.tmrow.co

Olivier

On 31 Dec 2016, 13:02 +0100, Matej Stuchlik notifications@github.com, wrote:

Is this something that someone new to the project could help with? If so, could you please specify in more detail what would be needed to get US on the map?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub (https://github.com/corradio/electricitymap/issues/143#issuecomment-269862037), or mute the thread (https://github.com/notifications/unsubscribe-auth/ABlEKFuZ4TZ3zGd0NaUH900wKcAlO7SLks5rNkQ7gaJpZM4K7H6i).

[corradio]

@sYnfo are you still up for helping?

[corradio]

I started working on this at https://github.com/corradio/electricitymap/tree/data/us

[garciaev]

Hi -

I am interested in getting the USA on the map. Where do I start?

[corradio]

Hi @garciaev ,

Thanks! First, we need to get data for the boundaries of each ISO, for which we already have production data. Can you help? I didn't succeed in finding geoJSON or shapefiles..

[garciaev]

OK just to be sure, we basically need geoJSON or shape files that have boundaries for the American ISOs here: http://www.isorto.org/about/default

Correct? I am currently searching for the California ISO boundary map.

[corradio]

That's right!

On Wed, Feb 15, 2017 at 6:52 AM, Victor Garcia notifications@github.com wrote:

OK just to be sure, we basically need geoJSON or shape files that have boundaries for the American ISOs here: http://www.isorto.org/about/default

Correct? I am currently searching for the California ISO boundary map.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/corradio/electricitymap/issues/143#issuecomment-279923484, or mute the thread https://github.com/notifications/unsubscribe-auth/ABlEKBNJZiFuFBnPED6lhSYwi8q3cjFeks5rcpKOgaJpZM4K7H6i .

[jacobbaratta]

http://www.eia.gov/beta/realtime_grid/ has two vector graphics that may be convertible with some work. I ripped the simpler vector graphic into a standalone svg, but I have zero experience with GIS and my short foray into investigating the feasibility of converting the SVG indicated that while it's probably possible, it's too complicated for a complete novice like me to attempt. I've attached the SVG to this post. Even if it can't be converted it can still probably help in some way with construction of a North American map.

eia-mod.svg.zip

[garciaev]

Hmm... not sure those EIA regions are the same as ISO member boundaries here? http://www.isorto.org/about/default

I think I am just gonna call the customer service and see if they can forward me to someone who can provide us with the boundary files.

[jacobbaratta]

Hmm, apparently part of the issue is that the EIA calculates their regions slightly differently from the ISOs, leading to excessive fuzziness. http://www.eia.gov/beta/realtime_grid/#/status makes this clearer (in the site but not so much the graphic). I wonder if we could just pull the data from the EIA API and use their borders? It would probably provide superior coverage of the US, although it likely lacks some of the granularity of ISO data (you might be able to get the import/export data though?). Unfortunately the EIA's data for MISO mirrors MISO's data, so that does need to be compensated for (since MISO crosses the border into Canada to cover Manitoba)...

I fooled around with qGIS and got the first SVG file imported and aligned, but I realized that the resulting raster file isn't what's wanted...and it still lacks Canada entirely and still has the unavoidable misalignment problem... Even if I get the raster converted into a shapefile it'll still have the CA/misalignment issue, so I dropped that again for now.

IMO it might actually be easier at this point to just make a FOIA request for the data the EIA uses (since it doesn't seem to be available on their site). But that still leaves excessive fuzziness with the borders, and the issue of handling MISO in Manitoba, plus AESO and IESO. Theoretically those three independent regions are literally each divided solely along provincial/national borders, so you could probably just get border data elsewhere and call it a day, but you still have to merge the MISO Manitoba and MISO US regions.

[jacobbaratta]

I have crude (some annoying alignment issues with borders that can be obscured via thick borders) shapefiles (as individual outlines of regions) for MISO (this one is technically two shapefiles and I followed the coverage indicated on the MISO site for the Manitoba MISO shapefile, which is apparently only a small part of Manitoba rather than the entire province (not sure why but I'm not arguing with MISO's official map)), SPP, PJM, NYISO, and ISO-NE. I also have ERCOT and CAISO files, although the ERCOT file might not be up to snuff (alignment issues - but I can replace it with my alternate source that aligns better). I also found a shapefile with all North American electrical border crossings, not sure if that's useful but I left them in the preview below. AESO and IESO regions do not have shapefiles yet and I don't feel like creating them at this time (doing it manually is a pain, haven't found a viable way to do it automatically yet). There were no other major North American ISOs I could cover, so there are large coverage gaps.

Here's what it currently looks like zoomed out:

Should I even bother continuing to clean this up and finish it or is the misalignment too bad to use? Ignore any misalignment alongside the ERCOT border and keep in mind that I set border line widths this thick to obscure misalignment on other borders (depending on zoom levels line widths might be reducible a bit further but not by much). All the data is georeferenced vectors saved as shapefiles.

[corradio]

Hi @jacobbaratta,

Sorry for my late reply, and thanks so much for the work here. It is very appreciated. I've been fooling around with those things myself, and I'd say that any solution that approximates reality well enough is good to start with. Basically, here's the ISOs for which we have production data for, and for which we need the corresponding boundaries: https://pyiso.readthedocs.io/en/latest/options.html The script that generates the boundaries for Europe (and the US) can be accessed here: https://github.com/corradio/electricitymap/blob/master/web/topogen.sh It uses shapefiles, so if we can generate shapefiles for the different ISOs required, we should be able to merge them with the existing US border given in the script (and retrieved from www.naturalearthdata.com).

Does that make sense? I believe a simple solution to start with is always good enough. We can always improve later. The important part is to get started!

[jacobbaratta]

Okay so I need to redo the ERCOT border first and foremost...

I looked into the feasibility of adding BPA and/or NVEnergy since they are on the pyiso list. Unfortunately there is no good map for BPA (maps exist but they're too fragmented to use), so that will need to be shelved indefinitely.

NVEnergy appears to be generalizable to the whole state of Nevada, so I could likely just manually trace the outline of that state and call it a day - in fact that is my current plan next to fixing ERCOT.

It looks like pyiso won't give us the data for SPP, AESO, and IESO, which is quite a pity given that SPP is a huge market, AESO is relatively large, and IESO is both huge and rather notable for having completely phased out coal. Is it feasible to obtain the data for those sources via another method? I know IESO makes their data relatively easily accessible and I would personally really like to see them included. SPP/AESO are less easily accessible but it looks like the data could probably be obtained with some reasonable effort. Ideally I want to do the AESO/IESO (primarily IESO) borders as well, but I'll likely shelve that until it's possible to confirm that the borders would actually be used (so that there is data available for use with the shapefiles).

I have evaluated numerous other ISOs/RTOs in the US/CA and unfortunately they're pretty much all not going to be coverable. This excludes a substantial amount of power demand, generation, and interchange - from very large sources to relatively small ones. BPA, TVA, Duke, SOCO, FLA, PSCO, WACM, PACE, BANC, LDWP, PGE, PACW, PSEI, AVA, AECI, LGEE, SCEG, SC, I haven't looked into Canada in much more detail yet but I know Hydro Quebec is apparently not sharing their data, so that's a major blow.

I have a few possible ideas for ways to format the shapefiles that would reduce the visibility of the misalignment. I have already attempted to determine if the misalignment was due to mapping/resizing/projection mistakes (which took a while to run tests for), and so far it appears that none of these are at fault. I think the original data is flawed and it's just much more visible after scaling inwards. But anyways, worst-case we can just use the thick borders like I did, they're not too ugly at high zoom levels.

I'll download ne_50m_admin_1_states_provinces_lakes to get a better idea of how this will look in production.

So, optimistically, I should be able to deliver the following (italicized = highest priority):

• CAISO (√)
• ERCOT (revision needed to area/boundaries)
• SPP (data source fix needed)
• MISO (√)
• PJM (√)
• NYISO (√)
• ISO-NE (√)
• NVEnergy (area definition needed - this is the entire state of NV)
• IESO (data source fix needed; area definition needed - this is the entirety of Ontario)
• AESO (data source fix needed; area definition needed - I think this is the entirety of Alberta?)
• All cross-border electric crossings (no data on ratings/throughput though - you might just discard this data anyways in favor of ISO/RTO figures and some guesswork or artistic placement of grid interties.)

This would be plenty for a good sample of major US utilities, assuming we can get alternative data sources for those three areas. Without alternative data sources I think we'll have a major issue when it comes to Canada due to lack of coverage outside of part of Manitoba. I have no clue how you'd handle between-ISO/RTO transmission, but I've got inter-border transmission marked and will include that shapefile when I'm finished, so there's something at least.

I can't give an ETA on completion of the remaining 2 primary tasks on my side (ignoring the 3 data source issues), but I'll try to get it done soon. I'm not sure how easy it'll be to integrate ne_50m_admin_1_states_provinces_lakes with 9-11 additional overlays (each as its own shapefile) for NA data if you're used to solely using ne_50m_admin_0_map_subunits for EU data but it should still be quite doable. It's not an issue in qGIS but I'm not sure how much that applies with the web app. Anyways, as soon as the two regions left are ready, I'll export each shapefile individually and send them all here.

[corradio]

Sounds good! Let's discuss on Slack the smaller details. Let's start small (maybe with only one ISO to begin with) and then we can extend slowly, instead of taking a mouthful too big :)

Olivier

[jacobbaratta]

The first attached zipfile contains shapefiles for ERCOT (based on EIA map), ISO-NE (based on EIA map), MISO (based on EIA map), NYISO (based on EIA map), PJM (based on EIA map), SWPP (based on EIA map), MISO_CA (the part of MISO that's in Manitoba - not based on EIA map (based on MISO's map)), and NVEnergy (I just manually traced the entirety of Nevada - it isn't 100% accurate but it's close enough that we can use it - not based on EIA map (not sure exactly what source I used to verify the boundaries but I'm confident that it's accurate enough)). The second one contains the CAISO shapefile and was obtained directly from a public source (it is a bit too detailed IMO due to the inclusion of other balancing regions but I didn't feel like mapping yet another region from the EIA map - I'll probably end up doing that at some point, but this is a good enough placeholder until then). Please evaluate them and see if they are of any use. My styling in QGIS will not carry over so that will need to be reimplemented to obscure the gaps between region borders. Note that all files in my zipfile use EPSG 3857 except for MISO_CA and NVEnergy, which use EPSG 4326. The CAISO shapefile (separate zipfile) uses EPSG 3310. For reference, ne_50m_admin_1_states_provinces_lakes uses EPSG 4326.

I'm still investigating the cause of the errors with my projections. It seems that there are quite a large number of contributing factors, and so far I have been unable to devise a working solution that increases accuracy to the necessary extent, although I have a few more ideas to explore before giving up. With 0.75mm solid line widths the errors in the projection mostly vanish for viewing at the scales it would likely be used at online, so that may be a solution for the interim.

I have double-checked all other major ISOs/regions - the only notable candidates with data available are BPA (unfortunately I cannot map their area accurately via any realistic means), AESO (area can be manually mapped relatively easily but data source isn't directly available via the library you use), and IESO (area can be manually mapped with some difficulty but data source isn't directly available via the library you use). Some data for other ISOs/regions could possibly be obtained via the EIA (as the organizations do not release it directly) but I'm assuming this approach is not at all viable and have scrapped all plans for mapping those regions. I'll try to look at pyiso at some point to see if adding AESO/IESO handlers is actually realistic, I believe it is, but I have not tested anything yet.

Anyways, here are the files. Hopefully they're usable.

all_self_SHP-v1.zip Balancing_Authority_Areas.zip

[jacobbaratta]

Conclusions regarding inaccuracy with my projections:

• Inaccuracy comes from a lot of sources
• Much of the inaccuracy is unavoidable. My base dataset is one of the EIA's SVGs, which had to be cleaned up, converted, opened in Illustrator, cleaned up again, then processed to extract every relevant (now wireframe) region of interest, each of which was multiplied by a fixed scaling factor, roughly centered, then exported as a maximum-resolution png file. This file was then imported into QGIS as a raster for manual georeferencing, then converted to a vector/shapefile. GIS is extremely complicated and not entirely intended or able to address this use case very effectively.
• Issues with the accuracy of common projection types are likely responsible for a large portion of the observed error. The Pseudo Mercator projection in particular is notorious for causing issues (although literally every projection type has some sort of issue with it, they're all tradeoffs, and the pseudo mercator is the defacto standard). At least some of the observed errors are attributable directly to the distortions caused by this projection technique. Unfortunately I have no viable working alternative - I have successfully produced rasters using other coordinate systems/projection types, but I would have to remap literally everything to get those to work somewhat well, and distortion becomes an unexpectedly major issue when doing this (for different reasons). For all intents and purposes, it seems that the Pseudo Mercator is the best coordinate system/projection technique for my uses, flawed though it may be.
• Very slight imprecisions in image scaling in Illustrator could have affected the result significantly.
• The settings used while creating georeferenced rasters may have a significant influence on the accuracy of the result, but it is difficult and very time-consuming to assess the impact of different options here. So far I have had little success with modifying these settings (which I have tried pretty extensively).
• Issues with the mismatch/difference in quality/simplification between the EIA files used and my reference maps used to georeference rasters, compounded by projection distortions, make aligning the rasters (and resulting vectors) very difficult.
• Adjustment of raster/vector size/borders is a no-go so far. QGIS does not seem to handle this very well when automated (freezing for excessive periods of time when running affine transformations for example), and manually editing the sheer number of points per ISO/RTO outline is impractical (and also an issue due to the potential to introduce further inaccuracy in the process). I was hoping affine transformations would be a possible way to get the inaccuracy reduced a bit, but they operate far too slowly to utilize effectively (single-threaded only, extremely slow execution despite my fast single-core performance). It may be that the reason for the slowness of affine transformations is related to my specific platform, but I doubt this and suspect it's primarily related to the number of points and the algorithm used. Attempts at simplifying the number of points haven't gone well. You would think that an affine transformation would be absolutely perfect for GPU acceleration (via CUDA perhaps?), but QGIS does not seem to support this. At any rate, affine transformations are not the holy grail, they would be a partial fix at best, so I suppose it doesn't matter too much.
• Manual realignment of the entire raster/vector is a no-go due to distortion issues. Affine transformations or painstaking point-by-point adjustments are the only viable ways to improve the amount of error observed, and as explained above, neither are particularly viable options at the moment.

So, here's the overall conclusion:

• For North America, a different approach will need to be taken than was taken for the EU. Unlike the EU, you can't just neatly slice things up along country lines - you need overlays in virtually every case, and those are not going to align very prettily.
• As the inaccuracy currently has no viable solution, it will have to be dealt with. This is doable.
• The complexity of the script needed to generate the (GeoJSON?) files is unavoidably going to increase for the US map, and will likely require stylistic tweaks (not sure where these would be applied) to obscure the inaccuracy with the vectors.
• If vector inaccuracy is obscured to a significant enough degree (depends on zoom level), this vector data is quite definitely an acceptable approximation of a number of the most major US/CA ISO/RTO boundaries that can be used for a US/CA version of electricitymap.

Tl;Dr: The inaccuracy in the data is extremely complex in origin and very much unavoidable right now, but it can still definitely be obscured, and if it is obscured, this data should easily be sufficient for an initial US/CA version of electricitymap. I am out of ideas for resolving the inaccuracy problems so I'm not planning on spending much more time on that in the immediate future - what files I uploaded as the best we've got unless someone's been hiding a bunch of really high quality shapefiles somewhere hidden from Google.

[corradio]

Hi @jacobbaratta,

Thanks for the tremendous effort. I'm currently working on making a mobile version of the map, but once I get a bit more time I'll try to see if I can integrate your shapefiles. If anyone wants to give a go at integrating the shapefiles into the map, please reach out on Slack and I can help get you started.

Hopefully this will work!

[jarek]

Research notes for Hawaii, in case they help someone in the future:

Currently we draw all of U.S. in one colour. However the source for this, WattTime's pyiso, actually only reports generation within contiguous U.S. and doesn't have anything for Alaska and Hawaii. (Alaska seems to have at least two grids not connected to each other or anything else, and surely quite a lot of off-grid as well)

Hawaii has six main inhabited islands, in order of population: Oahu, Hawaii (Big Island), Maui, Kauai, Molokai, and Lanai.

Kauai is served by http://website.kiuc.coop/, it appears they don't have real-time data.

The other islands are served by HECO, confusingly doing business under three separate names: Hawaiian Electric Company for Oahu, Hawaii Electric Light Company for Big Island, and Maui Electric for the other three islands.

They have a dashboard for renewable energy that features daily figures, but: 1) they seem to be updated irregularly during the day and not at all at night, and 2) they are only as images, no text data.

The dashboards are at:

• Oahu https://www.hawaiianelectric.com/clean-energy-hawaii/integration-tools-and-resources/renewable-watch
• Big Island https://www.hawaiielectriclight.com/clean-energy-hawaii/integration-tools-and-resources/renewable-watch
• Maui / Lanai / Molokai https://www.mauielectric.com/clean-energy-hawaii/integration-tools-and-resources/renewable-watch

I have not seen other real-time data from either utility or government/regulatory sources. http://puc.hawaii.gov/energy/ appears to be the regulator. At least Maui definitely doesn't have an ISO/RTO per http://www.utilitydive.com/news/isorto-model-best-alternative-to-hawaiian-electric-service-maui-report-co/412456/

I wrote a message to HECO via the form at https://www.hawaiianelectric.com/customer-service/contact-us asking if they have the data in text form and updated at least roughly every 2 hours, but unfortunately they replied "Apologies but we currently don’t make the data available."

[corradio]

Thanks for looking into this. That's very helpful

On Aug 9, 2017 20:48, "jarek" notifications@github.com wrote:

Research notes for Hawaii, in case they help someone in the future:

Currently we draw all of U.S. in one colour. However the source for this, WattTime's pyiso, actually only reports generation within contiguous U.S. and doesn't have anything for Alaska and Hawaii. (Alaska seems to have at least two grids not connected to each other or anything else, and surely quite a lot of off-grid as well)

Hawaii has six main inhabited islands, in order of population: Oahu, Hawaii (Big Island), Maui, Kauai, Molokai, and Lanai.

Kauai is served by http://website.kiuc.coop/, it appears they don't have real-time data.

The other islands are served by HECO, confusingly doing business under three separate names: Hawaiian Electric Company for Oahu, Hawaii Electric Light Company for Big Island, and Maui Electric for the other three islands.

They have a dashboard for renewable energy that features daily figures, but: 1) they seem to be updated irregularly during the day and not at all at night, and 2) they are only as images, no text data.

The dashboards are at:

• Oahu https://www.hawaiianelectric.com/clean-energy-hawaii/ integration-tools-and-resources/renewable-watch https://www.hawaiianelectric.com/clean-energy-hawaii/integration-tools-and-resources/renewable-watch
• Big Island https://www.hawaiielectriclight.com/clean- energy-hawaii/integration-tools-and-resources/renewable-watch https://www.hawaiielectriclight.com/clean-energy-hawaii/integration-tools-and-resources/renewable-watch
• Maui / Lanai / Molokai https://www.mauielectric.com/ clean-energy-hawaii/integration-tools-and-resources/renewable-watch https://www.mauielectric.com/clean-energy-hawaii/integration-tools-and-resources/renewable-watch

I have not seen other real-time data from either utility or government/regulatory sources. http://puc.hawaii.gov/energy/ appears to be the regulator. At least Maui definitely doesn't have an ISO/RTO per http://www.utilitydive.com/news/isorto-model-best-alternative-to-hawaiian- electric-service-maui-report-co/412456/

I wrote a message to HECO via the form at https://www.hawaiianelectric. com/customer-service/contact-us, but unfortunately they replied "Apologies but we currently don’t make the data available."

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/tmrowco/electricitymap/issues/143#issuecomment-321346838, or mute the thread https://github.com/notifications/unsubscribe-auth/ABlEKOAWLHU5QEgZ4duDTf7ftGo1zS5Lks5sWf7mgaJpZM4K7H6i .

[systemcatch]

I've been playing around with getting shape files/geojson using this link. However looking at the data closer it seems that I'm just repeating work done by others already.

Searching stackexchange I found this question about converting svg to geojson which might be interesting. In particular it mentions d3 and using svg files directly.

Finally this link may be useful. Personally I think our best option is to use @jacobbaratta shape files. However maybe we could try creating the geojson polygons for each ISO manually then insert them into the current USA shape? We don't need to be super accurate imo.

[corradio]

I think we should start small by starting to add just CAISO as soon as we have the data (see #787). The shape should be the one of California, so that should be simple. Hopefully this will start a movement where we can get some attention and get the proper polygons. I think it's important that people can relate to the regions we display. That's why I'm a bit reluctant on splitting by ISO.

[HansHyde]

Hi Corradio, et.al., love your work so far! Great job!

Sorry I can't add any programming assistance, but maybe I can add some practical information/experience that will help in this project's development.

Points re: issues above (in no particular order)

ERCOT/SPP/Mexico interconnections... 1) there is one 800 MW interconnection between SPP/ERCOT 1a) there is an overlap of ERCOT/SPP ranging from 100-150 wide over a length of about 300 miles. Trying to define a specific line to clear up "fuzziness" is futile. It's also worth noting, much of north & west Texas wind was originally connected to SPP (i.e. NoTrees Wind Farm) substations until CREZ transmission was completed. If they were then connected to ERCOT system (such as at the Ector County North Switch built as part of CREZ), then effectively (theoretically) there are many more DC interconnections than reported. Each wind farms DC inverter becomes an interconnect at the switch between two regional transmission lines; 1 (old) to SPP & 1 (new) to ERCOT. In theory at least, they can sell into whichever market is offering better prices, given they did not decommission the old lines to SPP. 2) There are 4 DC interconnects to the Mexican grid.... all mostly small for local balancing. 3) There is an AC connection between ERCOT & the eastern grid (Entergy) outside Houston. It was used once after a hurricane some years ago. It can be ignored for all practical purposes. 4) flow rates & directions are available on ERCOT's site 5) no ERCOT interconnections to the western grid.

BPA - Bonneville Power Authority (& TVA - Tennessee Valley Authority) are both federal power generation & transmission entities. BPA does not have a mappable "territory" per se, but it does supply as far south as CAISO via WAPA lines. TVA would be similar, as they generate & supply via transmission to countless distribution Munis & CoOps.

Drilling down to the level of NV Energy may be futile, it's part of the Western Grid - where the US has several thousand "utilities". Covering the major ISO/RTOs (CAISO, PJM, MISO, NEISO, NYISO, ERCOT, etc) might be the best one can do. For example, Germany has not been divided into its 4 transmission regions.

I see you all have found WattTime. EIA also has a project under works that may be of assistance... https://www.eia.gov/beta/realtime_grid/?src=home-f1#/status?end=20171115T06

Synapse Energy may also be a worthwhile resource as they have US powerplant information (which can be used on Google Earth).... http://www.synapse-energy.com

Anyways, hope this helps.

I've built on Google Earth the majority of Southern Africa's substations through transmission to powerplant maps, including both AC & DC if that is of assistance to this overall project.

Cheers, Hans

[HansHyde]

Oh ps, here's a bit more info on why trying to get granular with the US might be impossible. Keeping it simple with the ISO/RTOs might be the best bet, everything else an entry into a maze of rabbit holes. And reading comments above, seems like you guys have your work cut out for you just getting ISO/RTO information in realtime. https://www.linkedin.com/pulse/why-us-energy-space-struggles-instead-surging-hans-hyde/

[systemcatch]

@HansHyde thanks for the extra info, I agree with you about trying to keep things simple with the complex US electrical grid.

I've opened a new issue for South Africa. #856

[corradio]

Hi all,

I suggest we start splitting up little by little. Seems like CAISO and NYISO are good places to start. I believe they can be tied to the states of California and New York?

[systemcatch]

@corradio I believe they roughly correspond to the states. How much work is it to add support for other geometries in #840 though? Quite a few parsers are blocked until we can use shapes other than natural earth. Adding the US is a big step because of their impact but I think it's key to improving the map by adding more users. If we can get the shapes/overlays working then we could add a big chunk of the US in one go using pyiso.

[corradio]

Tough to me to say. I have a hard time dedicating coding work right now, as we're promoting the project everywhere. I'm trying to find additional resources and this is definitely a priority.

On Thu, Dec 7, 2017 at 1:52 PM, systemcatch notifications@github.com wrote:

@corradio https://github.com/corradio I believe they roughly correspond to the states. How much work is it to add support for other geometries in

840 https://github.com/tmrowco/electricitymap/issues/840 though? Quite

a few parsers are blocked until we can use shapes other than natural earth. Adding the US is a big step because of their impact but I think it's key to improving the map by adding more users. If we can get the shapes/overlays working then we could add a big chunk of the US in one go using pyiso.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/tmrowco/electricitymap/issues/143#issuecomment-349958416, or mute the thread https://github.com/notifications/unsubscribe-auth/ABlEKCn7tBsRAnTe8UAplyNs_kQMwQYkks5s99-agaJpZM4K7H6i .

[systemcatch]

I understand. I'll take a closer look and try to figure out how to make this work.

[HansHyde]

Hey guys, check this out for the 100s,1000s of electric retailers in the US

https://hifld-geoplatform.opendata.arcgis.com/datasets/c4fd0b01c2544a2f83440dab292f0980_0?geometry=-151.196%2C21.72%2C-44.673%2C46.756

here is for the major ISO/RTOs in the US... https://hifld-geoplatform.opendata.arcgis.com/datasets/9d1099b016e5482c900d657f06f3ac80_0

If I'm not mistaken, they are the shapefiles you are looking for...

[HansHyde]

Here is NYISO realtime (updates every 5 min) system demand with flows in/out from IESO (Ontario), PJM, HydroQuebec and NEISO

http://www.nyiso.com/public/markets_operations/market_data/maps/index.jsp

Here gives you generation within NYISO (which does not include the imports), so potentially some assumptions can be made from the data in the Daily Fuel & Current Fuel tabs.... http://www.nyiso.com/public/markets_operations/market_data/graphs/index.jsp

Sorry, I can't search around ERCOT.com anymore... they appear to have a security measure that is blocking access from outside the US.

[HansHyde]

@systemcatch & @corradio

Hey guys, found this great 1-line diagram for the entirety of the Eastern Interconnect which includes all of eastern Canada, all of the eastern USA (from the eastern border of Montana to the panhandle of Oklahoma), and the "external" connections to ERCOT & the Western USA.

http://images.slideplayer.com/39/10987225/slides/slide_4.jpg

What you see are all the major balancing agencies (there are 26) with all their simplified interconnections. I know @systemcatch put effort into getting state lines onto the map (with what I am assuming will contain the parsers), but I believe this is going in a direction that will lead to many dead ends & wasted efforts.

To reach the goal of getting the US/Canada completed, I would highly suggest we establish the parsers based on the above 26 "balancing agencies" and move away from the current "state lines" approach. Each line on the diagram equates to an interconnection, simplified (there may be more than one physical interconnection for some of them), so this would be the basis for flows between to calculate each balancing agency's current generation & CO2eq.

TBH state lines only work for NY (marginally) and the Canadian provinces. From a practical purpose, "state lines" only apply to state level Public Utility Commissions, which have no day-to-day involvement in generation or electric delivery - they are regulatory bodies - and we won't find any realtime data from them.

From 26 parsers, we would then be covering 36 States plus at least 5 Provinces.

Can I suggest we open up a new thread for "Eastern Interconnection"?

[HansHyde]

Hi guys, please check this out.... I sorted out all the BAs and determined all the Interconnects (about 70) - the BAs will be our best chance for finding accessible datastreams. If I'm understanding correctly how EM works, the blue table would be the parsers and the orange table are known (so far) interactions between them (flows).

https://www.dropbox.com/s/xizh6khtxcfqdqo/Eastern%20Interconnection%20-%20BAs%20%26%20Interconnections.xlsx?dl=0

[systemcatch]

There is real time data for the NEISO area at https://www.iso-ne.com/isoexpress/.

It looks really good!!!

Seems to map directly onto the states of Maine, New Hampshire, Vermont, Massachusetts, Connecticut and Rhode Island.

[HansHyde]

@corradio - can you close this thread, and open one for NE-ISO (@systemcatch 's info could go there) & NYISO?

The information above is lengthy, and overall generally no longer relevant.

[corradio]

By the way @jacobbaratta your work is not in vain. Once we finish #913 I believe we will be able to incorporate the geometries you attached to this thread. @HansHyde / @systemcatch feel free to open an issue about NE-ISO (but keep in mind we can't implement anything if state geometries do not match NE-ISO)

[systemcatch]

@corradio Fortunately the state lines match exactly for NE-ISO in #924. I will probably open a separate issue about updating all of the North American exchanges though.

[alixunderplatz]

Not sure if this will lead anywhere, but here is a map from the NERC website with "NERC regions", which stands for North American Electric Reliability Corporation.

Maybe it is worth to check all of these for data, too. These areas interefere and overlap with some ISOs.

You can also display these areas on the EIA map website https://www.eia.gov/state/maps.php?src=home-undefined

[systemcatch]

As a further idea we could use congressional boundaries to give us more flexibility with drawing the US ISO areas. See https://stackoverflow.com/questions/14565963/topojson-for-congressional-districts for more details.

[jacobbaratta]

@alixunderplatz I have a bunch of resources like that NERC map that I can share if people are interested, I collected quite a lot of different resources to use as references while working on my original shapefiles.

@HansHyde I saw those linked on eia's site a while back and went through them. Some of them are quite nifty, and it's the first time some of that data has been officially released in shapefile format, but there's nothing new in there that is of actual value to us for this particular project. Their ISO-level shapefiles are no more accurate than the ones I created/collected (if anything they are considerably cruder and less accurate), which is a major disappointment. Their only benefits are that they have nice metadata (not really relevant for our use case) and that they may seem somewhat more "official" (which is a subjective argument, not an objective one).

@corradio, has anyone made any progress with merging third-party shapefiles/geometries into the map? I've given it a try once or twice a while back and made no significant headway before giving up -- I'm just not familiar enough with the methods and libraries you're using, and there's no obvious way to handle merging additional shapefiles/geometries (plus it's not just getting it to display, but also getting it to work as usable regions, since much of the app appeared to rely on hooking directly into attributes in the original geometry to tell regions apart and know what data goes to what area, etc).

[systemcatch]

@jacobbaratta I was able to split the Canary Islands using new shape files in #1022. @corradio & @maxbellec then really improved how the map generates topologies in #1064.

You might as well add some of your resources to this issue thread so people don't end up repeating work.

[systemcatch]

Some really good news everyone!

I found a shapefile showing Balancing Authority areas for the whole of the USA, https://hifld-geoplatform.opendata.arcgis.com/datasets/0242ddfc59554fadb8ffe8066c573c93_0

The attribute data for the shapes is very detailed. After a lot of work with qgis, topogen.sh and generate-geometries.js I've been able to make this.

[maxbellec]

Not the most beautiful map, but worth gold 😍 Send the PR!

[alixunderplatz]

@systemcatch Very nice! Seems like all of the areas more or less overlap with (many unused) others, which causes that "disintegration", esp. for CAISO and SPP?

Have you looked into the other shapefiles on that site already?

I really like the one for the large ISO areas. Maybe that could reduce the fragmentation (and required effort) to some extent? https://hifld-geoplatform.opendata.arcgis.com/datasets/9d1099b016e5482c900d657f06f3ac80_0

[systemcatch]

@alixunderplatz I had a look at some of the other shapefiles but they didn't seem to represent the data parsers are currently getting very well. The large ISO areas are good but big parts of the USA are missing and wouldn't show up on the map. We'd have to layer the ISO's onto a base shape of the US (:thinking: possible?) and find shapes for BPA and IPC.

As regards the image I posted there's problems with overlapping and highlighting everywhere. A big question is whether some of the balancing authorities contained within other BA's are included in the larger BA's data from the parsers. If this is the case then the smaller areas can be dropped.

[jarek]

I think we can be selective, so including for instance NY / New England from Natural Earth and SPP from a shapefile. Just as long as they line up decently. In the screenshot above a part of Maine on the border with New Brunswick got flooded ;) :ocean:

[brunolajoie]

I've found a crazy map of the US grid. http://www.eei.org/about/members/uselectriccompanies/Documents/EEIMemCoTerrMap.pdf

Does anyone understand the difference between all these "EEI companies", Utilities, ISO grids, IPCC? I'm starting to get lost with the US power system ;)

[systemcatch]

My understanding of that map is that those are all for profit companies that serve consumers. I recognize some of the companies in California that are part of CAISO, doesn't look to be any electric coops on that map.

[corradio]

@systemcatch should we close this issue?