search

electricitymaps / electricitymaps-contrib

A real-time visualisation of the CO2 emissions of electricity consumption
https://app.electricitymaps.com
GNU Affero General Public License v3.0
3.49k stars 927 forks source link

US: ERCOT access #1160

Closed jkfm closed 4 years ago

jkfm commented 6 years ago

Hi,

I was interested in adding data from ERCOT. I cannot access their website though and according to their helpdesk, one needs to fill out a detailed sheet to get your IP whitelisted. Did anyone here have communication with them?

Regards, Johannes

jkfm commented 6 years ago

Ok, so it seems American IP addresses can access the domain (I am in Germany). But the server running the parsers does not have a US IP address, right?

systemcatch commented 6 years ago

Hi @jkfm I have tried going on to http://www.ercot.com/ before but as you say they implement geo blocking for all non American IP addresses. Unfortunately they also block their contact page and only provide a US phone number :roll_eyes:. I will check the site again later and see if anything else has changed.

corradio commented 6 years ago

Same here. I can ask for access for the ElectricityMap project if you can tell me the procedure!

On Wed, Feb 28, 2018 at 10:45 AM, Chris notifications@github.com wrote:

Hi @jkfm https://github.com/jkfm I have tried going on to http://www.ercot.com/ before but as you say they implement geo blocking for all non American IP addresses. Unfortunately they also block their contact page and only provide a US phone number 🙄. I will check the site again later and see if anything else has changed.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/tmrowco/electricitymap/issues/1160#issuecomment-369184432, or mute the thread https://github.com/notifications/unsubscribe-auth/ABlEKPp5j4ZnSm6tRGmWEcSzQJwCJ3A6ks5tZSBJgaJpZM4SVjFy .

jkfm commented 6 years ago

Hi @corradio,

they sent me this via email: http://output.jsbin.com/zafexecasu It'd be great if you could ask for access for the project, I guess I will try to write the parser with a proxy.

corradio commented 6 years ago

Great I've sent them an email.

On Wed, Feb 28, 2018 at 4:32 PM, jkfm notifications@github.com wrote:

Hi @corradio https://github.com/corradio,

they sent me this via email: http://output.jsbin.com/zafexecasu It'd be great if you could ask for access for the project, I guess I will try to write the parser with a proxy.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/tmrowco/electricitymap/issues/1160#issuecomment-369275011, or mute the thread https://github.com/notifications/unsubscribe-auth/ABlEKJOsaSBeu9gwmXykHg2BJiaLfWEoks5tZXB_gaJpZM4SVjFy .

corradio commented 6 years ago

I've been able to gain access for the Electricity Map project (from the EM server). Is anyone able to work on a parser? I think that person would need to be cleared for access as well.

systemcatch commented 6 years ago

What form is the data in Oli? If it's a csv file or something send it to me and I can work on it without needing access.

corradio commented 6 years ago

I don't know actually, I haven't gone to the website (I just saw that our server has access).

On Fri, May 4, 2018 at 12:57 PM Chris notifications@github.com wrote:

What form is the data in Oli?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/tmrowco/electricitymap/issues/1160#issuecomment-386567380, or mute the thread https://github.com/notifications/unsubscribe-auth/ABlEKKnoXpP-crUAOhRP_D4N4AzXO1Zuks5tvDQTgaJpZM4SVjFy .

systemcatch commented 6 years ago

I now have access to the site. I've been looking around for a breakdown by fuel type but so far there's only a split into thermal, solar & wind.

systemcatch commented 6 years ago

Quick update, though I have access to the main site I don't have access to mis.ercot.com which contains the solar and wind data. I've sent an email about this but so far there's been no response (just like my access email).

corradio commented 6 years ago

ok. They didn't answer me but gave me access anyway. Thanks for your patience. Hope they fix it soon!

On Tue, Jun 26, 2018 at 12:17 PM Chris notifications@github.com wrote:

Quick update, though I have access to the main site I don't have access to mis.ercot.com which contains the solar and wind data. I've sent an email about this but so far there's been no response (just like my access email).

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/tmrowco/electricitymap/issues/1160#issuecomment-400257083, or mute the thread https://github.com/notifications/unsubscribe-auth/ABlEKKs-2eL-02_liKEPDss1gRuNkynJks5uAgo1gaJpZM4SVjFy .

systemcatch commented 6 years ago

Still no luck getting access to the part I need. I suggest someone else build this parser so we aren't losing data, the EM server has full access right?

corradio commented 6 years ago

I managed to use the electricityMap server as a proxy.

There are DC flows here: http://www.ercot.com/content/cdr/html/real_time_system_conditions.html image

THere's a lot of data but I didn't manage to find generation mix. I could only find wind/solar production and total production.

Here's the source code of the html in case someone wants to take a look at it: Real-Time System Conditions.txt

@jkfm what kind of data did you want to add? Do you know about generation mix data?

janikgar commented 6 years ago

I've been intrigued by this issue, as my husband works in the energy industry. Can we reliably infer information from this document? This is ERCOT's forecast for fuel mix through 2027. Look at p. 20 for the winter fuel mix and p. 32 for summer. They include a system-wide fuel mix. Until we can find specific information, would it be worth it to use these numbers (along with the near real-time wind and solar numbers)?

Edit: I'm also going to check with husband to see if they have any publicly-accessible resources for fuel mix. I'd be surprised if they didn't, since his utility primarily works in MISO.

snarfed commented 6 years ago

@janikgar did you find any publicly accessible resources for fuel mix?

@corradio if we have total output and fuel mix forecast from the PDF @janikgar linked, i'm guessing you all wouldn't want to just blindly apply the forecasted ratios to the total output and claim that's realtime data, right?

i may look at writing a parser that reports the real thermal, wind, and solar, and then lumps the rest into unknown.

corradio commented 6 years ago

@snarfed we strive to never use forecasts to display data, as those might be incorrect. Reporting wind, solar and putting in rest into unknown is not a bad solution to start with.

systemcatch commented 5 years ago

We have wind, solar and total demand right? That might be enough to write a basic parser?

corradio commented 5 years ago

sounds like it indeed!

On Fri, Nov 23, 2018 at 11:39 AM Chris notifications@github.com wrote:

We have wind, solar and total demand right? That might be enough to write a basic parser?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/tmrowco/electricitymap-contrib/issues/1160#issuecomment-441207096, or mute the thread https://github.com/notifications/unsubscribe-auth/ABlEKHR_H3EzqwsYvHZc4pFml2MLA_WCks5ux9BtgaJpZM4SVjFy .

brunolajoie commented 5 years ago

Yes, that's the plan! 💪

brunolajoie commented 5 years ago

ERCOT is split in 4 markets (also called Geographical zones / "Forecast Zones" on ercot.com) source image

I've checked ERCOT website, and the same data is available at the "market" level than at the "state level":

What we might miss, we need to check in more detail, is the real-time exchange flow between each 4 ERCOT markets

typicalTYLER commented 5 years ago

This isn't an especially pressing matter, but I think ERCOT now technically covers most of the panhandle of Texas as well: image This image taken from the ERCOT website for prices. I read that Lubbock is also close to being in ERCOT, so I imagine a lot of the panhandle counties are joining in the near future.

image Another more up to date and detailed county map of what is and isn't ERCOT. (from the ercot website) I think it's probably safe to keep Texas and ERCOT as the same geographical boundary for now. Especially as the majority of populated areas in Texas are probably within ERCOT.

typicalTYLER commented 5 years ago

Another point, I was hoping to use ERCOT's forecast and historical data to build a tool to help me schedule when to charge my electric car based off carbon intensity. And I thought maybe adding those to the US-TX parser would be a good way to do so, but it seems like all access to that data is behind paid APIs only, even if I were to be the one to implement them. Am I understanding correctly?

corradio commented 4 years ago

Closing due to inactivity. Thank you all for your inputs! If new information pops up, please re-open this issue