Switch Ontario to using IESO XML data

[jarek]

Use http://reports.ieso.ca/public/IntertieScheduleFlow/PUB_IntertieScheduleFlow_20180108.xml (note that URL changes, using Toronto timezone = UTC-5 / -4) rather than http://live.gridwatch.ca/WebServices/GridWatchWebApp.asmx/GetHomeViewData_v2

Advantages:

• breakdown for every 5 minutes (in browser view, hover/tap on the "flow" values for each hour to see breakdown)
• the "flow" values appear to take into account Lake Erie Loop between Michigan and New York - basically "flow" is the actual electrons on the wire, "exp" is the accounting value

First pointed out by @HansHyde in https://github.com/tmrowco/electricitymap/pull/925#issuecomment-353621704

edit: also use IESO data for generation mix, as it is more official. Live data is at http://www.ieso.ca/en/power-data?chart=supply and historical as XMLO at http://reports.ieso.ca/public/GenOutputCapability/

[HansHyde]

@jarek

The NY - Ontario exchange is also available in 5min increments here... http://mis.nyiso.com/public/csv/ExternalLimitsFlows/currentExternalLimitsFlows.csv

represented by this line here... "01/08/2018 08:10,SCH - OH - NY,23317,904.9,1750,-1900"

The "OH" must represent "Ontario Hydro" its name prior to deregulation and the IESO being formed, as opposed to "OH" equal Ohio

[corradio]

Any status here?

[jarek]

I was looking at this, but haven't had time yet. It should still be done, if only to get the MISO/NYISO Lake Erie Loop-affected exchanges correct.

I think IESO also have historical generation mix available through XML files that we can switch to, as well. Their generation data is at http://www.ieso.ca/en/power-data?chart=supply and historical at http://reports.ieso.ca/public/GenOutputCapability/

[jarek]

For whatever reason, the Quebec-Ontario interchange is wrongly showing as 0 in Electricity Map. http://live.gridwatch.ca reports Ontario importing total ~1.1 GW from Quebec at 14:00-15:00 2018-07-05, and http://reports.ieso.ca/public/IntertieScheduleFlow/PUB_IntertieScheduleFlow_20180705.xml says ~1.4 GW, but we have 0. For that matter it's been 0 whenever I looked at EM in the past few weeks.

[jarek]

In looking at this, I found the cause of the Quebec bug: https://github.com/tmrowco/electricitymap-contrib/blob/6b3f2dd8669db10a05b3a92bc0441ef778014d2f/parsers/CA_ON.py#L179 makes keys an iterator, but we then attempt to consume it twice: to get the datetime and then the netFlow. The problem is by the time we get to the netFlow, the iterator is completely consumed and will be empty. map will happily run over a consumed iterator, simply returning an empty sequence.

So it looks like import from Quebec would have been 0 from day 1 (since https://github.com/tmrowco/electricitymap-contrib/commit/202383b244d2b6047148c3289f01dac6f419c971)

I will try to reimplement using IESO data. It looks like 3 months' history is reachable (currently http://reports.ieso.ca/public/IntertieScheduleFlow/PUB_IntertieScheduleFlow_20180414.xml loads, 20180413 doesn't), perhaps it can be recrawled.

[corradio]

@jarek can we close this now?

[jarek]

I am working on changing the production parser so that it can handle target_datetime (for up to 3 months in the past). We can close this ticket if you like, I have the URL saved now :)

[jarek]

If anyone else is wondering: this "choppy" import chart today is because 0 MW values on the full hour were inserted by the old code. After deploy of #1515, the corrected values since midnight Toronto time were fetched with every-5-minute timestamps, but the older 0 values were not overwritten, resulting in 0 import from QC on full hour, and correct values in between.

The same is also the case with exports to MISO and NY (the full-hour values are the accounting values), though not visible on a chart, only by scrolling through history.

This will not be an issue going forward as only the new code will be used.

If you wish to re-crawl the past 3 months, you should first invalidate the values for CA-ON->CA-QC currently in the database for before today.