California (CAISO)

[corradio]

Unfortunately we only have renewable data live. Can someone help us get access to fossil?

[corradio]

I've been able to gather the following potential sources:

• https://github.com/WattTime/pyiso/blob/master/pyiso/caiso.py
• http://content.caiso.com/green/renewrpt/20171009_DailyRenewablesWatch.txt (09 Oct), only available for the day before.

And I got a reply from CAISO:

Good morning and thank you for the inquiry. This data can be found on oasis.caiso.com where it can be downloaded in CSV or XML format under the System Demand tab. There is also an API specification that allows users to pull this data without going to the User Interface found here - http://www.caiso.com/Documents/OASIS-InterfaceSpecification_v4_3_3Clean_Independent2016Release.pdf The ISO also offers the below option in CSV as well. http://www.caiso.com/Pages/TodaysOutlookCSVFileInstructions.aspx

I'm asking for breakdown of renewables.

[systemcatch]

They responded to your email which is a good sign, and the API spec looks quite hefty. If we can get a renewable breakdown that includes hydro we may be in business.

[corradio]

New response:

Thank you for your e-mail. Yes, we have changed our Daily Renewable Watch. The information that we have provided is what is sent to that report.

!

http://content.caiso.com/green/renewrpt/20171009_DailyRenewablesWatch.txt now reports breakdown over last day. Can someone take the lead and make a parser?

[systemcatch]

When I try to set the date for today it gives a 404 error. Assume I'm missing something? http://content.caiso.com/green/renewrpt/20171101_DailyRenewablesWatch.txt

[corradio]

I'm reaching out to them.

On Wed, Nov 1, 2017 at 1:04 PM, systemcatch notifications@github.com wrote:

When I try to set the date for today it gives a 404 error. Assume I'm missing something? http://content.caiso.com/green/renewrpt/20171101_DailyRenewablesWatch.txt

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/tmrowco/electricitymap/issues/787#issuecomment-341088104, or mute the thread https://github.com/notifications/unsubscribe-auth/ABlEKF6WysA4kLpLmE4WYdPNLMQ5DORmks5syF5EgaJpZM4PtlLQ .

[systemcatch]

At a guess today's txt file is on the server and a script writes new lines to the file every hour. Or it's possible that the data is stored elsewhere and the txt file is auto generated at the end of each day. Their live renewable graph is just an image so it must be fed today's data from elsewhere then updated on the page.

[7hibault]

Hello, I'm new here, I'd like to help. I assume this is generated at the end of each day, as the latest available is yesterday's: http://content.caiso.com/green/renewrpt/20171112_DailyRenewablesWatch.txt

I could work on the parser if there's no urgency, I should be able to start working on it in a week or so

[corradio]

Yes but data is only available at midnight every day. I'm trying to get in touch with them but I'm struggling to get a response.

Olivier

On Mon, Nov 13, 2017 at 3:12 PM, 7hibault notifications@github.com wrote:

Hello, I'm new here, I'd like to help. I assume this is generated at the end of each day, as the latest available is yesterday's: http://content.caiso.com/green/renewrpt/20171112_DailyRenewablesWatch.txt

I could work on the parser if there's no urgency, I should be able to start working on it in a week or so

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/tmrowco/electricitymap/issues/787#issuecomment-343930213, or mute the thread https://github.com/notifications/unsubscribe-auth/ABlEKAXJvgjpz6tWeUgSZM6zoIjlzMeRks5s2E45gaJpZM4PtlLQ .

[corradio]

CAISO is not being very open to increasing the rate at which they publish data. I suggest writing this scraper. The view of the last 24h will still be very useful, and hopefully it will start bringing attention to this issue!

Olivier

[7hibault]

I'll get started with this if this is fine for everyone.

Is it conceivable to fill in the data with predictions at some point?

[corradio]

I personally don't like it (because I'd rather show missing data to motivate data providers to fill their gaps).

Olivier

On Wed, Nov 15, 2017 at 10:26 AM, 7hibault notifications@github.com wrote:

I'll get started with this if this is fine for everyone.

Is it conceivable to fill in the data with predictions at some point?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/tmrowco/electricitymap/issues/787#issuecomment-344534543, or mute the thread https://github.com/notifications/unsubscribe-auth/ABlEKJwsV9wIYHWwHkGDwd14mGEYO8zNks5s2q4xgaJpZM4PtlLQ .

[7hibault]

That makes sense.

I have a question about the parser: the fetch_production, fetch_price and fetch_exchange functions are supposed to return the last known information. Here, the last known (hourly) information will always be at midnight from the day before, no matter when you query it. How does the backend get its historical information? Is there a way to fetch historical data or is it based on previous queries that are stored? If it's the latter, we might have to tweak the functions to return something like the information from 24 hours ago, but it's a bit ugly to change the meaning of the functions in one parser.

[corradio]

You can actually return a list of dictionaries containing the history: that is how the backend fetches previous data. I suggest that the fetch_production method returns all the data contained in the last valid file exposed by CAISO.

Olivier

On Mon, Nov 20, 2017 at 9:54 AM, 7hibault notifications@github.com wrote:

That makes sense.

I have a question about the parser: the fetch_production, fetch_price and fetch_exchange functions are supposed to return the last known information. Here, the last known (hourly) information will always be at midnight from the day before, no matter when you query it. How does the backend get its historical information? Is there a way to fetch historical data or is it based on previous queries that are stored? If it's the latter, we might have to tweak the functions to return something like the information from 24 hours ago, but it's a bit ugly to change the meaning of the functions in one parser.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/tmrowco/electricitymap/issues/787#issuecomment-345630026, or mute the thread https://github.com/notifications/unsubscribe-auth/ABlEKJrVPJo6Sor4DcKSH5vFgZY1P53Qks5s4T4qgaJpZM4PtlLQ .

[7hibault]

Here's a first version of the parser: https://github.com/7hibault/electricitymap/blob/master/parsers/US_CA.py

I've only tested it locally (aka checking the list I'm getting from the request, not integrating it in the map, I'm having issues with docker). It returns a list of objects with the energy mix. Some choices I've made:

• I've added hydro and small hydro for hydro
• I've added solar pv and solar thermal for solar
• Measurements are dated at the end of the past hour UTC-8, e.g their line for the first hour is dated at 16:59 UTC (which is 00:59 in California)
• I've chosen the country code US_CA
• Their info about thermal energy isn't used in the mix
• Imports aren't used yet

Is it somewhere near what you expected?

[corradio]

very good. Can you submit a PR?

On Sun, Nov 26, 2017 at 6:14 PM, 7hibault notifications@github.com wrote:

Here's a first version of the parser: https://github.com/7hibault/ electricitymap/blob/master/parsers/US_CA.py

I've only tested it locally (aka checking the list I'm getting from the request, not integrating it in the map, I'm having issues with docker). It returns a list of objects with the energy mix. Some choices I've made:

• I've added hydro and small hydro for hydro
• I've added solar pv and solar thermal for solar
• Measurements are dated at the end of the past hour UTC-8, e.g their line for the first hour is dated at 16:59 UTC (which is 00:59 in California)
• I've chosen the country code US_CA
• Their info about thermal energy isn't used in the mix
• Imports aren't used yet

Is it somewhere near what you expected?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/tmrowco/electricitymap/issues/787#issuecomment-347023662, or mute the thread https://github.com/notifications/unsubscribe-auth/ABlEKJ0rHZ2A3DaCB0qicGt6yXXf4GcNks5s6ZxcgaJpZM4PtlLQ .

[systemcatch]

@7hibault Thanks for taking the time to write this, it's appreciated.

As oli said submit a pull request so it can be reviewed properly. Couple of points for now.

• Both fetch_price and fetch_exchange functions can be deleted as they won't be used in this parser.
• You may want to look at using arrow for datetimes as currently they don't seem correct.

[7hibault]

Should we close this to keep the issue tracker clean?

[corradio]

All good! Thanks for the help

[corradio]

Re-opening because live data is now available here: http://www.caiso.com/TodaysOutlook/Pages/supply.aspx

[systemcatch]

@corradio Looks like your emails paid off! The breakdown is really detailed.

[7hibault]

Nice, congrats @corradio ! There's a warning at the bottom of the page:

Today’s Outlook is provided for informational purposes ONLY and should not be relied upon by any party for the actual billing values or operational planning. Official data is available in OASIS.

I can't find the same information on http://oasis.caiso.com/mrioasis/logon.do

[HansHyde]

Well done. Great information sources. Any further details on imports (and what they include)? Currently imports are between 35-40% of system load.

[corradio]

Unfortunately I don't know more. No problem with warming. That's expected. Maybe we can take a look at how the graphs get their data? Didn't dig, but indeed I saw that oasis didn't have that info yet.

On Thu, Dec 14, 2017 at 1:16 PM, Hans Hyde notifications@github.com wrote:

Well done. Great information sources. Any further details on imports (and what they include)? Currently imports are between 35-40% of system load.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/tmrowco/electricitymap/issues/787#issuecomment-351694943, or mute the thread https://github.com/notifications/unsubscribe-auth/ABlEKNrgoU15niFkitASO2H1L4B9acKPks5tARGTgaJpZM4PtlLQ .

[systemcatch]

The main graphs get their data from www.caiso.com/outlook/SP/fuelsource.csv

[7hibault]

Should we use this data as well then even if they explicitly say not to?

[corradio]

They didn't say not to use it?

On Thu, Dec 14, 2017 at 4:21 PM, 7hibault notifications@github.com wrote:

Should we use this data as well then even if they explicitly say not to?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/tmrowco/electricitymap/issues/787#issuecomment-351741440, or mute the thread https://github.com/notifications/unsubscribe-auth/ABlEKFniQmH61HHOEfvGSZoIDpXhWuMxks5tAT0XgaJpZM4PtlLQ .

[7hibault]

They say it's not official, for informational purposes only and unreliable for operational purposes. That's what I meant by "they said not to use it"

[corradio]

What they mean is that we shouldn't use it for billing. That's because they might get late information and update their data. That's the same with ENTSOE. We can still use it to display. They basically don't want to be liable in case decisions involving $ are taken :)

On Thu, Dec 14, 2017 at 4:32 PM, 7hibault notifications@github.com wrote:

They say it's not official, for informational purposes only and unreliable for operational purposes. That's what I meant by "they said not to use it"

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/tmrowco/electricitymap/issues/787#issuecomment-351744747, or mute the thread https://github.com/notifications/unsubscribe-auth/ABlEKKmYWjdOTDf6UHjlgqJS-vwQGlsOks5tAT-bgaJpZM4PtlLQ .

[HansHyde]

@systemcatch fyi on fuelsource graphs - last night NYISO...

Daily Mix is NOT registering imports - Checked numbers

• Load (Reported) = 23,185 MW
• Generation (Reported) = 18,967 MW <-- from the Generation Mix graph
• Diff (Load - Gen) -4,218 MW
• Imports (Reported) = 3,862 MW

As I posted above, 35-40% of CAISO's load/generation is "unknown" for CO2eq calculations.

[7hibault]

@corradio won't the pro version involve $$ ?

[corradio]

No billing information, so it's fine. But thanks for questioning this!

On Dec 14, 2017 17:39, "7hibault" notifications@github.com wrote:

@corradio https://github.com/corradio won't the pro version involve $$ ?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/tmrowco/electricitymap/issues/787#issuecomment-351765320, or mute the thread https://github.com/notifications/unsubscribe-auth/ABlEKB2PT6jBOc-YT9NN-eCWmi3bvH1Cks5tAU83gaJpZM4PtlLQ .

[7hibault]

Updated version of the parser here: https://github.com/7hibault/electricitymap/blob/master/parsers/US_CA.py

I will submit a PR but a few things before that:

• We should check the remarks from @HansHyde before adding this to prod. Have you checked that the difference is not somewhere in storage?
• How is the previous data generated? I'm not sure if the way I'm setting the date would break the history data generation.
• What should we do with the reported imports?
• Do you agree with this mapping?

  production_map = {
      'Solar': 'solar',
      'Wind': 'wind',
      'Geothermal': 'geothermal',
      'Biomass': 'biomass',
      'Biogas': 'gas',
      'Small hydro': 'hydro',
      'Coal': 'coal',
      'Nuclear': 'nuclear',
      'Natural gas': 'gas',
      'Large hydro': 'hydro',
      'Other': 'unknown'
  }
  storage_map = {
      'Batteries': 'battery'
  }

[corradio]

@7hibault let's rather talk on the PR to avoid too many messages on the issue. Mapping looks good to me. My only concern is whether or not we should mix biogas and gas. Do we know where imports come from?

[HansHyde]

@7hibault I am currently working on an excel workbook that will be able to account for these variations & "unknowns" for NYISO. It should be able to be applied to CAISO & all the other Balancing Authorities in the US.

Here is a screenshot of the worksheet, that is supported by other worksheets in the excel workbook.

Maybe there is a way to import realtime data (ie from CAISO or NYISO) into Excel that you are aware of? If so, that would be a great assistance to me, instead of manually importing current "realtime" data.

Thanks!

[brunolajoie]

@corradio @7hibault for all other countries we"ve mapped TSO's biogas category to EMap's biomass one. Then, we can alway refine the c-int factor for biomass!

[corradio]

Sounds like the next step is to add imports/exports. Do we have any data for this? I think they have exchanges with multiple neighboring zones, so taking compounded imports won't work..

[HansHyde]

This is likely the case.... have you looked at the CAISO "ISO Today" app? In that they give pricing across CA, NV, Utah.... Oregon??? in a regional all inclusive electric market perspective. This will likely be the route to determining import/export pathways, then through other methods determining generation mix.

[corradio]

New issue specifically for imports: https://github.com/tmrowco/electricitymap/issues/959

Thanks all for making this happening!

[Methanogen]

Hi, It seems that you have done a lot of good work in this area! I have been looking at negative prices for CAISO and have heard that Hydro is a factor. The daily sheet you refer to does provide 5-minute data for Small Hydro and for Large Hydro. Have either of you compiled a time series that goes back months or even a year by any chance? Regards, David Willson

[brunolajoie]

@Methanogen thanks! yes we do, please contact us at hello@electricitymap.org!