search

electricitymaps / electricitymaps-contrib

The open source repository for Electricity Maps App and data parsers that enables a real-time visualisation of the CO2 emissions of electricity consumption
https://app.electricitymaps.com
GNU Affero General Public License v3.0
3.6k stars 946 forks source link

AUS-QLD installed solar capacity way off mark. #2580

Closed mjohn425 closed 3 years ago

mjohn425 commented 4 years ago

Hi,

The installed solar capacity for AUS-QLD is listed as 20MW as of writing this. We have single solar farms that exceed this value (Yarranlea for example). Wikipedia lists the currently operational large solar farms as 1.7 GW and CEC lists small scale solar capacity at 3 GW as of last year. Unsure of how best to collate data, but one of the sources that you have listed: https://pv-map.apvi.org.au/live seems to at least be far closer, at least in the same ball park compared to what is shown on the map.

The source above indicates solar output at around 1.4GW at midday (50% of capacity) and accounts for around 20% of our total energy usage.

I suspect a similar issue for basically the entirety of Australia (NSW at the very least). I wish I had time to contribute but am flat out at work at the moment, love the idea though.

FelixDQ commented 4 years ago

Nice find @mjohn425 !

The capacities are defined here if anyone wants to look into this

jarek commented 4 years ago

Also, Australian solar data (other than WA) should be coming from https://pv-map.apvi.org.au/live via https://github.com/tmrowco/electricitymap-contrib/blob/master/parsers/lib/AU_solar.py but judging by lack of any solar production reported in last 24 hours that's broken as well...

chalg commented 4 years ago

Hi - Sorry I'm new to ElectricityMap. I produce visualisations based on the free API but have been receiving questions around lack solar in SA and QLD. Just wondering how to test the http://pv-map.apvi.org.au/data API as I'm not convinced it is working properly. Can I run AU_solar.py on it's own to see if I can pull any data? Is there any reason why we use https://pv-map.apvi.org.au/live as opposed to AEMO such as here https://nemweb.com.au/Reports/Current/ROOFTOP_PV/ACTUAL/ cie_sa_static_box_hr With the above hourly numbers since Jun 2019, you would expect to see a reduction in carbon intensity during the day with rooftop solar.

It should be possible to reconcile generation to here also: https://opennem.org.au/energy/sa1/

jamesy1988 commented 4 years ago

The main issue is the underlying data source for the NEM is very poor. The data source that should be utilised in the AEMO MMS Dispatch SCADA files: https://nemweb.com.au/Reports/Current/Dispatch_SCADA/

Happy to collaborate to write a new parser to incorporate this data source.

chalg commented 4 years ago

Thanks for your help. It is my understanding that your source above only contains commercial solar farms and you also need to merge in https://nemweb.com.au/Reports/Current/ROOFTOP_PV/ACTUAL/ for rooftop solar, which is at 30 minute granularity. Just out of interest, I contacted ARENA (https://arena.gov.au/) via their web form to see if they could recommend a definitive source, but have had no response.

jamesy1988 commented 4 years ago

The commercial solar farms are typically >30MWe. The data source for those (and all other registered generators) is provided on a five-minute basis. The AEMO rooftop solar encompases all other PV assets, and as you note, is provided on a 30-minute basis.

The Australian PV Institute have a 5min based data source for distributed solar output, which I think is already utilised in ElectricityMap? I have heard there is an API for this data set also.

https://pv-map.apvi.org.au/live

Don't forget that wind generation that is also missing

chalg commented 4 years ago

The Australian PV Institute data source seems to be broken though, which I suspect is why this issue was raised. I can't return anything using this script locally: https://github.com/tmrowco/electricitymap-contrib/blob/master/parsers/lib/AU_solar.py. Might be just user error too!

jamesy1988 commented 4 years ago

I get an error also. I note the APVI data displayed on the webpage is only provided in 15 minute increments?

Is a short term solution to utilise the .../ROOFTOP_PV/ACTUAL/ data in the short term by holding the value constant or otherwise calculated for each five minute period over the thirty minute refresh period? The magnitude of error will likely be relatively small. This can be updated when a new/better dataset is found.

I feel the priority should be addressing the lack of solar/wind in the current calculations, rather than being 100% correct.

Thoughts?

jamesy1988 commented 4 years ago

OpenNEM have a rooftop solar source. https://opennem.org.au/energy/nem/

https://github.com/opennem/opennempy

chalg commented 4 years ago

Yes, I'm wondering if we can leverage that, or whether we have to create another parser. I guess we can get in contact with one of the developers. Also not sure what format is required for ElectricityMap as this will be processed further to get carbon intensity I assume?

jamesy1988 commented 4 years ago

I'm not sure if they have exposed an API as part of the project. Simon Holmes a Court and Dylan McConnell are quiet active on twitter discussing energy, so could be worth an approach.

https://twitter.com/dylanjmcconnell https://twitter.com/simonahac

jamesy1988 commented 4 years ago

The OpenNEM API (below) works a treat,. Minor issues about pandas.np, but otherwise brings in the data set nicely. Parser would have to identify the current timestamp.

https://github.com/opennem/opennempy

chalg commented 4 years ago

Looks like Simon is open to helping: https://twitter.com/simonahac/status/1315106334514180098

jamesy1988 commented 4 years ago

The OpenNEM .json output appears to provide all of the data required for ElectricityMap. The web_api.py provided in this repository (https://github.com/opennem/opennempy) may provide an easier means to collect the data at five minute intervals to avoid having to sort through a couple of days worth of data.

I'm not sure as to the frequency that the ElectricityMap parsers run. Anyone know the answer?

I'm happy to reach out to either Dylan or Simon to work them on this.

jarek commented 4 years ago

I'm not sure as to the frequency that the ElectricityMap parsers run. Anyone know the answer?

The parsers run every 15 minutes but can return more frequent data (e.g. a list of datapoints for every 5 minutes or I think even every minute) if that's available in the data source

chalg commented 4 years ago

Simon mentioned liaising with @AnthonyBriggs — he's begun investigations.

Kongkille commented 4 years ago

Hey guys, is anyone currently working on this?

cc @chalg @AnthonyBriggs

AnthonyBriggs commented 4 years ago

I've been looking at this, but free time's been a bit scarce. Doesn't look screamingly hard, it's just a question of getting to it. If someone wants to beat me to it, feel free :)

chalg commented 4 years ago

My time is also scarce and my python is a bit rusty, but some initial questions I would have, not having contributed to ElectricityMap before:

corradio commented 3 years ago

Hi all, I've started working on a parser here: https://github.com/tmrowco/electricitymap-contrib/pull/2860 It reads from OpenNEM. However I wasn't able to find exchange data in OpenNEM

simonhac commented 3 years ago

hi, i’m the product manager for opennem. we’d love to help.

i don’t hang out here, so probably best to DM me on twitter @simonahac.

jufemaiz commented 3 years ago

Perhaps in the meantime it would be useful to mention to viewers of the site that the values in emission intensity may be significantly overestimated due to ignoring contribution by der and some other vre sources?