Closed mjohn425 closed 3 years ago
Nice find @mjohn425 !
The capacities are defined here if anyone wants to look into this
Also, Australian solar data (other than WA) should be coming from https://pv-map.apvi.org.au/live via https://github.com/tmrowco/electricitymap-contrib/blob/master/parsers/lib/AU_solar.py but judging by lack of any solar production reported in last 24 hours that's broken as well...
Hi - Sorry I'm new to ElectricityMap. I produce visualisations based on the free API but have been receiving questions around lack solar in SA and QLD. Just wondering how to test the http://pv-map.apvi.org.au/data API as I'm not convinced it is working properly. Can I run AU_solar.py on it's own to see if I can pull any data? Is there any reason why we use https://pv-map.apvi.org.au/live as opposed to AEMO such as here https://nemweb.com.au/Reports/Current/ROOFTOP_PV/ACTUAL/
With the above hourly numbers since Jun 2019, you would expect to see a reduction in carbon intensity during the day with rooftop solar.
It should be possible to reconcile generation to here also: https://opennem.org.au/energy/sa1/
The main issue is the underlying data source for the NEM is very poor. The data source that should be utilised in the AEMO MMS Dispatch SCADA files: https://nemweb.com.au/Reports/Current/Dispatch_SCADA/
Happy to collaborate to write a new parser to incorporate this data source.
Thanks for your help. It is my understanding that your source above only contains commercial solar farms and you also need to merge in https://nemweb.com.au/Reports/Current/ROOFTOP_PV/ACTUAL/ for rooftop solar, which is at 30 minute granularity. Just out of interest, I contacted ARENA (https://arena.gov.au/) via their web form to see if they could recommend a definitive source, but have had no response.
The commercial solar farms are typically >30MWe. The data source for those (and all other registered generators) is provided on a five-minute basis. The AEMO rooftop solar encompases all other PV assets, and as you note, is provided on a 30-minute basis.
The Australian PV Institute have a 5min based data source for distributed solar output, which I think is already utilised in ElectricityMap? I have heard there is an API for this data set also.
https://pv-map.apvi.org.au/live
Don't forget that wind generation that is also missing
The Australian PV Institute data source seems to be broken though, which I suspect is why this issue was raised. I can't return anything using this script locally: https://github.com/tmrowco/electricitymap-contrib/blob/master/parsers/lib/AU_solar.py. Might be just user error too!
I get an error also. I note the APVI data displayed on the webpage is only provided in 15 minute increments?
Is a short term solution to utilise the .../ROOFTOP_PV/ACTUAL/ data in the short term by holding the value constant or otherwise calculated for each five minute period over the thirty minute refresh period? The magnitude of error will likely be relatively small. This can be updated when a new/better dataset is found.
I feel the priority should be addressing the lack of solar/wind in the current calculations, rather than being 100% correct.
Thoughts?
OpenNEM have a rooftop solar source. https://opennem.org.au/energy/nem/
Yes, I'm wondering if we can leverage that, or whether we have to create another parser. I guess we can get in contact with one of the developers. Also not sure what format is required for ElectricityMap as this will be processed further to get carbon intensity I assume?
I'm not sure if they have exposed an API as part of the project. Simon Holmes a Court and Dylan McConnell are quiet active on twitter discussing energy, so could be worth an approach.
https://twitter.com/dylanjmcconnell https://twitter.com/simonahac
The OpenNEM API (below) works a treat,. Minor issues about pandas.np, but otherwise brings in the data set nicely. Parser would have to identify the current timestamp.
Looks like Simon is open to helping: https://twitter.com/simonahac/status/1315106334514180098
The OpenNEM .json output appears to provide all of the data required for ElectricityMap. The web_api.py provided in this repository (https://github.com/opennem/opennempy) may provide an easier means to collect the data at five minute intervals to avoid having to sort through a couple of days worth of data.
I'm not sure as to the frequency that the ElectricityMap parsers run. Anyone know the answer?
I'm happy to reach out to either Dylan or Simon to work them on this.
I'm not sure as to the frequency that the ElectricityMap parsers run. Anyone know the answer?
The parsers run every 15 minutes but can return more frequent data (e.g. a list of datapoints for every 5 minutes or I think even every minute) if that's available in the data source
Simon mentioned liaising with @AnthonyBriggs — he's begun investigations.
Hey guys, is anyone currently working on this?
cc @chalg @AnthonyBriggs
I've been looking at this, but free time's been a bit scarce. Doesn't look screamingly hard, it's just a question of getting to it. If someone wants to beat me to it, feel free :)
My time is also scarce and my python is a bit rusty, but some initial questions I would have, not having contributed to ElectricityMap before:
what are the rules around using different libraries like those required for opennempy?
Should we be using opennempy to pull in as much data as possible rather than just "distributed / small-scale / rooftop photovoltaic solar generation", which is the goal of the AU_solar.py script?
As AU_solar.py doesn't work, I'm not sure what output format is actually required
What downstream functions/processes are used to convert this output to carbon intensity?
Hi all, I've started working on a parser here: https://github.com/tmrowco/electricitymap-contrib/pull/2860 It reads from OpenNEM. However I wasn't able to find exchange data in OpenNEM
hi, i’m the product manager for opennem. we’d love to help.
i don’t hang out here, so probably best to DM me on twitter @simonahac.
Perhaps in the meantime it would be useful to mention to viewers of the site that the values in emission intensity may be significantly overestimated due to ignoring contribution by der and some other vre sources?
Hi,
The installed solar capacity for AUS-QLD is listed as 20MW as of writing this. We have single solar farms that exceed this value (Yarranlea for example). Wikipedia lists the currently operational large solar farms as 1.7 GW and CEC lists small scale solar capacity at 3 GW as of last year. Unsure of how best to collate data, but one of the sources that you have listed: https://pv-map.apvi.org.au/live seems to at least be far closer, at least in the same ball park compared to what is shown on the map.
The source above indicates solar output at around 1.4GW at midday (50% of capacity) and accounts for around 20% of our total energy usage.
I suspect a similar issue for basically the entirety of Australia (NSW at the very least). I wish I had time to contribute but am flat out at work at the moment, love the idea though.