Meta-issue: verify or overhaul Australia parsers

[jarek]

Australian parser for AEMO has always been wonky and has several remaining outstanding issues, see: https://github.com/tmrowco/electricitymap-contrib/issues/1196, https://github.com/tmrowco/electricitymap-contrib/issues/799

Additionally Western Australia currently has no data, although it worked before.

Currently AEMO is still wonky, see for example the origin of electricity chart below, the flat-line followed by the jump around 4pm (my time) is likely an indication that data was missing or being rejected.

I think it needs a general overheaul, or at least a general verification that this is the best data available.

It would be great if someone could dig into it, get data Electricity Map has collected (comment here to get started) and compare with errors logged (see Kibana link in #1196).

There are also some alternate APIs listed in the issues linked - evaluating those and seeing how they compare to what we're using now would be awesome.

[anirudnits]

I would like to work on this issue. If nobody else is working on this, can I be assigned to this?

[jarek]

Looks like it should be good for you to take a look, @anirudnits!

[anirudnits]

@jarek thanks. Will do.

[alixunderplatz]

I would make this issue "top priority" due to the ongoing fires in Australia and the need for transparent electricity data. As we all know the AU production data we use is often faulty for some reason. That's why we often discussed switching to opennem.

Here are the links to the JSON files for each of the Australian zones (except Western Australia!) with 5 minute data (very good!) for the last 7 days up to real-time: NEM total (all AU-states except Western Australia): https://data.opennem.org.au/power/nem.json South Australia: https://data.opennem.org.au/power/sa1.json Victoria: https://data.opennem.org.au/power/vic1.json Tasmania: https://data.opennem.org.au/power/tas1.json New South Wales: https://data.opennem.org.au/power/nsw1.json Queensland: https://data.opennem.org.au/power/qld1.json

Compare with https://opennem.org.au/energy/nem for what the data should look like on our map. Select "simplified" above the table on the right to aggregate the different coal+solar technologies.

It should be farily easy to impement these beautiful JSON datasets and rewrite the Australia-parser. The source for the exchange flows between the zones could remain as it is now. We'd get rid of the annoying issues and have a perfect dataset for it. the scripts in lib/AU_solar.py and AU_battey.py could probably be dismissed.

Here is the mapping for the types in the json file:

production_technology = {
    "nem.fuel_tech.biomass.power": "biomass",
    "nem.fuel_tech.black_coal.power": "coal",
    "nem.fuel_tech.brown_coal.power": "coal",
    "nem.fuel_tech.distillate.power": "oil",
    "nem.fuel_tech.gas_ccgt.power": "gas",
    "nem.fuel_tech.gas_ocgt.power": "gas",
    "nem.fuel_tech.gas_recip.power": "gas",
    "nem.fuel_tech.gas_steam.power": "gas",
    "nem.fuel_tech.hydro.power": "hydro",
    "nem.fuel_tech.solar.power": "solar",
    "nem.fuel_tech.wind.power": "wind",
    "nem.fuel_tech.rooftop_solar.power": "solar"
    }

storage_technology" = {
    "nem.fuel_tech.pumps.power": "hydro",
    "nem.fuel_tech.battery_discharging.power": "battery",
    "nem.fuel_tech.battery_charging.power": "battery"
    }

Please note that "nem.fuel_tech.pumps.power" ONLY represents the power which is used to STORE energy. Discharging hydro is included in the usual production from "hydro". For battery, both charging and discharging power are given (yay!).

[con-cat]

I'm wondering if the issue I found with AU_solar.py in PR #2140 might have also been causing the flat line on the graphs for states other than WA.

AU_WA.py calls AU_solar.fetch_solar_all for the previous 24 hours, so it was always trying to get the previous day's data and failing on that api call.

AU.py calls AU_solar.fetch_solar_for_date, which then calls fetch_solar_all for the default two hours in the past, so there would have only been a two-hour window each day where it was trying and failing to get the previous day's solar data. That seems to match with the flat line on the graphs, and it looks like that's no longer there since the PR was merged.

[systemcatch]

We've mentioned using https://github.com/opennem/opennempy before to handle the data, however there is no PyPi package for that at present. PyPi can build from a git repo if needed.

When I download the CSV data from https://opennem.org.au/energy/nem/ each point comes with a timestamp, whereas the json seems to provide start and end points along with an interval e.g. 5m. While this isn't a blocker it's slightly annoying to handle.

Extra bonus is that historical data is available if we want to fill in the blanks in the current dataset (https://data.opennem.org.au/testing/nem/energy/daily/2019.json for all of 2019).

[simonhac]

fyi, the opennem backend is getting quite the overhaul right now. ping @nc9.

[corradio]

Duplicate of https://github.com/tmrowco/electricitymap-contrib/issues/2580