search

opennem / opennem-fe

An Open Platform for National Electricity Market Data
https://opennem.org.au
MIT License
65 stars 12 forks source link

Renewable power proportion discrepancy #227

Open simonhac opened 1 year ago

simonhac commented 1 year ago

We say 68.5% while GPE NEMLog2 says 67.23% for 11am AEST on Wednesday 6 September.

image
simonhac commented 1 year ago

@chienleng what's our formula for calculating renewable power proportion in this context?

chienleng commented 1 year ago

@simonhac Renewables: 18234.608

All Sources: 26629.002

Renewables/AllSources*100 = 68.476%

dylanjmcconnell commented 1 year ago

The number from GPE NEMLog2 looks to be wind and solar only (...though percentages still don't match if you drop off hydro/biomass/biogas either).

simonhac commented 1 year ago

oops… the original article does say "Wind and solar hit a new record share of the country’s main grid on Wednesday – reaching a new peak of 67.23 per cent on Wednesday".

thanks @dylanjmcconnell for alerting me that it was probably just wind and solar. sorry for the false alarm!

…that said there is still a discrepancy —66.23% vs 66.55% — as dylan pointed out.

_(10337 solar_rooftop + 4393 solarutility + 2992 wind) / (26629 everything) = 66.55%

i've messaged geoff eldridge (the guy behind GPE NEMLog) to seek his opinion.

simonhac commented 1 year ago

from geoff:

Yes. I noticed the difference also and have always put it down to difference RooftopPV values, but in this case it is the same (I use the AEMO 30 min values, interpolated to 5min values)

You make make it 66.55%

(10337 solar_rooftop + 4393 solar_utility + 2992 wind) / (26629 everything) = 66.55%

I have ..

(10338 solar_rooftop + 4436 solar_utility + 2992 wind) / (26426 everything [1]) = 67.22924393%

[1] 26426 = 16088 + 10338 (ie DEMAND_AND_NONSCHEDGEN + ROOFTOP_PV)

There is a difference in the Utility Solar (I have 4436 and you have 4393). I have downloaded the Utility Solar Units (forwarded), checked for duplicates and summed to 4435.3876 MW which is my Utility Solar value.

There is also a difference in the denominator, the Native Demand (ie DEMAND_AND_NONSCHEDGEN + ROOFTOP_PV), I have 26426 and you have 26629.

simonhac commented 1 year ago

Here are the NEM Solar Units I have in NEMLog:

"time","REGIONID","CO2E_ID","DUID","SCADAVALUE"
2023-09-06 11:00:00.000,NSW1,15,AVLSF1,156
2023-09-06 11:00:00.000,NSW1,15,BERYLSF1,73.8
2023-09-06 11:00:00.000,NSW1,15,BOMENSF1,73.4
2023-09-06 11:00:00.000,NSW1,15,BROKENH1,34.3
2023-09-06 11:00:00.000,NSW1,15,COLEASF1,129
2023-09-06 11:00:00.000,NSW1,15,CRWASF1,26.8
2023-09-06 11:00:00.000,NSW1,15,DARLSF1,224
2023-09-06 11:00:00.000,NSW1,15,FINLYSF1,114
2023-09-06 11:00:00.000,NSW1,15,GNNDHSF1,92.5
2023-09-06 11:00:00.000,NSW1,15,GOONSF1,28.4
2023-09-06 11:00:00.000,NSW1,15,GRIFSF1,23.5
2023-09-06 11:00:00.000,NSW1,15,GULLRSF1,7.40
2023-09-06 11:00:00.000,NSW1,15,HILLSTN1,83.3
2023-09-06 11:00:00.000,NSW1,15,JEMALNG1,43.2
2023-09-06 11:00:00.000,NSW1,15,JUNEESF1,21.1
2023-09-06 11:00:00.000,NSW1,15,LIMOSF11,210
2023-09-06 11:00:00.000,NSW1,15,LIMOSF21,29.0
2023-09-06 11:00:00.000,NSW1,15,MANSLR1,0
2023-09-06 11:00:00.000,NSW1,15,METZSF1,100
2023-09-06 11:00:00.000,NSW1,15,MLSP1,11.9
2023-09-06 11:00:00.000,NSW1,15,MOLNGSF1,0
2023-09-06 11:00:00.000,NSW1,15,MOREESF1,49.5
2023-09-06 11:00:00.000,NSW1,15,NEVERSF1,92.5
2023-09-06 11:00:00.000,NSW1,15,NEWENSF1,0
2023-09-06 11:00:00.000,NSW1,15,NEWENSF2,0
2023-09-06 11:00:00.000,NSW1,15,NYNGAN1,98.5
2023-09-06 11:00:00.000,NSW1,15,PARSF1,21.5
2023-09-06 11:00:00.000,NSW1,15,SEBSF1,66.0
2023-09-06 11:00:00.000,NSW1,15,SUNRSF1,178
2023-09-06 11:00:00.000,NSW1,15,SUNTPSF1,121
2023-09-06 11:00:00.000,NSW1,15,WAGGNSF1,34.5
2023-09-06 11:00:00.000,NSW1,15,WELLSF1,132
2023-09-06 11:00:00.000,NSW1,15,WRSF1,19.8
2023-09-06 11:00:00.000,NSW1,15,WSTWYSF1,79.5
2023-09-06 11:00:00.000,NSW1,15,WYASF1,26.0
2023-09-06 11:00:00.000,QLD1,15,BARCSF1,13.3
2023-09-06 11:00:00.000,QLD1,15,BLUEGSF1,0
2023-09-06 11:00:00.000,QLD1,15,CHILDSF1,23.9
2023-09-06 11:00:00.000,QLD1,15,CLARESF1,2.13
2023-09-06 11:00:00.000,QLD1,15,CLERMSF1,29.1
2023-09-06 11:00:00.000,QLD1,15,COLUMSF1,148
2023-09-06 11:00:00.000,QLD1,15,CSPVPS1,40.7
2023-09-06 11:00:00.000,QLD1,15,DAYDSF1,117
2023-09-06 11:00:00.000,QLD1,15,DDSF1,91.7
2023-09-06 11:00:00.000,QLD1,15,EDENVSF1,147
2023-09-06 11:00:00.000,QLD1,15,EMERASF1,0.0100
2023-09-06 11:00:00.000,QLD1,15,GANGARR1,70.3
2023-09-06 11:00:00.000,QLD1,15,HAMISF1,40.4
2023-09-06 11:00:00.000,QLD1,15,HAUGHT11,75.5
2023-09-06 11:00:00.000,QLD1,15,HAYMSF1,40.7
2023-09-06 11:00:00.000,QLD1,15,HUGSF1,5.45
2023-09-06 11:00:00.000,QLD1,15,KEPSF1,0.870
2023-09-06 11:00:00.000,QLD1,15,KSP1,46.1
2023-09-06 11:00:00.000,QLD1,15,LILYSF1,82.8
2023-09-06 11:00:00.000,QLD1,15,LRSF1,0.510
2023-09-06 11:00:00.000,QLD1,15,MARYRSF1,23.0
2023-09-06 11:00:00.000,QLD1,15,MIDDLSF1,22.9
2023-09-06 11:00:00.000,QLD1,15,MOUSF1,0.0226
2023-09-06 11:00:00.000,QLD1,15,OAKEY1SF,1.02
2023-09-06 11:00:00.000,QLD1,15,OAKEY2SF,1.01
2023-09-06 11:00:00.000,QLD1,15,RRSF1,95.5
2023-09-06 11:00:00.000,QLD1,15,RUGBYR1,56.0
2023-09-06 11:00:00.000,QLD1,15,SMCSF1,109
2023-09-06 11:00:00.000,QLD1,15,SRSF1,52.1
2023-09-06 11:00:00.000,QLD1,15,WANDSF1,0.103
2023-09-06 11:00:00.000,QLD1,15,WARWSF1,0.100
2023-09-06 11:00:00.000,QLD1,15,WARWSF2,0.100
2023-09-06 11:00:00.000,QLD1,15,WDGPH1,195
2023-09-06 11:00:00.000,QLD1,15,WHITSF1,0
2023-09-06 11:00:00.000,QLD1,15,WOOLGSF1,109
2023-09-06 11:00:00.000,QLD1,15,YARANSF1,77.8
2023-09-06 11:00:00.000,SA1,15,ADPPV1,0.0990
2023-09-06 11:00:00.000,SA1,15,BNGSF1,0.260
2023-09-06 11:00:00.000,SA1,15,BNGSF2,0.300
2023-09-06 11:00:00.000,SA1,15,BOWWPV1,0.00300
2023-09-06 11:00:00.000,SA1,15,HVWWPV1,4.01
2023-09-06 11:00:00.000,SA1,15,MAPS2PV1,6.01
2023-09-06 11:00:00.000,SA1,15,MAPS3PV1,5.99
2023-09-06 11:00:00.000,SA1,15,MBPS2PV1,5
2023-09-06 11:00:00.000,SA1,15,MWPS1PV1,4.48
2023-09-06 11:00:00.000,SA1,15,MWPS2PV1,4.22
2023-09-06 11:00:00.000,SA1,15,MWPS3PV1,0
2023-09-06 11:00:00.000,SA1,15,MWPS4PV1,2.18
2023-09-06 11:00:00.000,SA1,15,PAREPS1,76.7
2023-09-06 11:00:00.000,SA1,15,TB2SF1,0
2023-09-06 11:00:00.000,SA1,15,TBSF1,0
2023-09-06 11:00:00.000,VIC1,15,BANN1,0.530
2023-09-06 11:00:00.000,VIC1,15,COHUNSF1,0.0700
2023-09-06 11:00:00.000,VIC1,15,GANNSF1,42.6
2023-09-06 11:00:00.000,VIC1,15,GLRWNSF1,0
2023-09-06 11:00:00.000,VIC1,15,KARSF1,0.100
2023-09-06 11:00:00.000,VIC1,15,KIAMSF1,161
2023-09-06 11:00:00.000,VIC1,15,NUMURSF1,1.04
2023-09-06 11:00:00.000,VIC1,15,WEMENSF1,0.100
2023-09-06 11:00:00.000,VIC1,15,WINTSF1,1.98
2023-09-06 11:00:00.000,VIC1,15,YATSF1,0.190
chienleng commented 1 year ago

note I am not using demand as denominator but total generation. If I was to use demand (which includes pumps -726.3 and battery charging -233.6), I think it will be 69% renewables.

dylanjmcconnell commented 1 year ago

... FWIW my numbers for utility solar match Geoff's (almost exactly, with exception of some different precision - see below). Suspect this means that openNEM is missing some utility solar plants atm.

SETTLEMENTDATE REGIONID DUID Geoff Dylan
2023-09-06 11:00:00.000 NSW1 AVLSF1 156 156.492
2023-09-06 11:00:00.000 NSW1 BERYLSF1 73.8 73.8096
2023-09-06 11:00:00.000 NSW1 BOMENSF1 73.4 73.3806
2023-09-06 11:00:00.000 NSW1 BROKENH1 34.3 34.2995
2023-09-06 11:00:00.000 NSW1 COLEASF1 129 128.799
2023-09-06 11:00:00.000 NSW1 CRWASF1 26.8 26.7971
2023-09-06 11:00:00.000 NSW1 DARLSF1 224 223.894
2023-09-06 11:00:00.000 NSW1 FINLYSF1 114 113.829
2023-09-06 11:00:00.000 NSW1 GNNDHSF1 92.5 92.5244
2023-09-06 11:00:00.000 NSW1 GOONSF1 28.4 28.3732
2023-09-06 11:00:00.000 NSW1 GRIFSF1 23.5 23.5004
2023-09-06 11:00:00.000 NSW1 GULLRSF1 7.4 7.40489
2023-09-06 11:00:00.000 NSW1 HILLSTN1 83.3 83.3331
2023-09-06 11:00:00.000 NSW1 JEMALNG1 43.2 43.22
2023-09-06 11:00:00.000 NSW1 JUNEESF1 21.1 21.1302
2023-09-06 11:00:00.000 NSW1 LIMOSF11 210 210.035
2023-09-06 11:00:00.000 NSW1 LIMOSF21 29 29.0279
2023-09-06 11:00:00.000 NSW1 MANSLR1 0 0
2023-09-06 11:00:00.000 NSW1 METZSF1 100 100.189
2023-09-06 11:00:00.000 NSW1 MLSP1 11.9 11.887
2023-09-06 11:00:00.000 NSW1 MOLNGSF1 0 0
2023-09-06 11:00:00.000 NSW1 MOREESF1 49.5 49.468
2023-09-06 11:00:00.000 NSW1 NEVERSF1 92.5 92.51
2023-09-06 11:00:00.000 NSW1 NEWENSF1 0 0
2023-09-06 11:00:00.000 NSW1 NEWENSF2 0 0
2023-09-06 11:00:00.000 NSW1 NYNGAN1 98.5 98.5
2023-09-06 11:00:00.000 NSW1 PARSF1 21.5 21.5015
2023-09-06 11:00:00.000 NSW1 SEBSF1 66 65.9703
2023-09-06 11:00:00.000 NSW1 SUNRSF1 178 178.141
2023-09-06 11:00:00.000 NSW1 SUNTPSF1 121 120.968
2023-09-06 11:00:00.000 NSW1 WAGGNSF1 34.5 34.5183
2023-09-06 11:00:00.000 NSW1 WELLSF1 132 132.427
2023-09-06 11:00:00.000 NSW1 WRSF1 19.8 19.8102
2023-09-06 11:00:00.000 NSW1 WSTWYSF1 79.5 79.487
2023-09-06 11:00:00.000 NSW1 WYASF1 26 25.989
2023-09-06 11:00:00.000 QLD1 BARCSF1 13.3 13.3
2023-09-06 11:00:00.000 QLD1 BLUEGSF1 0 0
2023-09-06 11:00:00.000 QLD1 CHILDSF1 23.9 23.9
2023-09-06 11:00:00.000 QLD1 CLARESF1 2.13 2.12928
2023-09-06 11:00:00.000 QLD1 CLERMSF1 29.1 29.09
2023-09-06 11:00:00.000 QLD1 COLUMSF1 148 147.774
2023-09-06 11:00:00.000 QLD1 CSPVPS1 40.7 40.689
2023-09-06 11:00:00.000 QLD1 DAYDSF1 117 117.346
2023-09-06 11:00:00.000 QLD1 DDSF1 91.7 91.6644
2023-09-06 11:00:00.000 QLD1 EDENVSF1 147 147.047
2023-09-06 11:00:00.000 QLD1 EMERASF1 0.01 0.01
2023-09-06 11:00:00.000 QLD1 GANGARR1 70.3 70.339
2023-09-06 11:00:00.000 QLD1 HAMISF1 40.4 40.4
2023-09-06 11:00:00.000 QLD1 HAUGHT11 75.5 75.5302
2023-09-06 11:00:00.000 QLD1 HAYMSF1 40.7 40.65
2023-09-06 11:00:00.000 QLD1 HUGSF1 5.45 5.445
2023-09-06 11:00:00.000 QLD1 KEPSF1 0.87 0.87
2023-09-06 11:00:00.000 QLD1 KSP1 46.1 46.1
2023-09-06 11:00:00.000 QLD1 LILYSF1 82.8 82.8375
2023-09-06 11:00:00.000 QLD1 LRSF1 0.51 0.51
2023-09-06 11:00:00.000 QLD1 MARYRSF1 23 22.97
2023-09-06 11:00:00.000 QLD1 MIDDLSF1 22.9 22.88
2023-09-06 11:00:00.000 QLD1 MOUSF1 0.0226 0.02257
2023-09-06 11:00:00.000 QLD1 OAKEY1SF 1.02 1.022
2023-09-06 11:00:00.000 QLD1 OAKEY2SF 1.01 1.01
2023-09-06 11:00:00.000 QLD1 RRSF1 95.5 95.4644
2023-09-06 11:00:00.000 QLD1 RUGBYR1 56 56.0067
2023-09-06 11:00:00.000 QLD1 SMCSF1 109 109.417
2023-09-06 11:00:00.000 QLD1 SRSF1 52.1 52.11
2023-09-06 11:00:00.000 QLD1 WANDSF1 0.103 0.10312
2023-09-06 11:00:00.000 QLD1 WARWSF1 0.1 0.1
2023-09-06 11:00:00.000 QLD1 WARWSF2 0.1 0.1
2023-09-06 11:00:00.000 QLD1 WDGPH1 195 194.766
2023-09-06 11:00:00.000 QLD1 WHITSF1 0 0
2023-09-06 11:00:00.000 QLD1 WOOLGSF1 109 109.2
2023-09-06 11:00:00.000 QLD1 YARANSF1 77.8 77.82
2023-09-06 11:00:00.000 SA1 ADPPV1 0.099 0.099
2023-09-06 11:00:00.000 SA1 BNGSF1 0.26 0.26
2023-09-06 11:00:00.000 SA1 BNGSF2 0.3 0.3
2023-09-06 11:00:00.000 SA1 BOWWPV1 0.003 0.003
2023-09-06 11:00:00.000 SA1 HVWWPV1 4.01 4.009
2023-09-06 11:00:00.000 SA1 MAPS2PV1 6.01 6.01
2023-09-06 11:00:00.000 SA1 MAPS3PV1 5.99 5.99
2023-09-06 11:00:00.000 SA1 MBPS2PV1 5 5
2023-09-06 11:00:00.000 SA1 MWPS1PV1 4.48 4.48
2023-09-06 11:00:00.000 SA1 MWPS2PV1 4.22 4.22
2023-09-06 11:00:00.000 SA1 MWPS3PV1 0 0
2023-09-06 11:00:00.000 SA1 MWPS4PV1 2.18 2.18
2023-09-06 11:00:00.000 SA1 PAREPS1 76.7 76.71
2023-09-06 11:00:00.000 SA1 TB2SF1 0 0
2023-09-06 11:00:00.000 SA1 TBSF1 0 0
2023-09-06 11:00:00.000 VIC1 BANN1 0.53 0.53
2023-09-06 11:00:00.000 VIC1 COHUNSF1 0.07 0.07
2023-09-06 11:00:00.000 VIC1 GANNSF1 42.6 42.6
2023-09-06 11:00:00.000 VIC1 GLRWNSF1 0 0
2023-09-06 11:00:00.000 VIC1 KARSF1 0.1 0.1
2023-09-06 11:00:00.000 VIC1 KIAMSF1 161 160.78
2023-09-06 11:00:00.000 VIC1 NUMURSF1 1.04 1.04
2023-09-06 11:00:00.000 VIC1 WEMENSF1 0.1 0.1
2023-09-06 11:00:00.000 VIC1 WINTSF1 1.98 1.98
2023-09-06 11:00:00.000 VIC1 YATSF1 0.19 0.19
simonhac commented 1 year ago

geoff says:

I guess I have always assumed the denominator to the DEMAND_AND_NONSCHEDGEN + ROOFTOP_PV. With so many definitions of demands (there is 27-30 page AEMO docuement with demand definitions) I have just kept it simple. Maybe I should just go with total generation as you do. Thanks for clarifying.

@dylanjmcconnell what would you say to geoff?

simonhac commented 1 year ago

@nc9 can you look to see if we are missing some solar plants? (and if we are, how we are missing them?)

nc9 commented 1 year ago

we are - they're being imported later today

edit: it involves wrangling with stations.json

simonhac commented 1 year ago

we need to not go a day with orphaned DUIDs — otherwise any records we announce are wrong, charts we reproduce are wrong, research based on our data is wrong.

@nc9 how does our system pick this up and tell us?

simonhac commented 1 year ago

we are - they're being imported later today

is there a ticket for them?

nc9 commented 1 year ago

no it's just something I do when alerted

dylanjmcconnell commented 1 year ago

At a guess - it looks like there's one missing in NSW, and another one or two in South Australia - the discrepancy in South Australia is relatively significant (and was probably obscured by the fact that there was a lot of curtailment occurring in the middle of the day on the 6th Sept etc). e.g. see below. SA-check

c.f. Victoria VIC-check

simonhac commented 1 year ago

we need visibility on known unallocated DUIDs.

nc9 commented 1 year ago

it’s in #monitoring every day

On 12 Sep 2023, at 3:06 pm, simonhac @.***> wrote:

we need visibility on known unlocated DUIDs. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

simonhac commented 1 year ago

this?

image

simonhac commented 1 year ago

or rather, in the modern era (today): image

dylanjmcconnell commented 1 year ago

geoff says:

I guess I have always assumed the denominator to the DEMAND_AND_NONSCHEDGEN + ROOFTOP_PV. With so many definitions of demands (there is 27-30 page AEMO docuement with demand definitions) I have just kept it simple. Maybe I should just go with total generation as you do. Thanks for clarifying.

@dylanjmcconnell what would you say to geoff?

I actually also tend to use DEMAND_AND_NONSCHEDGEN + ROOFTOP_PV to represent a kind of "underlying demand" within a region- it's relatively easy and common (I think) - It's basically summation of all generation - scheduled, non-scheduled and rooftop solar. One benefit of just using DEMAND_AND_NONSCHEDGEN + ROOFTOP_PV is that you can just look those numbers up directly from the MMS tables / not have to derive it yourself. Somewhat easier to check against AEMO.

But basically agree with Geoff in that there is a lot of different demand terms - (and there is merit in keeping simple). But there is also no single "right" answer ... as long as it's documented somewhere.

nc9 commented 1 year ago

bd5ba70c589381886327c93e6ac925333d270d70

nc9 commented 1 year ago

will be on prod in ~5 min

simonhac commented 1 year ago

which ones are being added? (or rather, are there any unallocated ones remaining?)

nc9 commented 1 year ago

i’m adding BBATTERY right now

On 12 Sep 2023, at 2:51 pm, simonhac @.***> wrote:

@nc9 can you look to see if we are missing some solar plants? (and if we are, how we are missing them?) — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

nc9 commented 1 year ago

this is what is remaining unmapped from facility_scada, with their total generation to date alongside (ie. 0)

[2023-09-12 16:01:06,433] opennem.monitors.facility_seen          INFO] Unmapped: NEM ASNSEL2 0.0
[2023-09-12 16:01:06,433] opennem.monitors.facility_seen          INFO] Unmapped: NEM ASPAPM01 0.0
[2023-09-12 16:01:06,433] opennem.monitors.facility_seen          INFO] Unmapped: NEM ASRMGE01 0.0
[2023-09-12 16:01:06,434] opennem.monitors.facility_seen          INFO] Unmapped: NEM ASRMGE02 0.0
[2023-09-12 16:01:06,434] opennem.monitors.facility_seen          INFO] Unmapped: NEM ASRMGE03 0.0
[2023-09-12 16:01:06,434] opennem.monitors.facility_seen          INFO] Unmapped: NEM ASSDE1 0.0
[2023-09-12 16:01:06,434] opennem.monitors.facility_seen          INFO] Unmapped: NEM ASSSN1 0.0
[2023-09-12 16:01:06,434] opennem.monitors.facility_seen          INFO] Unmapped: NEM ASVIEL2 0.0
[2023-09-12 16:01:06,434] opennem.monitors.facility_seen          INFO] Unmapped: NEM DRACAN1 0.0
[2023-09-12 16:01:06,434] opennem.monitors.facility_seen          INFO] Unmapped: NEM DRAESN1 0.0
[2023-09-12 16:01:06,434] opennem.monitors.facility_seen          INFO] Unmapped: NEM DRFIIN01 0.0
[2023-09-12 16:01:06,434] opennem.monitors.facility_seen          INFO] Unmapped: NEM DRSHVN01 0.0
[2023-09-12 16:01:06,434] opennem.monitors.facility_seen          INFO] Unmapped: NEM DRSHVS01 0.0
simonhac commented 1 year ago

we had:

(10337 solar_rooftop + 4393 solar_utility + 2992 wind) / (26629 everything) = 66.55%

we now have:

(10337 solar_rooftop + 4419 solar_utility + 2992 wind) / (26655 everything) = 66.58413056%

geoff has:

(10338 solar_rooftop + 4436 solar_utility + 2992 wind) / (26426 everything [1]) = 67.22924393%

[1] 26426 = 16088 + 10338 (ie DEMAND_AND_NONSCHEDGEN + ROOFTOP_PV)

disturbs me that we are 1MW below NEMlog on solar_rooftop, and 17MW below NEMlog on solar_utility.

simonhac commented 1 year ago

dylan just made this, from our JSON generated at 2023-09-12 23:05:21+10:00.

image

ie. there's still a discrepancy between OpenNEM and AEMO's MMS for utility_solar in SA1.

nc9 commented 1 year ago

One way to do it. Here is all of our solar_utility for SA1 - which one are we missing? I'm going to assumed this is a mismapped DUID

nc9 commented 1 year ago

fixed in:

https://github.com/opennem/opennem/commit/c8623a0e9e1399c5c634b0514d6dcc0d99de5522

where we were filtering DUID's and it was filtering out new units in certain conditions.

facs added in:

https://github.com/opennem/opennem/commit/92e0950cc1d76c26246a907c13c71e00372dab82

prod should be up-to-date in ~15 minutes.

dylanjmcconnell commented 1 year ago

Based on the table Nik posted above - looks like openNEM is missing:

Screenshot from 2023-09-13 10-46-54

nc9 commented 1 year ago

I'll explain what happen - our facility_seen process which alerts us to new DUID's was broken because it made a poor attempt at filtering the noise of VPP's, loads, reserve traders and more:

This is the old version:

https://github.com/opennem/opennem/blob/e0e568f336e18e78a14edaa4a4c8e951cae11369/opennem/monitors/facility_seen.py

It accidentally caught some legit new facilities (specifically, in cases where pumping stations now have solar as well)

New version is less fancy and just alerts (4 times a day) on any new DUID that shows generation > 0

https://github.com/opennem/opennem/blob/master/opennem/monitors/facility_seen.py#L107

The good new is new mappings auto-backfill data

ideally we would prep new facilities and DUIDs prior to first generation, and prior to REL/GI updates - If anybody has any ideas on how we can get insight into the reg. process, we can plug them in ahead of time, that would be even better. We already have a process to import stations -> facilities in ('planning', 'committed') et al [0] since we don't use DUID as a primary key. It's all about getting the DUID's in (and DUID name changes!) in asap.

[0] per simon, I don't believe we want planning stages in our stationfacilities database as these often change/fail and there is a lot of noise, but it might be nice to have 'planning' on a per-case basis where we know they're likely to migrate through the process, or high-profile cases like the HEZ

aleith commented 1 year ago

ideally we would prep new facilities and DUIDs prior to first generation, and prior to REL/GI updates - If anybody has any ideas on how we can get insight into the reg. process, we can plug them in ahead of time, that would be even better. We already have a process to import stations -> facilities in ('planning', 'committed') et al [0] since we don't use DUID as a primary key. It's all about getting the DUID's in (and DUID name changes!) in asap.

I could ask Dean Sharafi who is GM of systems at AEMO WA on behalf of openNEM if you want me too. he's been really helpful in answering questions in the past. What me to?

I previously had concerns about new hybrid facilities where wind farms were to add PV and/or batteries and dispatch under a single facility code. Not sure the ever got answered, I think the SCADA data distinguishes between what happens behind the meter at generation facilities with storage etc but cant be sure now, was a long time ago. Presumably AEMO needs visibility of these things like BTM storage and dispatch to storage to manage contingencies? Even if it's just picking up the phone and getting answers quickly.

nc9 commented 1 year ago

What me to?

Yes please. It would help us immensely. At the moment, we just wait for new DUIDs to appear in scada data or appear in the REL/GI lists (which is a manual process since they are difficult to process programmatically as the format seems to change regularly and the data is not consistent.)

aleith commented 1 year ago

No worries, @nc9 . Sorry I only just checked back here when I was reflecting on it in my head!

aleith commented 1 year ago

What me to?

Yes please. It would help us immensely. At the moment, we just wait for new DUIDs to appear in scada data or appear in the REL/GI lists (which is a manual process since they are difficult to process programmatically as the format seems to change regularly and the data is not consistent.)

I had a nightmare trying to reconcile facilities in emissions databases from different sources. no common codes between sources. no commonality of key details that could be used to discern which facilities were a match and which were not the same facility. So I can imagine what a PITA it is for you monitoring for new facilities.

nc9 commented 1 year ago

So I can imagine what a PITA it is for you monitoring for new facilities.

we're building a fully fresh solution right now that'll be live soon that helps with all of this :) - details to come, it'll be open to contributors

aleith commented 1 year ago

i stated writing an article for reneweconomy about the mismatch of emissions data between the CER safeguard mechanism reporting and SCADA /opennem data. it's seriously out of whack in places.but confusion around unpairable facilités dragged me down and i lost interest. i will renew my interest when this work is completed.

aleith commented 1 year ago

hey @nc9 do you have time for a phone call this week to remind me of OpenNEM issues around DUIDs, so when I call Dean I'm not wasting his time with too much hand waving vagueness?

I'll have to remind myself of the cases where I got unstuck with the SCADA data and count match facilities with OpenNEM data also.

Yes please. It would help us immensely. At the moment, we just wait for new DUIDs to appear in scada data or appear in the REL/GI lists (which is a manual process since they are difficult to process programmatically as the format seems to change regularly and the data is not consistent.)

nc9 commented 1 year ago

ey @nc9 do you have time for a phone call this week to remind me of OpenNEM issues around DUIDs

I sure do - email me on nik at opennem.org.au and I'll send you my number and details

edit: and thanks again for your help on this, it would really help us out if we could find a way to pull this off

aleith commented 1 year ago

i'll email now, looking to call you on Tuesday.