Make missing values consistent in Generators and Scheduled Loads

UNSW-CEEM / NEMOSIS

NEMOSIS: NEM Open-source information service. A Python package for downloading historical data published by the Australian Energy Market Operator (AEMO)

Other

60 stars 31 forks source link

Make missing values consistent in Generators and Scheduled Loads #35

Closed mdavis-xyz closed 6 months ago

mdavis-xyz commented 6 months ago

In the 'Generators and Scheduled Loads' table, columns such as Fuel Source - Primary have '', '-' and nan values. Do they mean something different?

Note that these different values appear to come from the raw excel file:

fuel-excel

If these values mean the same thing, I think this library should coerce them into the same nan value. (Even if it's AEMO's 'fault')

nick-gorman commented 6 months ago

Yep that's a good point, I'll make this change in the next release.

mdavis-xyz commented 6 months ago

Also, do you know why BassLink is listed here as a generator? No other interconnector is there (as far as I can see). (Again, the nemosis behavior just matches the underlying AEMO data. I'm hoping you can explain AEMO's data to me.)

(It's also classified as non-scheduled. Aren't DC links fully dispatchable, and thus scheduled?)

from nemosis import defaults, static_table

raw_data_cache = "C:\\Users\\DAVIS_M\\Documents\\nem\\data-cache"
df = static_table('Generators and Scheduled Loads', raw_data_cache)
df[df['Participant'].str.contains('Basslink')]

mdavis-xyz commented 6 months ago

The more I look into this data, the more confused I get.

Some batteries are only listed as one DUID, with an empty/- fuel souce. Others are listed as two DUIDs (one for load, one for generation), with Grid as the source. Except for Hornsdale, which appears with Wind as the fuel source for generation, and - as the fuel source for load.

I'm a bit lost now. How do people normally use this data? I want to categorise grid volume into % renewables, coal, gas etc. I knew handling batteries would be tricky, but I didn't expect the raw static data to be this messy.

fuel-type

nick-gorman commented 6 months ago

Hey Matt,

This data is definitely not great, and I've even found a few technology-type errors in it before, but it appears to be the only source for some info. A few comments that may be helpful:

The Basslink interconnector is probably in there because it's the only Market interconnector. That is a good point about being non-scheduled; I agree that it doesn't make sense.
As far as I know, at the moment, all batteries are registered as both a generation unit and a load unit; I think this has just been AEMO's way of integrating them into the dispatch process. Having said that, I don't think it necessarily applies to batteries that are co-located with other generators. See this doc: https://aemo.com.au/-/media/files/electricity/nem/participant_information/new-participants/registering-a-battery-system-in-the-nem.pdf?la=en

nick-gorman commented 6 months ago

Addressed in release 3.6.0: https://github.com/UNSW-CEEM/NEMOSIS/releases/tag/v3.6.0