ISO queue Change log

[bendnorman]

Create a change log of ISO queue projects where each record is a project at a particular status. Projects will either have one or two records: one for when the project enters the queue and one for when the project is withdrawn or becomes operational.

With this change log, we should create a table that contains the number of projects that become operational, enter the queue and are withdrawn for a given geography and time frame. This table should allow users to understand queue changes for a given geography over time.

TODOs

• [x] Add documentation
• [x] Integrate withdrawn projects into SPP
• [x] Fix change log so operation and withdrawn projects are counted towards new projects on there queue_date.
• [x] Make sure all new capacities in change log add up to total capacity_mw in projects long
• [x] Why do NYISO and SPP have significantly less capacity in the changelog than the long format
• [x] Figure out why there are 200 more counties with project information.
• [x] Why are infra_petrochemicals_and_plastics_proposed_pm2_5_tonnes_per_yea and , n_tracts_risk_management_plan_proximity_low_income in the database but not in the parquet files / bigquery? n_tracts_risk_management_plan_proximity_low_income wasn't because it never existed in a dataframe but it existed in the metadata. I removed it from the metadata. The untruncated infra_petrochemicals_and_plastics_proposed_pm2_5_tonnes_per_yea**r** is in the dataframe but gets truncated when loaded to postgres but not parquet.
• [x] Apply nearly certain labeling to active projects only
• [x] Figure out how to fill missing queue_status values.
• [x] Label NYISO offshore wind projects
• [x] Figure out if deduplication method can apply to withdrawn and operational projects
• [x] Rerun comparisons to LBNL
• [x] Figure out how to fill missing queue status values in LBNL
• [x] Decide how to add ISO region to change log. Some counties belong to multiple ISOs.

Out of Scope tasks

• [ ] Look into CAISO resource/project sheets. Might be out of scope for this issue.
• [ ] Figure out why our deduplication method removes a lot of NYISO projects
• [ ] Enforce natural key on queue_id and entity
• [ ] Ask PJM and MISO about completed dates for operational projects
• [ ] Ask PJm about "Date Suspended" column. It exists in the data dictionary but not the actual data.
• [ ] There are about 1400 projects with Unknown resource type. We could probably figure out the best category for these projects if we do a little more digging.
• [ ] Improve county coverage of projects. When creating the change log I noticed at least 500 projects were missing county id information. A lot of them are from recent years which was surprising. Makes me think we are incorrectly parsing the raw county data.

Options

• Use one year of LBNL data. Use withdrawn and operational dates to construct a change log. This won’t really work because about a third of operational and withdrawn projects in LBNL are missing operational date and withdrawn date respectively. 90% of operational projects do have proposed. proposed date is a decent estimate at when the project was actually completed. Most projects real and proposed operation date are within a year of each other.
• Use multiple years of LBNL data.
• Use one year of GS data + one year of LBNL data
• Use multiple snapshots of GS data and multiple snapshots of LBNL data

Withdrawn and operational date availability

Withdrawn

Overall 61% of withdrawn projects have a withdrawn date.

• MISO: has withdrawn date in raw data but LBNL does not include the withdrawn projects.
• CAISO: 97 percent of withdrawn projects have a withdrawn date.
• ERCOT: 36% of withdrawn projects have a withdrawn date. However, the cancellation update tab in the raw data does contain withdrawn dates.
• ISONE: 85% of withdrawn projects have a date.
• MISO: 88% of withdrawn projects have a date
• NYISO: 64% of withdrawn projects have a date. It looks like in the raw NYISO data it's more like 90%
• PJM: 94%
• Southeast: 27%
• West: 13%

Operational

Only 52% of projects have actual operational dates. However, 89% of projects have a proposed operational date. Of projects with both proposed and actual operational dates, 70% of the projects' dates were within a year of each other. We could use proposed as guestimate for operational projects without actual operational dates.

ERCOT Integration

• I think Inactive projects are “Suspended projects”. About 100 inactive projects in the most recent report
• “Cancellation Update” are projects that are canceled by developer that month. Contains an asterisk though: Includes only those projects cancelled by the developer after it makes a Full Interconnection Study (FIS) request, or in the case of Small Generators, after the Resource Registration has been approved. Only projects that make full FIS requests are included in the large Gens tab: A table that list project details for Large Generators as defined in ERCOT Planning Guide Section 5.2.1(3); only include projects for which a Full Interconnection Study has been requested
• Commissioning Updates: A table listing project commissioning approval milestones met for the month: energization, synchronization, and commercial operations approval. For the rest of the ISOs we are only considering Commercial operation projects as “Operational”.
• I think to integrate we’ll need to combine all the historic reports. The most recent Large Gens tab, inactive tab and all of the cancelled tabs. Operational will be tricky. Assume projects that land in this tab become operational they might appear multiple times because the tab tracks Synchronization, Energization and Commercial operation approval. I guess we’d just combine the tabs from all months and grab the latest status for each projects.
• I’m hesitant to build all of this into gridstatus. Maybe we could adjust the get_raw_interconnection queue method to take a month year combo and return the excel sheet? Then we integrate the raw data? Probably grab a snapshot of all the months each time we run the archiver.

[bendnorman]

315 also integrates the latest GS data. To make a fair comparison between GS data and LBNL, we remove all GS projects that entered the queue after 2022 and select all active projects or projects that were active in LBNL. These are the results when comparing total MW of all regions:

I think our deduplication methodology might be a little too aggressive which explains the slight decrease in MW. Otherwise I'm pretty comfortable with these changes.

[jdangerx]

@bendnorman I see a bunch of interesting investigation/data quality tasks left. Do we have a way of quantifying when this data is "good enough?" If we finish those tasks in the TODO (make sure capacities add up, reconcile NYISO/SPP capacity diffs, etc.) does that mean we're done with this, or do you think we'll find more data quality issues and keep chasing those down?