ISO queue month-to-month differences

[TrentonBush]

Interconnection Queues Over Time

Change over what time?

The minimal version is to compare two snapshots, treating the earlier one as a baseline for comparing with the later one. The maximal version is to integrate many snapshots to build up longitudinal time series at higher resolution. Scoping out the maximal version is a separate work item #300.

To start, let's do the minimal version using a time step of 3 months. That should be long enough to see change but not so long as to drown in it.

What changes are we looking for?

The interconnection queues describe proposed projects and their progress as they seek approval to begin construction. Projects are described mostly by fuel type, location, and capacity. Approval status is described differently by each of the 7 ISOs, but users have simplified this into a 4-step taxonomy of "not started", "actionable", "nearly certain", and "completed or withdrawn". Those are the primary dimensions of interest.

Some of our users intervene in regulatory proceedings for specific projects or jurisdictions. The holy grail is to find evidence that these interventions caused positive improvements to occur. I doubt it is possible to find such conclusive evidence, but one goal of this analysis is to create the descriptive metrics that, if strong enough, could support such claims.

Another goal is to simply build a monitoring system for the interconnection queues so users can be alert to any noteworthy changes: What fuels are being proposed more/less than in the past? How many projects have moved to the next stage of approval? How many new projects are there? How much capacity and of which fuel type has been withdrawn? What regions see the most activity? Etc.

Scope

Our role here is to build the dataset that backs a reporting system, not to make the final visuals. Our visualizations/tables/etc are for prototyping and exploration -- they don't need to be fully polished.

Practical questions:

• are the provided IDs consistent across time or do we need to do record linkage? (queue_id, iso) is unique for most ISOs, but iirc in ISONE that only works for active projects, not withdrawn. Is that a problem?
• Currently, GridStatus is not consistent across ISOs about integrating active, withdrawn, and operational projects -- it varies by ISO. But all that data is available for every ISO and we can get it if we need it. Do we need it? What questions could we answer with that data that we cannot answer now?

[e-belfer]

Task list:

• Is it possible to track a single project through time consistently through one ISO dataset?
• Where has been the most activity and where have projects progressed the most? What kinds of projects have those been?
• Looking at percent changes and mapping activity by county chloropleth to make visualizing changes either

[bendnorman]

Being able to compare different versions of data would also be helpful because users like understanding what changed when we update a dataset. I have a couple ideas how we can implement this:

• Create a script that takes snapshots of bigquery tables each time we update the data. This isn't a flexible solution if we every change storage backends and I don't think it's really the intended use of the feature. I think it's more for data recovery and backup, not versioning.
• I think a better solution would be to write the tables from postgres to a directory of parquet files where each directory is a tagged version, similar to how we release PUDL data. The most recent tagged version and "dev" parquet files could then be loaded into bigquery. This would be convenient for doing ad hoc data change analyses and incorporating previous version data into future versions for example if we wanted a column "MW % change in the last quarter" in county_wide.