Explore gridstatus iso queues

[bendnorman]

Our ISO data is curated by LBNL and is released once a year. Near real-time ISO queue data could be useful for downstream analysis. gridstatus pulls ISO queue data from the same original sources as LBNL but it is updated as the ISOs update their spreadsheets. gridstatus also attempts to align the column names for different ISOs.

Pros

• Our downstream analyses are unlikely to change dramatically because the number of projects and capacity distributions of grid status and LBNL are very similar. See the final section of the notebook for more information.
• Offers monthly and, in some cases, daily updates to ISO queue information. LBNL data for 2022 isn't available until a few months into 2023.
• We can improve ISO queue data for all users of gridstatus as we continue to work with the data.

Cons

• Based on previous conversations with LBNL it sounds like they are committed to updating the data each year. A team in a national laboratory is probably more reliable than a new open-source project with only a couple of contributors.
• Some columns like Status and Fuel Type will have to be harmonized.
• We'd probably need to archive the outputs from gridstatus because the raw data gridstatus is pulling is likely to change or disappear.
• LBNL's report also provides ISO queue data from the 30 largest non ISO utilities, which gridstatus does not provide. We'd have to join the LBNL utility and gridstatus ISO queue data together.
• My initial pass through the data didn't many duplicates but there are probably some in there! We'd need to spend some time doing a deep dive into potential duplicates.

None of these cons are blocking issues. If we care about real-time ISO queue data, gridstatus is a solid option. To move forward with gridstatus we should understand the downstream consequences of having delayed ISO queue data. Maybe there aren't any consequences? We already know the best places for wind and solar in the US, so a year delay on proposed projects won't change counties to focus on.

[TrentonBush]

Nice! It doesn't look too different to integrate than the NREL version. A couple of extra seemingly minor category harmonizations on top of the main challenge of deduplication.

My take on the different challenges/benefits:

• we still need NREL for the non-ISO regions, but merging them looks easy because it's easy to filter for non-ISO data. Future analysis would have to keep this difference in mind though, or we risk underestimating non-ISO proposals
• Even if gridstatus disappeared in 3 months, we could revert back to NREL.
• We would need to at least partially automate the QC and deduplication if we do frequent updates
• Do frequent updates mean we shouldn't store the data directly in git? We could drop all withdrawn projects before saving if we want to save disk space