Proposal header discussion

[travis-qualus]

According to the spec, the proposal header is a required element of all Forecast Proposals. But I have questions/concerns about the "default-emergency-durations" and "power-system-resources" as required elements.

"default-emergency-durations" - these need to be pre-defined emergency levels and durations between the RC and TOs. Most use LTE, STE, and (optionally) DAL levels, even if the durations themselves are different across markets. But they are defined by the RC. It makes no sense for the TO to propose their own durations, and much less sense to provide that again and again in every single proposal every hour.

"power-system-resources" - similarly, the RC will expect a given TO to provide ratings for a pre-defined set of TO resources, and to identify those resources with specific pre-defined "resource-id" identifiers (like the mrid). So why should the TO need to provide the complete dictionary of all their resources in every single Forecast Proposal, as well as all the various other aliases that might be used? It seems like if you want a way for the TO to validate their resource inventory and aliases, you could have another 'setup' get/set API to do that occasionally, but not every hour in every single proposal.

A compromise could be to make these optional.

[catkins-miso]

Hi @travis-qualus 👋🏻 thanks for opening this discussion.

Regarding emergency durations, there are a few use cases to consider here, as different Transmission Provider / Reliability Coordinators use different emergency durations and have different requirements. MISO, for example, will only require uniquely determined emergency rating with a minimum duration of XX minutes, though another rating tier "load shed" must be supplied with a minimum YY-minute duration which may have the same rating as the emergency rating.

Additionally, due to recent changes related to FAC-011 / SOL exceedances, MISO needs to know what the actual duration used by the TO when determining the rating. MISO will require that information at Modeling time from its TOs, but allowing it to be supplied in a TROLIE message provides the flexibility to address other practices. In particular I'm anticipating use cases in interop between RCs whose emergency durations differ and a TROLIE implementation needs to select the appropriate rating by duration.

Regarding power-system-resources in the header, this is discussed a bit in https://trolie.energy/decision-log/naming.html#consequences, but to be clear the client is only required to supply the pre-negotiated primary name (which is often an mrid as you noted). The anticipated use cases for allowing the client to send multiple names for a given power-system-resource is to aid testing and debugging efforts during the setup or changing of integrations. This could also be useful in RC-RC communications where neither party can compel the other to adopt a primary name.

Nonetheless, I think I agree with your proposal to make power-system-resources optional in the header. If I recall correctly, the primary rationale for requiring the resource-id for each of those entries was to support preprocessing, e.g., authorization, message integrity checks, routing, etc., but that's not strictly required for interop. So we can say a TROLIE-conformant server must be able to handle this header element, and a given implementation might require it as part of their own deployment to, say, a testing environment, but it would not be required for interop.

What do you think @getorymckeag?

[travis-qualus]

From a TO perspective, I think they've assumed THEY are responsible for reporting limits to each of their RCs using the calculation algorithms, durations, and identifiers required by each respective RC. I guess it wasn't anticipated that the RCs would share those submitted limits with other RCs, and therefore have to provide generic durations and all other identifiers for all other potential RCs.

Regardless, this is basically model information that is fairly static. So my concern was more whether model info needed to be provided EVERY hour, versus having a way to update model data periodically via this or another process.

I do understand the issue of perhaps wanting a 'correlation id' that could be returned in error messages for debugging. But the TO already needs its own internal translation table to map from their line id to your mrid (and other RC's desired resource id), so going backwards isn't that hard.

[catkins-miso]

guess it wasn't anticipated that the RCs would share those submitted limits with other RCs,

In the case of a jointly owned RC-to-RC intertie, the TOs only have an obligation to provide their ratings to their RC, but an overall limit needs to be determined from the most limiting of those TOs' ratings. Furthermore, most RCs monitor so-called "tier 1" facilities in adjacent RCs that are not on the seam; those ratings have to be obtained RC-to-RC as well.

The discussion here is about the header requiring the durations to be defined. I don't think we're likely to change that at this point, for the reasons already covered.

So far in this GitHub issue, I've agreed that it makes sense for the proposal-header.power-system-resources element to be optional, but I need @getorymckeag to weigh in.

[getorymckeag]

So far in this GitHub issue, I've agreed that it makes sense for the proposal-header.power-system-resources element to be optional, but I need @getorymckeag to weigh in.

Well... OK. I'm going to play some devils advocate here. Yes, they are "static". However, if they're optional, it seems more likely than not that they'll simply never get sent. Where I see the most likely use for these is on quarterly model updates when things change. We know TOs almost always use different names for their facilities than ISOs, and mapping errors will be common.

Also, we've discovered a perhaps unexpected performance benefit, especially for forecast proposals. When you're producing the message, yes its true obviously this is extra "stuff". However, when reading it, it turns out that the header is quite helpful. Given that the size of forecast proposals are potentially very large, we do not read them all into memory all at once. Rather, the objects are processed one resource at a time as they stream in.

In order to validate the proposals, we have to be able to map the resource IDs to real modeled resources in the database. Given a proposal for 2000 resources (this is a reasonable order of magnitude for some MISO TOs), if there are no resource IDs in the header, we have to execute 2000 database queries (or obviously we would have to resort to cache lookups) as we walk through the stream. This is made doubly complicated because we (granted, a GEV decision) support user-defined alias use in the API, so if we cache we would have to cache by each possible alias. However, today since all power system resources are in the header, we can front-load this- we do a single SQL query that resolves all the resources at once, and it's kind of a non-issue. I would imagine we're not the only TROLIE server implementer who will see this benefit.

[catkins-miso]

it seems more likely than not that they'll simply never get sent

Absolutely.

Given that the size of forecast proposals are potentially very large, we do not read them all into memory all at once. Rather, the objects are processed one resource at a time as they stream in.

FWIW, I think this would be a common implementation pattern even if the JSON payload was considerably more succinct.

since all power system resources are in the header, we can front-load this- we do a single SQL query that resolves all the resources at once

We definitely talked through some potential preprocessing use cases when introducing the header, including early auth, but it's good to hear about this concretely and be reminded of that convo.

I would imagine we're not the only TROLIE server implementer who will see this benefit.

Agreed, let's keep it required.

[travis-qualus]

I assumed you would process as a stream anyway, since the ratings section itself is already grouped by resource?

Or are you saying you validate all the mrids at once, and lookup your internal database resource ids, in one pre-processing step using the header, so that you don't have to look them up on the fly as you process the ratings? It seems like you could also just use the source provider information (very first object in the header) to retrieve all the mrids and internal resource ids belonging to the source TO?

Does your processing logic require that the ratings must be ordered in the same way as ordered in the header?

Does your validation logic require that ALL resources belonging to a TO must be provided in one proposal payload? Otherwise, will you report missing ratings proposals? Or just that it needs to match the header?

A 2000 resource ratings proposal file is going to be ~175MB!?!

[catkins-miso]

Hey @travis-qualus, let's consolidate #122 with this thread. One of your comments from there I'd like to bring into this thread is:

I don't know that we should rely on compression to keep these files to a reasonable size.

Using the current schema, a full MISO forecast, 22k facilities each with 240 forecasted periods, is nearly ~1.7GB, yet compresses in a few seconds to less than 40MB with gzip.

Clearly, the current schema needs compression to be viable, but that was an intentional trade-off. The rest of the comment is a strawman proposal meant to explore different trade-offs.

It leverages the header to define the layout of the forecast proposal. The goal is to keep some of the flexibility of the current schema but make it less verbose in the most common case. It assumes a few new implicit processing constraints:

• The rating set (continuous, long-term emergency, short-term emergency, etc.) are provided in order of decreasing duration.
• The ratings all use the units specified by a new header field limit-type.
• The forecasts are required to be in the same order the facility appears in the header.
• Each forecast must have the number of hourly forecasts corresponding to the new header field hours with the assumption that the begins header is the first entry and each subsequent entry represents the subsequent hour's forecast.

{
  "proposal-header": {
    "source": {
      "last-updated": "2025-10-31T15:05:43.044267100-07:00",
      "provider": "UTILITY-A",
      "origin-id": "5aeacb25-9b65-4738-8a00-ac10afa63640"
    },
    "begins": "2025-11-01T01:00:00-05:00",
    "limit-type": {
      "mva": null
    },
    "hours": 2,
    "default-emergency-durations": [
      {
        "name": "lte",
        "duration-minutes": 240
      },
      {
        "name": "ste",
        "duration-minutes": 30
      },
      {
        "name": "dal",
        "duration-minutes": 15
      }
    ],
    "power-system-resources": [
      {
        "resource-id": "8badf00d",
        "alternate-identifiers": [
          {
            "name": "segmentX",
            "authority": "TO-NERC-ID"
          }
        ]
      },
      {
        "resource-id": "f4d3d",
        "alternate-identifiers": [
          {
            "name": "segmentY",
            "authority": "TO-NERC-ID"
          }
        ]
      }
    ]
  },
  "proposals": [
    [ // 8badf00d
      [ // 2025-11-01T01:00:00-05:00
        120,
        160,
        165,
        170
      ],
      [ // 2025-11-01T02:00:00-05:00
        121,
        161,
        165,
        170
      ]
    ],
    [ // f4d3d
      [ // 2025-11-01T01:00:00-05:00
        80,
        110,
        120,
        120
      ],
      [ // 2025-11-01T02:00:00-05:00
        81,
        110,
        120,
        120
      ]
    ]
  ]
}