How to manage and partition testing across distinct Translator deployment environments?

Some questions exist about scoping of tests within the SRI Testing harness:

Against which x-maturity environment should the SRI Testing harness be run?
Would distinct subsets of validation (test edges) be more important in one environment than another? How would such distinct test data be indexed and used?
Would we avoid validation runs in production (staging would probably catch things, right?)
How do we manage our test endpoint specification over these environments?

To provide a context to these questions, we note that the Registry 'servers' block has entries with the following general format:

{
   "servers":[
      {
         "description":"Default server",
         "url":"https://automat.test.transltr.io/robokopkg/1.3",
         "x-location":"ITRB",
         "x-maturity":"testing"
      }
   ]
}

Translator defines four x-maturity deployment environments: production, staging, testing and development.

There are a couple of primary issues to ponder here:

1. Specifying testing by `x-maturity` environment

The SRI Testing framework was originally quite agnostic as to x-maturity and would take the first available server endpoint as the target for testing. This seemed ok since anecdotal observation of Translator SmartAPI Registry ("Registry") entries seemed to indicate that many entries with an info.x-trapi.test_data_location only had one x-maturity entry in their servers block. However, those server block entries appear to target any of the four x-maturity environments - there was no one environment consistently specified with respect to info.x-trapi.test_data_location annotated Registry entries.

Resolution

Fortunately, this challenge seems partially resolved by the newly extended Translator Extensions data model for the info.x-trapi.test_data_location property. Now, test data locations may be associated with a specific x-maturity (or simply set as the 'default' for all environments not specified).

The latest implementation (1.1.1 release of 14-Nov-2022) will consume values from the new info.x-trapi.test_data_location but is get to reflect the testing nuance implied in the new specification.

One practical issue which may remain (related to item 2 below) is when a default tagged entry is provided for the info.x-trapi.test_data_location JSON object specification, and the entry has multiple x-maturity environment endpoints defined in the Registry servers block. By definition, all such x-maturity endpoints would be targeted for testing; however, although convenient, that may not always be the intended outcome.

2. Specifying `x-maturity` endpoint for testing (and any redundancy thereof)

This works well enough in most situations, except that some Translator SmartAPI Registry entries appear to have multiple server endpoints under one or more of their x-maturity values.

In such cases, how would the SRI Testing framework know which endpoint should be used for testing (or, should all of them be tested... that seems like a source of unnecessary redundancy and overloading in the testing framework)?

Possible Resolution

Two conceivable workarounds here might be:

Formally exclude x-maturity environments with servers block values with multiple endpoints from testing, or (you've been warned!) just "choose one" based on some dumb heuristic but issue a log warning...
To add an optional second argument target (or equivalent key) to the info.x-trapi.test_data_location JSON object specification (which currently only has the JSON key url (BTW, an unfortunate choice given our problem here...). The value of the 'target' key would be a single string URL or (perhaps) list of URLs associated with the specific x-maturity server endpoint to be tested. This is, of course, backpedalling a bit from the idea of simply reusing the servers block, but in fact, if the servers block only has one endpoint recorded for the x-maturity that has test data, then it would be used if a target key-value is not provided.

TranslatorSRI / SRI_testing