interuss / monitoring

InterUSS Platform USS monitoring tools for federated UTM, including automated testing.
Apache License 2.0
11 stars 19 forks source link

Change reusable test steps to reusable test step fragments #380

Closed BenjaminPelletier closed 7 months ago

BenjaminPelletier commented 10 months ago

Is your feature request related to a problem? Please describe. Because of the clever way we wrap a flight planning attempt in "before" and "after" checks of the DSS to see if an operational intent was added, our test reports have the extremely odd characteristic of putting the planning query in step X while the "before" DSS query, which occurs chronologically before the planning query, is put in step X+1 (validation step). I would previously have expected to be confident that the earliest event (query, check, note) in step X+1 should always occur after the latest event in step X.

Describe the solution you'd like I believe the reason this occurs is that we really want to compose multiple reusable fragments into a single test step, but we can't currently do that because a test step can only link to a single "reusable test step" by linking that reusable test step in the heading title of the test step. To support composition of multiple reusable fragments into a single test step (e.g., "Check 'before' operational intents", "Plan flight", "Check 'after' operational intents"), I think we could:

I'm open to ideas for how to signal that we're linking to a reusable test step fragment; the best solution will make what is happening obvious to a reader unfamiliar with the technical backing, yet still be unambiguous and not burdensome to test designers. My tentative thought is to require each fragment link to be another one-level-deeper heading under the test step heading. I know this will put us even deeper into heading levels (we're already pretty deep), but I think it makes the most intuitive sense, and we can add non-linked content under same-level headings.

Describe alternatives you've considered I'm open to alternative suggestions for how to make sequential test steps actually be sequential chronologically (or an argument that this is not a problem).

Additional context See the time stamps for the queries in "Plan" and "Validate" sequential steps e.g., here -- note that the first of the two queries in the "Validate" step is initiated before the query in the (previous) "Plan" step.

mickmis commented 10 months ago

Agreed this is an issue, and agreed with the proposed solution. I think that would solve the different issues with no real downside.

we really want to compose multiple reusable fragments into a single test step, but we can't currently do that because a test step can only link to a single "reusable test step" by linking that reusable test step in the heading title of the test step.

Yes, I can confirm that has happened to me several times when implementing tests.

My tentative thought is to require each fragment link to be another one-level-deeper heading under the test step heading.

Not sure if this is what you had in mind, but I think it would be OK for test step fragments and checks not part of the fragment to be under the same level:


## Case 1 test case

### Step 1 test step

#### [Fragment 1 test step fragment](./fragment_1.md)

#### [Fragment 2 test step fragment](./fragment_2.md)

#### Check 1 check

That would allow some welcomed flexibility for development all the while being not too confusing for non-technical readers. And also probably retro-compatible with our current tests.

BenjaminPelletier commented 10 months ago

Yeah, that's exactly what I had in mind. Sounds good -- I'll see about implementing this.

BenjaminPelletier commented 7 months ago

This is complete with the exception of:

Remove all links from test step titles, and replace them with text in the body of the test step linking to the same Markdown file (now describing the fragment)

Instead, a test step title can still link to a fragment, and this is appropriate if that is the only fragment being used by the test step and no additional explanation is needed in documentation.