search

fediverse-devnet / feditest

A testing framework for distributed, heterogeneous systems communicating with complex protocols, such as the Fediverse
https://feditest.org/
MIT License
32 stars 6 forks source link

Assertion categories etc #166

Closed jernst closed 3 months ago

jernst commented 4 months ago

It's clear we need more nuance for our purposes than "test passed" or "test failed". Currently we have HardAssertionFailure and SoftAssertionFailure. But I think now we might want to distinguish more categories, along two dimensions:

  1. Dimension: what does the spec say

    • The spec says MUST and your code didn't.
    • The spec says SHOULD and your code didn't.
    • The spec was a little murky on the subject but we think it means this and your code didn't.
    • The spec is silent on the subject.

  2. Dimension: expected interop problems

    • Your code implements something that is known to / very likely to cause interoperability problems.
    • Your code implements something that may degrade interoperability.
    • What you implemented has likely no impact on interoperability.

It appears that all combinations of those two dimensions could appear. For example, the spec might say MUST but it has no impact on interop. Or the spec could be silent on something but in practice, unless you do X, interoperability is not going to happen.

In addition, we could have multiple specs that have a bearing on the subject that aren't agreeing with each other. We have this with HTTP status 422 which apparently didn't even exist at the time the WebFinger RFC was written.

On the reporting side it should be possible to filter in some fashion. For example, somebody under some circumstances may only care about problems with interop, not whether their code conforms to some spec. And vice versa.

jernst commented 4 months ago

So should we do this, @steve-bate? It could be something like changing hard_assert_that(...) and soft_assert_that(...) into assert_that(..., flags) where flags is a bitmap enum, so flags can be combined, but the assert_that could check that the provided value only combines values that may be combined (e.g. MUST + interop problems is allowed, but MUST + SHOULD is not allowed).

jernst commented 4 months ago

Related, may be subsumed into this one: https://github.com/fediverse-devnet/feditest-tests-fediverse/issues/99

jernst commented 3 months ago

Here are some possible beginnings: https://github.com/fediverse-devnet/feditest/compare/develop...jernst:feditest:pr-assertion-categories

Feedback appreciated before I go too far.

steve-bate commented 3 months ago

It seems that I can't comment directly on the branch in your repo.

Strangely, I apparently can create a PR from it (which I didn't). TIL

On Sat, Jun 8, 2024 at 2:11 AM Johannes Ernst @.***> wrote:

Here are some possible beginnings: develop...jernst:feditest:pr-assertion-categories https://github.com/fediverse-devnet/feditest/compare/develop...jernst:feditest:pr-assertion-categories

Feedback appreciated before I go too far.

— Reply to this email directly, view it on GitHub https://github.com/fediverse-devnet/feditest/issues/166#issuecomment-2155714230, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAGAYFCRB6XAJI5QSESLOPTZGJD3RAVCNFSM6AAAAABIUK6MWWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNJVG4YTIMRTGA . You are receiving this because you were mentioned.Message ID: @.***>

jernst commented 3 months ago

I've mostly done this, but I'm still working on how to show these categorizations. We want to keep it simple, but because we have two audiences -- standards people, and developers -- corresponding to having mostly an interest in the standards conformance vs the interop dimensions -- it's not obvious which gets the larger prominence graphically. I'm playing with making the cell background color reflect the interop dimension, and something like a fat left border color the standards conformance dimension.