Reporting: Soft vs Hard Failures and Errors

[steve-bate]

We're making progress on the error/failure and hard/soft distinctions, but I'm still a little confused. Sometimes unexpected HTTP statuses are "soft" and sometimes "hard"?

I suspect this might be an issue with the embedded WebFinger client doing its error checking instead of delegating that to the WebFinger test code. (?)

Test Matrix

Session Details

[jernst]

Basically the consequences of hard vs soft failures are that we are telling overworked developers to "fix this now" (hard) or "that's not right but is unlikely to cause problems so you can take your time" (soft).

I took a first stab at trying to make those distinctions, but I'm not certain my categories are correct.

Nope, the Imp does whatever it does, it's up to the test script to interpret what they Imp does and raise the correct exceptions for the purpose of test reporting.

[steve-bate]

Nope, the Imp does whatever it does, it's up to the test script to interpret what they Imp does and raise the correct exceptions for the purpose of test reporting.

Given the Imp is a test fixture, it feels like we'd want to implement it in a way that makes testing as effective as possible.

[jernst]

Actually the Imp is not a test fixture in the classical sense. It's a Node like Mastodon or any other, except that it has extremely limited functionality. Its role in the WebFinger tests is entirely optional and it is intended to be replaceable with any other implementation that implements the WebFingerClient interface -- such as a full Fediverse Node that has a richer API than the Mastodon client API -- either in-process like the Imp ("library") or as a separate process via gRPC.

[steve-bate]

Actually the Imp is not a test fixture in the classical sense.

We can debate this more later, but I currently don’t agree. The Imp not exactly a Node like any other. If you notice the perform_webfinger_query method has a different signature on Imp than it does on WebFingerClient and the tests depend on the Imp signature. That’s because the tests require some special controllability than most non-fixture clients will provide (like controlling redirect behavior, disabling JRD validation and/or content type checking, sending invalid requests, etc.).

Most or all the WebFinger servers we’re testing are also clients, but it would difficult to use them as clients to run the tests we have. Sure, we can probably indirectly cause them to execute a “sunny day” WebFinger query, but it would be difficult to cause them to send malformed requests, for example.

[jernst]

The different signature is simply a bug. There is even a comment in the Imp's implementation saying "override from WebFingerClient". Not sure how we ended up with the extra arguments either because I can't find that they are used anywhere.

[jernst]

I want to close this, because it's not quite actionable. If there are specific cells in the matrix that should be different, let's track those as separate issues.

[steve-bate]

I want to close this, because it's not quite actionable. If there are specific cells in the matrix that should be different, let's track those as separate issues.

I can't enter a specific issue for the HTTP Status codes assertions, but I think this is a more general problem than that even if this is the only current example (although I'm not sure it is).