search

fediverse-devnet / feditest

A testing framework for distributed, heterogeneous systems communicating with complex protocols, such as the Fediverse
https://feditest.org/
MIT License
27 stars 4 forks source link

Reporting: Soft vs Hard Failures and Errors for HTTP Status Codes #160

Closed steve-bate closed 4 weeks ago

steve-bate commented 1 month ago

See #145 for details. That issue is about a general problem with assertion exceptions for the same type of problem being raised from different sources and treated as different categories (hard/soft failure, error). This issue is for the specific HTTP status code case (which is described in #145).

jernst commented 1 month ago

I've been giving this topic some thought beyond HTTP Status Codes. #166

jernst commented 1 month ago

I think this is a duplicate by now, lots of related issues, let's consolidate in #166.

steve-bate commented 1 month ago

I'm going to reopen this one because I think it might be a subtly different topic than #166. I interpret issue #166 as discussing the set of categories that can be assigned to failures. The issue here (and I can try to rephrase it and/or create a new issue if that helps) is that given the currently-supported categorization, how we assign those categories to specific failures is not clear to me.

In general, one possibility is to only label a failure as a "hard failure" if there is a clear Fediverse interop issue. I think we may have very few, if any, of those since all these implementations appear to be currently interoperable for fediverse purposes. I know you are thinking of treating hard/soft failure and interop risk as orthogonal, but the pragmatic benefits of that approach isn't clear to me at the moment.

(Note: The original issue was more general than this specific HTTP status code issue. You had asked me to narrow it down to a specific type of error category, but I still think it's a more general topic.)

jernst commented 1 month ago

What we need is a proposal to addresses the issues. If we did #166 as proposed, what is missing?

steve-bate commented 1 month ago

What we need is a proposal to addresses the issues. If we did #166 as proposed, what is missing?

I made one proposal above. ;-) Again, I see #166 as defining the dimensions/categories of failures. Some of those categories are more objective than others. There's still plenty of opportunity to inconsistently apply the more subjective categorizations, but I can't identify specific issues until I see the resulting report.

Playing the Fedi developer role, I primarily care about interop problems. Dimension 1 isn't very interesting to me if it doesn't result in interop problems. I'm not sure how Dimension 2 maps to hard/soft failures (if at all) so I'll withhold comment on what may be missing until I see a report with the revised categories/dimensions.

This may be a non-issue at that point and we can close it.