search

fediverse-devnet / feditest-tests-fediverse

The tests for the fediverse testsuite
MIT License
5 stars 4 forks source link

WebFinger: minimize test result "shadowing" from unrelated issues #58

Closed steve-bate closed 1 month ago

steve-bate commented 1 month ago

This is a legitimate failure per the RFC, but it doesn't cause any serious federation issues AFAIK. The incorrect content type causes almost all the WebFinger tests to fail (shadows other more specific behavior). I don't have a suggested action. I'm just recording the observation. We may want a strict content type test and then be more forgiving for the other tests that are not focused on content type (although we should probably at least require application/json).

jernst commented 1 month ago

I think we need some other reporting category like soft-fail per https://github.com/fediverse-devnet/feditest/issues/83 -- not reporting it is bad, but uncompromisingly failing is bad, too.

steve-bate commented 1 month ago

I think we need some other reporting category like soft-fail

I don't think this is a soft-fail issue. I believe that tests results are easier to understand (and more generally useful) if they are related to the functionality being tested in that specific test. If a test is looking for a CORS header, for example, failing on an application/json instead of application/jrd+json content-type doesn't seem useful or relevant. We have a separate content-type test that will report the relevant failure.

jernst commented 1 month ago

So what exactly are you proposing? To add a test? If so, please do!

steve-bate commented 1 month ago

I think the issue I saw was caused by a merge-related regression (conditional content-type check in webfinger client) that's been fixed in my last set of PRs.