search

fediverse-devnet / feditest-tests-fediverse

The tests for the fediverse testsuite
MIT License
6 stars 4 forks source link

Comprehensible error messages #4

Closed jernst closed 4 days ago

jernst commented 12 months ago

From Ben Pate at https://mastodon.social/@benpate/111163007260414906:

Just this: there are tons of ways that an API call can fail, but most Fediverse software just returns something unhelpful, like 500 Internal Server Error.

For production software, that’s probably reasonable, but a well-written test suite should help people debug, not just tell them about the bug.

So if I have a voice, a test suite should really return errors like: “this message was not accepted because it is missing a ‘name’ field.”

Also comments from Mike Macgirvin in the same thread at https://fediversity.site/item/645bc210-0913-4b77-a395-1b9f4fca9fb9:

Nomad returns delivery reports indicating the disposition of a message once it hits the server and let's normal people examine the reports for any message they send. We needed this years ago because Diaspora silently dropped 1/3 of all communications and we needed to be able to prove the issue wasn't our software's fault. The assumption was that site operators of that period were often high school kids that couldn't read or interpret log files, so this let us find federation issues without involving them.

It would be no small feat getting this kind of thing standardised and adopted in ActivityPub, but we do provide feedback of the http error code for all sites -- and if something is stuck in the queue, one can see the results of the individual delivery attempts. Nomad sites provide much more detailed answers of what happened - lack of permission, filter/blocking rule, duplicate, recipient not found, delivered, etc...

and

In a nutshell, we just return an array of delivery reports in addition to an http status. For sites that don't support delivery reports, we manufacture a report holding the http status, and if there are queue retrys, we collect the associated curl logs into a linked queue summary.

https://codeberg.org/streams/streams/src/branch/dev/spec/Nomad/Messages.md

jernst commented 12 months ago

See also https://socialhub.activitypub.rocks/t/report-errors-in-server-processing/3006/11

snarfed commented 9 months ago

More discussion on this at https://fed.brid.gy/docs#error-handling , esp on sync vs async error handling.

snarfed commented 6 months ago

(...also I'd argue that returning 500s in production happens sometimes, but isn't reasonable. 500 means a bug or other problem on the server side. Any other 5xx though, or 4xx, sure, especially with a useful error message!)

jernst commented 4 days ago

I'm going to close this in favor of https://github.com/fediverse-devnet/feditest/issues/357, because I think that's about all we can do with FediTest.