Mapping problem details into a header?

[mnot]

Should we define a header that can carry problem details?

E.g., for the first example in the spec:

{
 "type": "https://example.com/probs/out-of-credit",
 "title": "You do not have enough credit.",
 "detail": "Your current balance is 30, but that costs 50.",
 "instance": "/account/12345/msgs/abc",
 "balance": 30,
 "accounts": ["/account/12345",
              "/account/67890"]
}

a workable mapping might be to use Structured Fields to represent the top-level members of the object, like this:

Problem-Details: type="https://example.com/probs/out-of-credit", 
          title="You do not have enough credit.", 
          detail="Your current balance is 30, but that costs 50.", 
          instance="/account/12345/msgs/abc", 
          balance=30

That drops any extensions that are not simple values, but I think that's a reasonable tradeoff...

Would this be useful?

[dret]

Interesting idea! Asking for some community feedback here: https://twitter.com/dret/status/1490617958305042433

[asbjornu]

I like the idea. Would it be a problem (🥁) to name the header just Problem?

[reschke]

Unfortunately, structured fields do not work well with prose if it uses non-ASCII characters...

[dret]

"not well" or "not at all", @reschke? https://datatracker.ietf.org/doc/html/rfc8941#section-4.1.6 sounds a bit like the latter unless we'd go through the effort of defining a specific mapping for non-ASCII characters. it would be an option to restrict values to ASCII for the Problem field, but that would not be great in terms of i18n support.

[reschke]

"not well" in that we would need to allow byte sequences (https://www.rfc-editor.org/rfc/rfc8941.html#name-byte-sequences) in addition to strings.

[vasilakisfil]

Can someone clarify what we are trying to solve with this suggestion?

[mnot]

In the past, some have expressed a desire to convey problem details outside the body of the response, for whatever reason. This would facilitate that, in a limited fashion.

[melvincarvalho]

Could the example in the first post, be used together with HTTP 402?

[darrelmiller]

This seems like it would be a potential solution to the the topic of "warnings" that keeps coming back. I am assuming that returning Problem-Details header with a 2XX status code would be allowed. Although, I am a little terrified about the potential for abuse with that.

[dret]

ping @andrecedik

[sdatspun2]

In the past, some have expressed a desire to convey problem details outside the body of the response, for whatever reason. This would facilitate that, in a limited fashion.

I am curious about the use cases where error would be sent in header and rest of the goods would be sent in response body or in some cases errors are sent in response body and sometimes they are not. I wonder how the developer experience would be in consuming such an API.

This could be explored to convey warnings though, I agree with @darrelmiller.

[andrecedik]

Thank you for the ping @dret

I agree with @sdatspun2

This could be explored to convey warnings though

Yes, it is basically what we were trying to achieve with our I-D "Communicating Warning Information in HTTP APIs". Although in the I-D we named the header Content-Warning (but that's "just naming things") - and we were trying to return the warning information in the body. Which brings us to the matter of developer experience, that @sdatspun2 was writing about.

TBH, it still feels weird to have problem details in the body and warnings in the header. Shouldn't both be in the body or both in a header? But maybe that is just my own stubbornness.

Anyway - it's very confusing when you can return data in either the body or in a header or - god forbid - in both at the same time. Who knows maybe someone will even put different content in each. Which is probably what @darrelmiller was thinking about when he's thinking about abuse.

...

I've been thinking about this the whole day and I've come to the conclusion that there is probably only one solution.

Problem Details in the body. Warning Details in a header This way problem details are being transmitted the same way the original RFC proposed. There is no confusion, because problem details can be in the body or not. If there is a warning it can be transmitted as a header, thus keeping the integrity of the original RFC.

There's probably one caveat. In my mind there's always the use case where multiple warnings are being returned. This could lead to problems when the value gets too big for a server to handle.

Which is why I'd love to have the warnings in the body. One can simply return more data in the body. Also, having one "severity level of information" in the body and another one on a header feels weird to me, maybe this is a compromise I'll have to make.

All the other options don't seem feasible, due to the side effects stated above.

---

@mnot do you have insights into the use cases for having problem details in the header? The only reason that comes to my mind is when a "204 - No Content" response is being returned. Then it wouldn't seem right to have problem details being returned as the only content.

[dret]

liking @andrecedik's reasoning here which could mean for RFC7807bis to potentially subsume the https://datatracker.ietf.org/doc/html/draft-cedik-http-warning-02 draft (probably renaming the header field but using the same general idea). or we could stay the course, stick with RFC 7807's model of content-only, and possibly revive and better align the http-warning draft, potentially even adopting the Problem name suggestion made by @asbjornu. i kind of like the second option better so that RFC7807bis stays lean and mean without adding too many new things.

[mnot]

If we think that this is going to be done / get used, we should probably do it now, to avoid the overhead of opening another spec.

That said, I don't see a strong response -- I'm happy to leave this floating as an idea that we might get to one day if demand eventuates.

[awwright]

What if I want to transmit multiple problems—for example, what if I'm operating a gateway and I want to add a problem header without changing the semantics of an existing upstream one? Shouldn't the fields be delimited with a semicolon ;?

[asbjornu]

@andrecedik,

TBH, it still feels weird to have problem details in the body and warnings in the header. Shouldn't both be in the body or both in a header? But maybe that is just my own stubbornness.

I think warnings and errors are distinctly different and thus should be treated differently. In most programming languages, errors are conveyed by having their own keywords (throw or raise) and types (Exception or Error), while warnings are usually delegated to logs. In the terminal, stderr is separate from stdout and while you'd expect a CLI application to continue executing through the occurrence of warnings, you'd expect a non-zero exit if it were to encounter an error.

For this reason, I think the treatment of errors and warnings warrants a different behavior in HTTP as well. Delegating warnings to a header; allowing the app to continue working – while allowing errors to take over the entire response and thus stopping the application from working – feels both natural and right to me.

What I do think warnings and errors share, though, is structure; which is why I'm supportive of a suggested Problem header conveying the same data and metadata as that of RFC 7807. And as @awwright writes, a Problem header is even able to convey multiple problems in a simpler and more ergonomic way than what is possible in the HTTP body, by separating each problem occurrence (i.e. warning) with a comma (with each field within a problem occurrence separated by a semicolon).

While I don't think it's very important to bake the Problem header into the same spec as rfc7807bis, I do see the benefit of maintaining them in synchrony to avoid feature drift and (worst case) incompatibilities. So if was to come down to a vote, I would give a +1 to roll the Problem header into rfc7807bis.

[mnot]

@awwright the problem object can only serialise one problem, I think the same would apply here.

[andrecedik]

Hey @asbjornu

I'd like to challenge you a bit on your statement. Sure, at first it seems that errors and warnings are handled differently since they are separated from each other by different "logfiles". But the actual process of logging the error or warning is the same. You send an (error or warning) string to a file.

But I can relate to what you're trying to say. What you are logging can be totally different. Most often errors come with a stack trace, giving a more detailed look into the "state of the machine" at the time. Thus when looking at an error file you'd expect different types of log entries, then in a file that warnings have been written to.

Anyhow, I think we're both trying to achieve the same thing here. As I wrote earlier, I'd be happy to have warnings in a header field. Unfortunately, the challenge I see is still there, when returning warnings one would probably want to return more data, than with an error, thus resulting in messages that could get too large.

If this is something everyone thinks is negligible then let's go for the header field.

[awwright]

@awwright the problem object can only serialise one problem, I think the same would apply here.

I would expect if one problem document can serialize one problem, then one Problem header would serialize one problem. It's not obvious to me that this would "bubble up" to the entire HTTP message.

Also I would expect HTTP semantics to apply, if I append a Problem header, that it won't change the meaning of existing headers (or at least, I would expect this to be a detectable error or ambiguity).

[olijaun]

I just came across this discussion by incident. I'm a bit late for the party... so please ignore me if you think the following makes no sense:

I don't really see the point in having problems/warnings in a header field. Isn't a problem a "business problem"? Something like "out-of-credit". It is not some meta data but it is something essential that is relevant to "the business". The same is the case for warnings. To me this is an essential part of the response, so why would I want to add this to a Header-Field? It should be part of the body oft the response. It is still possible to define some kind of standard structure for this kind of stuff and then you embed that into your response (as draft-cedik-http-warning-02 suggests).

[awwright]

I'm not sure if there's a specific use defined, but I would use it similar to 4xx errors. If there's no specific 4xx code for the error, this enables you to use a Problem header field with a 400 Client Error to make your specific class of error more machine-readable. Whereas normally, 400 requires human intervention because it could mean any error.

[dret]

On 2022-04-11 20:36, Austin Wright wrote:

I'm not sure if there's a specific use defined, but I think you can see it similar to 4xx errors. If there's no specific 4xx code for the error, this enables you to use a Problem header field with a 400 Client Error to make your specific class of error more machine-readable. Whereas normally, 400 requires human intervention because it could mean any error.

i agree in principle, but i'd argue that you could make the specific error equally machine-readable by using a body with a problem report.

where a header may be useful is when you want your API to generate human-readable error responses (in HTML) but still want a machine readable version for machine clients. there would be other patterns to accomplish the same goal, but i think that's a line of reasoning you could follow.

[olijaun]

I don't really see why one would need both things at the same time (but maybe my view is too limited).

Either you request a HTML page and then you also expect a HTML response in case of an error. OR you add "application/problems+json" in addition to some "application/myapp+json" type to the accept header and you get the machine-readable response-

Assuming your request a HTML file and it is returned with the header in question: How would you access these header fields? If you load a page with a browser you can't afterwards access the HTTP-Headers of this page in Javascript (I had too look that up here: https://stackoverflow.com/questions/220231/accessing-the-web-pages-http-headers-in-javascript) .

Or you load the page with javascript but then why would you request a HTML? It would be possible of course but why?

Of course: If browsers started to support this header field natively then maybe...

[awwright]

@olijaun I would expect a Problem header along with an HTML document for the same reason I would expect a server to respond with a 4xx code (instead of a generic 400 one). It provides more information about the nature of the error to the user agent.

That is, the Problem header is probably intended for use by the user agent itself—not the webpage. (Although I agree, a webpage should be able to access the response headers that it was delivered with, except possibly Set-Cookie.)

[mnot]

Coming back to this, it seems like there's some support for adding this, but it's not especially strong. I can't see too many risks of including it in the spec (obvious worst case: it doesn't get used). That's not enough to justify including it, of course.

@richsalz @darrelmiller perhaps a consensus call on list is in order?

[richsalz]

Sure, a consensus call is fine. But could you write a summary of this issue and post to the list?

[benbucksch]

I like it a lot. Error messages are a serious problem point in most HTTP based APIs. Everybody rolls their own error spec, with slightly different semantics. It's good to have them specced consistently and sanely.

The semantics used in the spec are very sound. There's:

• a unique computer-readable ID (type),
• a human-readable and translated description of the generic problem,
• a specific human-readable and translated description of the specific problem with parameters, and
• custom computer-readable parameters for the values of this specific error. This is great. I also love that you say that "detail" shall not be parsed, but custom values ("extensions") be used.

I would just improve 2 aspects:

• Add "debug" and "stack" as explicit additional fields. It's great that you specifically mandate that "detail" shall not contain debugging info. That's wise to define, because a field called "detail" is bound to be used for that. Unfortunately, most devs do not read specs, so if you have a field called "detail", I'm certain that devs are going to put stack traces in there. To avoid that urge, I would recommend to add two explicit fields "debug" and "stack". Both optional of course. But they should be two separate fields, because the stack is well-defined, and "debug" can be a dump of internal variables or anything else useful for debugging.

• Be more tight in the definitions. Please mandate that the title and detail fields are in fact translated on the server side. The client cannot realistically do it, because the server can add new error conditions at any time, plus the business of including parameters in the user message. It's important that there is clarity about who is responsible for the translation of the error messages, and that this aspect is not left vague. So, I would define clearly: If there is a translation happening at all, then the server is translating the human-readable message, not the client.

Other than these two simple changes: I like the spec a lot, and it's very much needed.

Thanks for your push for it!