Reimplement Content Negotiation

[pietercolpaert]

When a request is not acceptable, throw "406 Not Acceptable"

e.g. you can test this with, curl http://tdt/test -H "Accept: application/onlyformatacceptable,*/*;q=0.0" which should return a 406

Read Ruben's comment about serverside quality lists and multiplying them ↓

[dive-michiel]

Ok, as long as we keep the list short, it might be a better response code in the HTTP framework, but we have to think about the users/developers as well.

[pietercolpaert]

I agree :)

As you mentioned, we should keep as close to the HTTP protocol though. I've just scrolled through https://en.wikipedia.org/wiki/List_of_HTTP_status_codes and I have made 2 issues: this one and #120. So no more HTTP error codes after these.

[dive-michiel]

Done for definitions, content negotiation will just use the default formatter. https://github.com/tdt/core/commit/fc9e48e0bd20769d415396ad40b0d0003382b9ee

[pietercolpaert]

This is not the proper way if I understant correctly. Only if * is given in the accept header or if no accept header has been set, the default formatter should be used. When only a wrong formatter is set in the accept header, a 406 should be returned.

[coreation]

I tend to agree with using the default formatter, the reason for this is because something the well known M. P. always uses as an argument: be strict with what you send, be loose with what you receive. Meaning that you can have you rules about how stuff works, and the interfacing part must mirror this (e.g. when we make calls to another datatank, api,... we're strict with what we send conform the rules) but loose with what we accept e.g. a required parameter, but with a default value that we fill in for the user.

[pietercolpaert]

After looking this up more thoroughly, I came across this remark in the HTTP spec in favor of your approach:

      Note: HTTP/1.1 servers are allowed to return responses which are
      not acceptable according to the accept headers sent in the
      request. In some cases, this may even be preferable to sending a
      406 response. User agents are encouraged to inspect the headers of
      an incoming response to determine if it is acceptable.

/cc REST specialist @RubenVerborgh

[RubenVerborgh]

Stepping in for Mr. Fielding here, who will presumably be too busy.

First of all curl http://tdt/test -H "Accept: application/thisformatisnotacceptable" does not mean that text/html is not acceptable. The following does: curl http://tdt/test -H "Accept: application/thisformatisnotacceptable,*/*;q=0.0"

Second, as long as you specify the correct Content-type on the server response, the client has a means of discovering whether its preferences were met. A well-behaved client should check the content type anyway before parsing. That means the server can always send something that is not acceptable. (For instance, some proxy servers do not obey Accept headers.)

Third, if nothing is acceptable (*/*;q=0.0), you should reply with 406. The “some cases” quoted above do not apply to automated clients, which only have a limited set of media types they can interpret. However, sending a 406 is not an obligation (see "second" above).

[pietercolpaert]

So to conclude:

• we should implement a 406 when */*;q=0.0 is set
• for all other cases where we don't match the the client's accept header, we return the default one, as is implemented.

Correct?

Then this should lead to:

• [x] rolling back https://github.com/tdt/core/commit/fc9e48e0bd20769d415396ad40b0d0003382b9ee
• [x] implement 406 when */*;q=0.0

[RubenVerborgh]

Correct.

Actually, one possible implementation is that the server has its own preferences with q scores. The scores are multiplied with those of the client that match, and the highest scoring match is returned. If the highest score is 0, then it's 406.

[pietercolpaert]

Oh! That would indeed be a beautiful implementation! Just out of curiosity, do you know cases that benefit from this design decision?

[RubenVerborgh]

Well, you should see q really as a quality parameter, as in this example. So the server goes like: my HTML representation is most complete, but I also have an Atom which is a bit less nice.

[coreation]

Currently it's implemented as described in the two check boxes by @pietercolpaert

Provide a 406 when /;q=0.0 is given, with a list of possible format. These formats are fetched from a class that can identify which formats are applicable to the data (json-ld, ttl will show up in the list for semantic data for example.)

If no /;q=0.0 is given, yet a not supported format is provided json (or ttl for semantic data) will be put forwards as default.

@pietercolpaert if this suffices, then you can close this.

[pietercolpaert]

Ooh! Already a great improvement Jan!

What we were aiming at was nonetheless a bit more work: we should keep a list somewhere of the formats per resource that we as a server prefer. We give this a quality on our side. For instance, for a resource called http://demo.thedatatank.com/dresden/rivers I would put text/map+html as the 1st option, then kml, then geojson, then json. This ranking on serverside is also done with qualities in the same way as client-side. The qualities are then multiplied with each other: this is real "negotiation" in the strict sense of the word: the client has its preference and the server has its preference. Multiplying the Q scores and taking the q end result gives you the format to return.

[coreation]

I see, I thought it was a nice to have since it wasn't listed ;). I'll leave it for now ( so not for the march release ) to return a list as it does now. The "real" negotiation should then automatically lead to a 406, since the calculation will result in e.g. a non supported format, and all other formats with a q of 0 (since * / * indicates that all others are to be multiplied by 0?), ergo, a 406 will be thrown. Correct?

[pietercolpaert]

Correct!

[coreation]

Using this library for the negotiation logic.

[coreation]

Provided some extra tests, although the negotiation library is extensively used and tested, it didn't have some serializers we provide such as jsonld, turtle and ntriples. Provided the necessary changes and tests, as well as one where /;q=0.0 should return a 406 (tested as an httpexception).

[pietercolpaert]

Cool! Congrats!

Do we create a new issue for the feature where we give separate priorities to different formats according to what we are returning?

[coreation]

You mean we provide our own priorities in the response? At the moment the q=x.x is taken into account to calculate the best possible match with the request formats and our supported formats. If not, then you'll have to make a new issue yes ;)