Support Range: in cowboy_rest

[dvv]

Hi!

Is the subj planned?

TIA, --Vladimir

[dvv]

Where is the right place to put restrictions on the range of served data?

[dvv]

Please, someone, review https://github.com/dvv/cowboy/commit/0bf71f319b44d7b00133518d356e0f83a2309547 -- will this be welcome in the core? How to pass and honor Offset/Length in content_function in cosher way? tia

[essen]

As-is no, for many reasons including adding a cowboy_util module when we have cowboy_http for all parsing needs.

But I didn't think about Ranges much yet, although we do want to support them, so it'll take a few weeks before I do a proposal including that.

[dvv]

i see. it was my first experience in Erlang.

Range: is the cleanest way to specify portions of data w/o abusing querystring, Content-Range: is well-suited for consistent client-side paging.

nothing pressing.

[essen]

It's not to be used for paging though, the Range value is in bytes and represents the byte range from the entity body returned. It's mostly used for resuming downloads.

[dvv]

2616 defines only "bytes" item, yes. but below is quite valid case i learned from Pintura and it fits REST (thus cowboy_rest) very well:

GET /Product/?type=shoe
Range: items=10-19

Content-Range: items 10-19/100

[egobrain]

Range can be set not only to 'bytes'

Here is a part of RFC 2616

3.12 Range Units

HTTP/1.1 allows a client to request that only part (a range of) the response entity be included within the response. HTTP/1.1 uses range units in the Range (section 14.35) and Content-Range (section 14.16) header fields. An entity can be broken down into subranges according to various structural units.

 range-unit       = bytes-unit | other-range-unit
 bytes-unit       = "bytes"
 other-range-unit = token

The only range unit defined by HTTP/1.1 is "bytes". HTTP/1.1 implementations MAY ignore ranges specified using other units.

Using Range header is a best way to organize pagination in RESTfull API.

[essen]

Missed the part on units. OK.

[essen]

The Ranch stuff got merged. We're one step closer to having this done!

[tuscland]

Hi Loïc, Is there anything missing to support Range header in cowboy_rest? I am not sure about this but it seems this issue can be closed, isn't it? Also, can we support returning 206 Partial content when a Content-Range header is specified in the response? Camille

[essen]

Well it's open because nothing has been done yet. I want full support following the rfc.

Loïc Hoguin http://ninenines.eu

-------- Original Message -------- From:Camille Troillard notifications@github.com Sent:Sun, 31 Aug 2014 15:58:10 +0300 To:ninenines/cowboy cowboy@noreply.github.com Cc:Loïc Hoguin essen@ninenines.eu Subject:Re: [cowboy] Support Range: in cowboy_rest (#306)

Hi Loïc, Is there anything missing to support Range header in cowboy_rest? I am not sure about this but it seems this issue can be closed, isn't it? Also, can we support returning 206 Partial content when a Content-Range header is specified in the response? Camille

— Reply to this email directly or view it on GitHub.

[tuscland]

Sounds great, thanks!

[s-kostyaev]

You can see partial implementation in https://github.com/ninenines/cowboy/pull/1059

[essen]

I've started work on full support for RFC7233. Here are the current plans:

• [x] A new callback ranges_provided/2 to be called after charsets_provided/2. It returns a list of range units and their associated callbacks. When a range request is to be served, the callback returned here will be called instead of the one in content_types_provided/2.
• [x] The accept-ranges header is built based on the output of ranges_provided/2. The header will only be sent for resources exporting the callback. An empty list means accept-ranges: none is sent.
• [x] At the end of the conditional checks (if-match and friends), and if both range and if-range headers are present, then if-range header is checked. If it evaluates to false, a full response will be sent, otherwise a ranged response might be sent.
• [x] After if-range, the range header is parsed, if it fails to parse, a 416 response is sent.
• [x] The range header is then checked against the ranges_provided/2 and if the unit is unknown, a 416 response is sent, otherwise a ranged response might be sent.
• [x] If all other preconditions are valid, the callback range_satisfiable/2 is called. This allows rejecting a range request with a 416 as a last resort, for example if it requests a byte range that's larger than available.
• [x] If all preconditions are valid, the callback defined in ranges_provided/2 is called instead of the one in content_types_provided/2 when the time comes to send the body. When it returns, it sends a 206 immediately instead of potentially calling multiple_choices/2.

And:

• [x] In addition, because RFC7233 defines multipart/byteranges and it'd be good to apply it automatically to existing REST handlers, ranges_provided/2 will accept the tuple {<<"bytes">>, auto} to indicate that cowboy_rest should take care of splitting the partial content on its own. In that case cowboy_rest will call the callback defined in content_types_provided/2 and will do the splitting itself and building the correct 206 response (with/without multipart as needed) automatically.

This last part will probably require me to implement https://github.com/ninenines/cowboy/issues/984 and https://github.com/ninenines/cowboy/issues/1113 at the same time.

Thoughts?

[essen]

I've just pushed 29043aa which contains a full RFC7233 implementation for cowboy_rest. I will also add the "auto" mode for byte ranges to simplify adding range support to resources and then I'll also add that to cowboy_static (which uses sendfile and therefore there will be no real performance impact).

[essen]

I've now added the ability to use sendfile while streaming the response body, so all that's left is implementing the auto mode and then applying it on cowboy_static.

[essen]

I've pushed three more commits:

• Add the "auto" mode for bytes units that can be enabled via ranges_provided/2
• Fix case where a resource responds to a multiple range request using sendfile tuples
• Enable range request support in cowboy_static

Other than documentation, which will be done once I've confirmed that this fits a customer use case, I believe everything has been added. Please experiment and open new tickets if you find any issues. Thanks!