net/http: handling range requests for empty files

MSevey commented 3 years ago

What version of Go are you using (`go version`)?

go version go1.16.3 darwin/amd64

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (`go env`)?

go env Output

$ go env

GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/matt/Library/Caches/go-build"
GOENV="/Users/matt/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/matt/Code/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/matt/Code/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GOVCS=""
GOVERSION="go1.16.3"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/matt/Code/go/skyd/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch x86_64 -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/f9/2yp855dn19s0cqcjwbrv_0900000gn/T/go-build2190681930=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

In our API package we add a range request to the HEADER like so

    req, err := c.NewRequest("GET", resource, nil)
    if err != nil {
        return nil, errors.AddContext(err, "failed to construct GET request")
    }
    req.Header.Add("Range", fmt.Sprintf("bytes=%d-%d", from, to))
    httpClient := http.Client{CheckRedirect: c.CheckRedirect}
        res, err := httpClient.Do(req)

When we extended our testing to include zero-byte (ie empty) files we found that the response from the httpClient.Do method is 416 Requested Range Not Satisfiable. This was with a from and to values of 0.

Now I know that technically that is correct since that range request is asking for the first byte and that doesn't exist in an empty file. However, I could not find in the documentation a way to submit range requests for empty files.

Would it be possible for the standard library to be updated to provided direction on how to submit a range request for an empty file? Or do you expect the caller to make that determination and just not submit a range if it is an empty file.

What did you expect to see?

There should be a standard way to handle a range request for a zero-byte (empty) file.

What did you see instead?

416 Requested Range Not Satisfiable

neild commented 3 years ago

What specific behavior do you want here?

From RFC 7433, Section 4.4:

The 416 (Range Not Satisfiable) status code indicates that none of the ranges in the request's Range header field (Section 3.1) overlap the current extent of the selected resource...

A request for "bytes=0-0" is a request for the first byte of a resource. If the resource is 0-length, the range [0,0] does not overlap the resource content and the server responds with a 416. That seems like a correct response.

RFC 7433, Section 4.4 does also state:

For byte ranges, failing to overlap the current extent means that the first-byte-pos of all of the byte-range-spec values were greater than the current length of the selected representation.

That could be read as stating that a request for [0,0] overlaps a 0-length representation, because the first-byte-pos (0) is not greater than the length (0). However, this interpretation would require responding with a 0-length 206 Partial Content response, which is not possible since the Content-Range header does not provide a way to encode a 0-length range. So I believe the interpretation of this sentence consistent with the rest of RFC 7433 is that a range fails to overlap if every first-byte-pos references a value past the end of the content.

MSevey commented 3 years ago

What specific behavior do you want here?

@neild thanks for the response

I agree with what you said. Using range requests and handling 0 byte files seems like a pretty common case to handle, so my main question was if we could add documentation to the go library for how to handle that case or have a special case for this.

When I was looking into it I did find this project issue that talked about using a 1-0 range request to indicate a 0 byte file. https://github.com/SitePen/dstore/issues/109#issuecomment-383220628

If you don' t feel it is appropriate for the go library to handle the 0 byte file case then I think it would be good to add something to the documentation so that the next time someone is searching for how to handle 0 byte file range requests in golang they come across the docs that say they don't :-) and that each application to handle those requests.

neild commented 3 years ago

This isn't really a Go issue. There's no way to issue an RFC 7433-conformant request for a range of a 0-length file. A range request is a request for a non-zero-length range of bytes. A 0-length file contains no bytes. I believe that a 416 Range Not Satisfiable response is correct in this situation. There are other correct responses (we could ignore the Range header entirely for 0-length files, for example, and return a 200 response), but I don't see a reason those other possibilities are better. I don't think there's anything surprising about responding to a request for data past the end of a file with, "Sorry, the file isn't that big".

A request containing bytes=1-0 is unambiguously invalid under RFC 7433:

A byte-range-spec is invalid if the last-byte-pos value is present and less than the first-byte-pos.

golang / go