How to correctly handle 416 Requested Range not Satisfiable

wadtech commented 8 years ago

If I use grab to download the same file multiple times consecutively, given a server that responds to unsatisfiable range requests with 416 and a response body, I will get a corrupted file on the second attempt.

Using the demo code provided:

package main

import (
    "fmt"
    "os"
    "time"

    "github.com/cavaliercoder/grab"
)

func main() {
    // get URL to download from command args
    if len(os.Args) < 2 {
        fmt.Fprintf(os.Stderr, "usage: %s url\n", os.Args[0])
        os.Exit(1)
    }

    url := os.Args[1]

    // start file download
    fmt.Printf("Downloading %s...\n", url)
    respch, err := grab.GetAsync(".", url)
    if err != nil {
        fmt.Fprintf(os.Stderr, "Error downloading %s: %v\n", url, err)
        os.Exit(1)
    }

    // block until HTTP/1.1 GET response is received
    fmt.Printf("Initializing download...\n")
    resp := <-respch

    // print progress until transfer is complete
    for !resp.IsComplete() {
        fmt.Printf("Progress %d / %d bytes (%d%%)\n", resp.BytesTransferred(), resp.Size, int(100*resp.Progress()))
        time.Sleep(200 * time.Millisecond)
    }

    // check for errors
    if resp.Error != nil {
        fmt.Fprintf(os.Stderr, "Error downloading %s: %v\n", url, resp.Error)
        os.Exit(1)
    }

    fmt.Printf("Successfully downloaded to ./%s\n", resp.Filename)
}

and the url https://s.w.org/screenshots/3.8/colors.png

I get the following output:

% ./test-download https://s.w.org/screenshots/3.8/colors.png
Downloading https://s.w.org/screenshots/3.8/colors.png...
Initializing download...
Progress 0 / 356063 bytes (0%)
Successfully downloaded to ./colors.png
% ./test-download https://s.w.org/screenshots/3.8/colors.png
Downloading https://s.w.org/screenshots/3.8/colors.png...
Initializing download...
Progress 356063 / 356452 bytes (99%)
Successfully downloaded to ./colors.png
% ./test-download https://s.w.org/screenshots/3.8/colors.png
Downloading https://s.w.org/screenshots/3.8/colors.png...
Initializing download...
Error downloading https://s.w.org/screenshots/3.8/colors.png: Existing file (356452 bytes) is larger than remote (356063 bytes)

Note the difference in calculated content length.

Using less to view the end of the file reveals the error body from the later requests:

... rest omitted ...
^C<EA><A1>nˢ<D4>|<F3><F9>k<F3>;<C6>6^F+<A5><E1>:%`<92>C<F2><U+DCFD><B8>^_j<E5><D3>fn<E5><BC><CB>1<92>N<A6><CC>e<DE>ֹ<C5><CB>        E<EB><AC>Cm_<8B>^N\<BE><92><C7>=<8B>8Q^LW<A8>(^Y<81>E<D1>C:
^B<A4><A3>e<B4>^\<E4>^O3     J^Gy<A8><A1>*M<A4><F9>^X^\$^S<BF>d+^RA]<EC>^_<8D>ee<A8><D5>(^S<C3>(<89><CC> R^B<A5>@.<97>ESC
<80>^W<D9>X;^A^H<D9><F2><A2>-^_<94><9F>cFZ<A2>/U<AB><EA>v<E5><A7>^T<FB>)I><E9>;^P^H^F^P<FA>t|<98>^@Q^D\^<AC>ESC<C1>Z'*<84><A2><E7>11u<A4><FA>>r<A9>1fE[i<A9>^R<B7>C<95><F3>#Ռ)^^<BF><D7><E3>
<83><CC>䆲<EB>4<FF><F2>\D^K<87>^L<FE>R_l^D^?^E<BB>M*1<D0>^*N<9C>^G!<EE>^D^Y<B8>ʾ<D1><D4>;,^@v<8E>'<BB>W/<F7><8A>
<9B>5<DF>]<8F>|<98>^^D<99>8Rw/e^K^\'K<A1><A9><FF>^R<9F><9B>dN<FF><C6>I<E2>d<AB>^Xډ<C6> ^X;<92>^Zؒ<EA><BB>^B<E8>!jf-l^O<C5><F7><A0><C2>@Y^@<AB>^Tv]<FF>wg^A/<E5><B6>,<88>^@^E|c<A7>p<BD><94><D9>W<80><A8><AF><F7>tZ<E6>V<CC>5<AA><F2>\^C<A8>^T<B0>uu<D0>^X<CF>^S^X<E5><EF><E2><9C>^Z<E6><D8>^@<CF>B^Y<D8>><FE>^K<90><EE><C4>>e^Y<B4>ݎ3 ^VLIl}<BF><C5>yQ"c^W<D5><CB><F3><89>Vi<D8>^Y<A5><E0><A8>
<EC><9D><F7><8B>^FD<93><DC>.V<AE><AB>>[^\<CB> [<C6>t<AE>4ɚ<C9><U+063B>w<EF>!<95>^\^_       ;<ED><FF>233<F3>Jf1gMr<86><98>^E<F7><C3>^P{^^<97><FF>/<C0>^@    g?<B9>G<A8><DE>H^@^@^@^@IEND<AE>B`
<82><?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
         "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
        <head>
                <title>416 - Requested Range Not Satisfiable</title>
        </head>
        <body>
                <h1>416 - Requested Range Not Satisfiable</h1>
        </body>
</html>

If this is something I should account for in my client code (I'm using the library similarly to the demo) then please could I have some pointers on where to look in the docs? If it's a bug I'm happy to help.

Thanks

cavaliercoder commented 8 years ago

Thanks for raising this. The issue was that requests for existing, complete files were still submitting a resume request to the server, starting from the end of the file which is obviously out of range.

HTTP response codes should always be checked by the client code, not grab. The reason for this is that it is not safe for grab to assume that an expected response is definitely 2XX. In this particular case however, the issue was definitely in grab's codebase.

wadtech commented 8 years ago

Thanks for fixing so quickly.

cavaliergopher / grab

How to correctly handle 416 Requested Range not Satisfiable #1