swift: readahead support when reading objects

[rogpeppe]

We make it more efficient for streaming and seeking readers by issuing reads ahead of time, meaning we can be processing some data while the HTTP request is being read.

Control over the amount of readahead is governed by the readAhead parameter to the new OpenObject method, which subsumes GetReadSeeker.

[jrwren]

My quick tests show it as performing much better, but this is using the behavior of io.Copy and io.Discard which has a certain pattern. Other patterns may vary.

Old method: reading a small 1193B file, 1000 times takes: 19.111059 seconds reading a larger 111062826B file, 1 time takes: 204.638050 seconds

    start := time.Now()
    n := int64(0)
    for i := 0; i < *count; i++ {
        req2, _, err := s.OpenObject(*container, *name, -1)
        if err != nil {
            fmt.Printf("ERROR:%s\n", err)
            return
        }
        m, err := io.Copy(ioutil.Discard, req2)
        if err != nil {
            fmt.Printf("ERROR:%s\n", err)
            return
        }
        n += m
        req2.Close()
    }
    fmt.Printf("%d bytes read in %f seconds\n", n, time.Now().Sub(start).Seconds())

New Method:

Reading a small 1193B file, 1000 times takes: 11.413s 1193000 bytes read in 11.413108 seconds

reading a larger 111062826B file, 1 time takes: 12.6s 111062826 bytes read in 12.678619 seconds

[rogpeppe]

Thanks for the measurements? What value for readAhead were you using? It could make a big difference (and readAhead=-1) is essentially equivalent to the old GetReader except you can still seek if you want to.

[rogpeppe]

$$merge$$

[mhilton]

$$merge$$

[jujubot]

Status: merge request accepted. Url: http://juju-ci.vapour.ws:8080/job/github-merge-goose

[jujubot]

Build failed: Merging failed build url: http://juju-ci.vapour.ws:8080/job/github-merge-goose/49