Change List.IterAll to read-ahead 8MB chunks over 6 threads

[kalman]

This copies the approach that Blob.Copy uses, with some extra logic to adjust the number of values to read as more values are read.

[kalman]

This isn't to submit yet, because it hardly makes a difference in csv-export. @rafael-atticlabs could you check it's sensible?

Here is what the csv exported used to do (400MB of csv data):

> go install ./samples/go/csv/... && (time bin/csv-export db::csv) >/dev/null

real    0m14.539s
user    0m22.550s
sys 0m4.139s

and with my changes:

> go install ./samples/go/csv/... && (time bin/csv-export db::csv) >/dev/null
read 1000 chunks (345 kB)
adjusted chunk size from 1000 to 24301 (scaled by 24.301)
read 1000 chunks (355 kB)
adjusted chunk size from 1000 to 23600 (scaled by 23.600)
read 1000 chunks (371 kB)
adjusted chunk size from 1000 to 22620 (scaled by 22.621)
read 1000 chunks (351 kB)
adjusted chunk size from 1000 to 23899 (scaled by 23.899)
read 1000 chunks (364 kB)
adjusted chunk size from 1000 to 23054 (scaled by 23.054)
read 1000 chunks (318 kB)
adjusted chunk size from 1000 to 26354 (scaled by 26.354)
read 1000 chunks (338 kB)
adjusted chunk size from 1000 to 24821 (scaled by 24.821)
read 1000 chunks (346 kB)
adjusted chunk size from 1000 to 24280 (scaled by 24.280)
read 26354 chunks (8.4 MB)
read 24280 chunks (7.7 MB)
adjusted chunk size from 24280 to 26305 (scaled by 1.083)
read 24280 chunks (7.7 MB)
adjusted chunk size from 24280 to 26392 (scaled by 1.087)
read 24280 chunks (7.7 MB)
adjusted chunk size from 24280 to 26331 (scaled by 1.084)
read 24280 chunks (7.7 MB)
adjusted chunk size from 24280 to 26286 (scaled by 1.083)
read 24280 chunks (7.8 MB)
adjusted chunk size from 24280 to 26246 (scaled by 1.081)
read 24280 chunks (7.9 MB)
adjusted chunk size from 24280 to 25825 (scaled by 1.064)
read 24280 chunks (7.8 MB)
adjusted chunk size from 24280 to 26189 (scaled by 1.079)
read 25825 chunks (8.3 MB)
adjusted chunk size from 25825 to 26240 (scaled by 1.016)
read 26189 chunks (8.4 MB)
read 26240 chunks (8.4 MB)
read 26240 chunks (8.4 MB)
read 26240 chunks (8.4 MB)
read 26240 chunks (8.4 MB)
read 26240 chunks (8.4 MB)
read 26240 chunks (8.4 MB)
read 26240 chunks (8.4 MB)
read 26240 chunks (8.4 MB)
read 26240 chunks (8.4 MB)
read 26240 chunks (8.4 MB)
read 26240 chunks (8.4 MB)
read 26240 chunks (8.4 MB)
read 26240 chunks (8.4 MB)
read 26240 chunks (8.4 MB)
read 26240 chunks (8.4 MB)
read 26240 chunks (8.4 MB)
read 26240 chunks (8.4 MB)
read 26240 chunks (8.4 MB)
read 26240 chunks (8.4 MB)
read 26240 chunks (8.4 MB)
read 26240 chunks (8.4 MB)
read 26240 chunks (8.4 MB)
read 26240 chunks (8.4 MB)
read 26240 chunks (8.4 MB)
read 26240 chunks (8.4 MB)
read 26240 chunks (8.4 MB)
read 26240 chunks (8.4 MB)
read 26240 chunks (8.4 MB)
read 26240 chunks (8.4 MB)
read 26240 chunks (8.4 MB)
read 26240 chunks (8.4 MB)
read 26240 chunks (8.3 MB)
read 26240 chunks (8.4 MB)
read 26240 chunks (8.4 MB)
read 26240 chunks (8.4 MB)
read 26240 chunks (8.4 MB)
read 26240 chunks (8.4 MB)
read 26240 chunks (8.4 MB)
read 26240 chunks (8.4 MB)
read 26240 chunks (8.4 MB)
read 26240 chunks (8.4 MB)
read 26240 chunks (8.4 MB)
read 26240 chunks (8.4 MB)
read 26240 chunks (8.4 MB)
read 26240 chunks (8.4 MB)
read 26240 chunks (8.4 MB)
read 26240 chunks (8.4 MB)
read 26240 chunks (8.4 MB)
read 26240 chunks (8.4 MB)
read 26240 chunks (8.4 MB)
read 26240 chunks (8.4 MB)
read 26240 chunks (8.4 MB)
read 26240 chunks (8.4 MB)
read 26240 chunks (8.4 MB)
read 26240 chunks (8.4 MB)
read 26240 chunks (8.4 MB)
read 26240 chunks (8.4 MB)
read 26240 chunks (8.4 MB)
read 26240 chunks (8.4 MB)
read 26240 chunks (8.3 MB)
read 26240 chunks (8.4 MB)
read 26240 chunks (8.4 MB)
read 26240 chunks (8.3 MB)
read 26240 chunks (8.4 MB)
read 26240 chunks (8.3 MB)
read 26240 chunks (8.4 MB)
read 26240 chunks (8.4 MB)
read 26240 chunks (8.4 MB)
read 26240 chunks (8.4 MB)
read 26240 chunks (8.3 MB)
read 23945 chunks (7.6 MB)

real    0m14.637s
user    0m22.263s
sys 0m4.393s

[kalman]

final observations before I dig into this again:

• the codec benchmark is also identical
• maybe this needs to run on S3 to see the change?
• marshal test fails. I think it's something to do with IterAll and (un)marshalling using the same internal state (like values()?) and getting confused.

[kalman]

latest patch is simpler/better but marshal test still fails, and csv export is unaffected (in fact it might be slower).

[kalman]

There we go.

[kalman]

OK, @rafael-atticlabs (and @arv) PTAL.

• The marshal tests are now passing - I guess the values changes that @arv suggested fixed that (though honestly it does still worry me a little bit why it was failing in the first place).
• I tried with a much larger dataset (~2GB), with a hacked-up csv-export to not actually construct the CSV, and the old code took 35s and the new code 30s, so that's quite encouraging.