Support HTTP range requests for downloading artifacts

[luqmana]

Tried to resume downloading a published artifact that was only partially downloaded using -C - with curl but instead got:

curl: (33) HTTP server doesn't seem to support byte ranges. Cannot resume.

[jgallagher]

Dropshot doesn't have much to support this, but it is possible; I added an example in https://github.com/oxidecomputer/dropshot/pull/776.

[jclulow]

As of 3a2cc1b171198f85bf052c3eef1f20e030648a2a we now support HTTP range requests on published file downloads. I have done some relatively extensive testing at this point, and I think it's actually working.

For example, this horrendous shell script:

#!/bin/bash

R=oxidecomputer/omicron
C=b4e1a285ef812bc0376959e177c7ab3f90893e73
B=https://buildomat.eng.oxide.computer/public/file/$R/rot-all/$C/repo.zip

rm -f repo.zip.parta
rm -f repo.zip.partb
rm -f repo.zip.sha256.txt

curl -o repo.zip.sha256.txt "$B.sha256.txt"
for p in a b; do
        while ! curl -Y 10000000 -y 8 -C - -o "repo.zip.part$p" "$B.part$p"; do
                sleep 0.1
        done
done
echo "want: $(<repo.zip.sha256.txt)"
echo "got: $(cat repo.zip.parta repo.zip.partb | digest -a sha256)"

Has this output:

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    65  100    65    0     0    190      0 --:--:-- --:--:-- --:--:--   190
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  3 1024M    3 34.5M    0     0  4385k      0 -0:03:59 -0:00:08 -0:03:51 4910k
curl: (28) Operation too slow. Less than 10000000 bytes/sec transferred the last 8 seconds
** Resuming transfer from byte position 36273942
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  2  989M    2 27.9M    0     0  3403k      0 -0:04:57 -0:00:08 -0:04:49 4759k
curl: (28) Operation too slow. Less than 10000000 bytes/sec transferred the last 8 seconds
** Resuming transfer from byte position 65596906
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  4  961M    4 41.6M    0     0  5285k      0 -0:03:06 -0:00:08 -0:02:58 4274k
curl: (28) Operation too slow. Less than 10000000 bytes/sec transferred the last 8 seconds
** Resuming transfer from byte position 109313250
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  4  919M    4 41.2M    0     0  5075k      0 -0:03:05 -0:00:08 -0:02:57 6494k
curl: (28) Operation too slow. Less than 10000000 bytes/sec transferred the last 8 seconds
** Resuming transfer from byte position 152529846

...

** Resuming transfer from byte position 1034346348
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
 51 37.5M   51 19.3M    0     0  2398k      0 -0:00:16 -0:00:08 -0:00:08 1901k
curl: (28) Operation too slow. Less than 10000000 bytes/sec transferred the last 8 seconds
** Resuming transfer from byte position 1054604882
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 18.2M  100 18.2M    0     0  3248k      0 -0:00:05 -0:00:05 --:--:-- 3630k
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  2  634M    2 14.8M    0     0  1883k      0 10:05:45 -0:00:08 10:05:37 1969k
curl: (28) Operation too slow. Less than 10000000 bytes/sec transferred the last 8 seconds
** Resuming transfer from byte position 15576837
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  3  619M    3 18.7M    0     0  2376k      0 -0:04:27 -0:00:08 -0:04:19 2356k
curl: (28) Operation too slow. Less than 10000000 bytes/sec transferred the last 8 seconds
** Resuming transfer from byte position 35249610

...

** Resuming transfer from byte position 604882836
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
 61 57.8M   61 35.4M    0     0  4489k      0 -0:00:13 -0:00:08 -0:00:05 6681k
curl: (28) Operation too slow. Less than 10000000 bytes/sec transferred the last 8 seconds
** Resuming transfer from byte position 642041487
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 22.3M  100 22.3M    0     0  4305k      0 -0:00:05 -0:00:05 --:--:-- 4046k
want: 33120131a64840dcaa982bb93b4c3db274747d23fa11776eb4bff84fcf4a9a1d
got: 33120131a64840dcaa982bb93b4c3db274747d23fa11776eb4bff84fcf4a9a1d