Closed luqmana closed 4 months ago
Dropshot doesn't have much to support this, but it is possible; I added an example in https://github.com/oxidecomputer/dropshot/pull/776.
As of 3a2cc1b171198f85bf052c3eef1f20e030648a2a we now support HTTP range requests on published file downloads. I have done some relatively extensive testing at this point, and I think it's actually working.
For example, this horrendous shell script:
#!/bin/bash
R=oxidecomputer/omicron
C=b4e1a285ef812bc0376959e177c7ab3f90893e73
B=https://buildomat.eng.oxide.computer/public/file/$R/rot-all/$C/repo.zip
rm -f repo.zip.parta
rm -f repo.zip.partb
rm -f repo.zip.sha256.txt
curl -o repo.zip.sha256.txt "$B.sha256.txt"
for p in a b; do
while ! curl -Y 10000000 -y 8 -C - -o "repo.zip.part$p" "$B.part$p"; do
sleep 0.1
done
done
echo "want: $(<repo.zip.sha256.txt)"
echo "got: $(cat repo.zip.parta repo.zip.partb | digest -a sha256)"
Has this output:
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 65 100 65 0 0 190 0 --:--:-- --:--:-- --:--:-- 190
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
3 1024M 3 34.5M 0 0 4385k 0 -0:03:59 -0:00:08 -0:03:51 4910k
curl: (28) Operation too slow. Less than 10000000 bytes/sec transferred the last 8 seconds
** Resuming transfer from byte position 36273942
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
2 989M 2 27.9M 0 0 3403k 0 -0:04:57 -0:00:08 -0:04:49 4759k
curl: (28) Operation too slow. Less than 10000000 bytes/sec transferred the last 8 seconds
** Resuming transfer from byte position 65596906
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
4 961M 4 41.6M 0 0 5285k 0 -0:03:06 -0:00:08 -0:02:58 4274k
curl: (28) Operation too slow. Less than 10000000 bytes/sec transferred the last 8 seconds
** Resuming transfer from byte position 109313250
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
4 919M 4 41.2M 0 0 5075k 0 -0:03:05 -0:00:08 -0:02:57 6494k
curl: (28) Operation too slow. Less than 10000000 bytes/sec transferred the last 8 seconds
** Resuming transfer from byte position 152529846
...
** Resuming transfer from byte position 1034346348
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
51 37.5M 51 19.3M 0 0 2398k 0 -0:00:16 -0:00:08 -0:00:08 1901k
curl: (28) Operation too slow. Less than 10000000 bytes/sec transferred the last 8 seconds
** Resuming transfer from byte position 1054604882
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 18.2M 100 18.2M 0 0 3248k 0 -0:00:05 -0:00:05 --:--:-- 3630k
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
2 634M 2 14.8M 0 0 1883k 0 10:05:45 -0:00:08 10:05:37 1969k
curl: (28) Operation too slow. Less than 10000000 bytes/sec transferred the last 8 seconds
** Resuming transfer from byte position 15576837
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
3 619M 3 18.7M 0 0 2376k 0 -0:04:27 -0:00:08 -0:04:19 2356k
curl: (28) Operation too slow. Less than 10000000 bytes/sec transferred the last 8 seconds
** Resuming transfer from byte position 35249610
...
** Resuming transfer from byte position 604882836
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
61 57.8M 61 35.4M 0 0 4489k 0 -0:00:13 -0:00:08 -0:00:05 6681k
curl: (28) Operation too slow. Less than 10000000 bytes/sec transferred the last 8 seconds
** Resuming transfer from byte position 642041487
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 22.3M 100 22.3M 0 0 4305k 0 -0:00:05 -0:00:05 --:--:-- 4046k
want: 33120131a64840dcaa982bb93b4c3db274747d23fa11776eb4bff84fcf4a9a1d
got: 33120131a64840dcaa982bb93b4c3db274747d23fa11776eb4bff84fcf4a9a1d
Tried to resume downloading a published artifact that was only partially downloaded using
-C -
withcurl
but instead got: