wget fails to download > 2GB (Buster works, Bullseye fails)

[holta]

Where is the best place to report this Raspberry Pi OS regression?

Facts:

• wget succeeds to download files that are larger than 2GB on Raspberry Pi OS Buster.
• wget fails to download files that are larger than 2GB 2GB on Raspberry Pi OS Bullseye.

Thank You!

[popcornmix]

I can confirm:

$ wget https://downloads.raspberrypi.org/raspios_full_armhf/images/raspios_full_armhf-2021-11-08/2021-10-30-raspios-bullseye-armhf-full.zip
--2021-11-15 16:00:49--  https://downloads.raspberrypi.org/raspios_full_armhf/images/raspios_full_armhf-2021-11-08/2021-10-30-raspios-bullseye-armhf-full.zip
Resolving downloads.raspberrypi.org (downloads.raspberrypi.org)... 2a00:1098:82:47::2:1, 2a00:1098:80:56::3:1, 2a00:1098:80:56::2:1, ...
Connecting to downloads.raspberrypi.org (downloads.raspberrypi.org)|2a00:1098:82:47::2:1|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2147483647 (2.0G) [application/zip]
Saving to: ‘2021-10-30-raspios-bullseye-armhf-full.zip’

2021-10-30-raspios-bullseye-armhf-full.zi 100%[===================================================================================>]   2.00G  8.55MB/s    in 1m 57s  

2021-11-15 16:02:49 (17.6 MB/s) - ‘2021-10-30-raspios-bullseye-armhf-full.zip’ saved [2147483647/2147483647]

$ ls -l 2021-10-30-raspios-bullseye-armhf-full.zip 
-rw-r--r-- 1 pi pi 2147483647 Oct 30 13:05 2021-10-30-raspios-bullseye-armhf-full.zip

Length: 2147483647 (2.0G)

Looks like it's getting truncated to 2G (actual file size is 3.0G)

[XECDesign]

Forwarded it to plugwash.

Looks like wget is getting the 'content-length' header wrong and truncating it to 2GB for some reason. Haven't stumbled across an obvious reason why yet. In the meantime, you should be able to use --ignore-length argument.

[XECDesign]

This might be all that's needed:

diff --git a/src/wget.h.old b/src/wget.h
index 6f20eab..f422737 100644
--- a/src/wget.h.old
+++ b/src/wget.h
@@ -144,7 +144,7 @@ typedef int64_t wgint;
 #define WGINT_MAX INT64_MAX
 typedef wgint SUM_SIZE_INT;

-#define str_to_wgint strtol
+#define str_to_wgint strtoll

 #include "options.h"

Lifted from https://advisories.mageia.org/MGAA-2021-0123.html

I'll test it tomorrow.

[plugwash]

I found the upstream issue.

It seems that wget decided to remove some crufty old code relating to support for old platforms/compilers, but they screwed up in doing so and used strtol instead of strtoll.

Specifically https://gitlab.com/gnuwget/wget/-/commit/8b1aeab78314ecdeaccbcd72e3589887d717f218 was where the bug was introduced and https://gitlab.com/gnuwget/wget/-/commit/90631a6fe54eabd9c80ede5c70bc916719e76cfe was where it was fixed. Unfortunately the version that ended up in bullseye was between these two versions.

I have applied the upstream fix to the package and uploaded it to raspbian, it should come through in the next update run. I have also reported the issue in Debian to hopefully get it fixed there.

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=999744

[holta]

Where do we nominate these Raspberry Pi folk for the Nobel Peace Prize?

Thank you being so responsive — in service of https://Internet-in-a-Box.org and our Raspberry Pi medical & educational work across the developing world — that really does need this critical fix.

[XECDesign]

Credit to the Raspbian folk (plugwash) in this case.

[holta]

It's confirmed fixed — after I installed this apt update to wget — that appeared overnight: (i.e. is now available to everyone)

root@box:~# apt list --upgradable
Listing... Done
wget/stable 1.21-1rpi1 armhf [upgradable from: 1.21-1]

[holta]

Old news, but just some wget version history context for the record:

http://raspbian.raspberrypi.org/raspbian/pool/main/w/wget/?C=M;O=D