debian_package_manager generate hits rate limits on snapshot.debian.org

[rcwoolley]

Running the generate application results in an i/o timeout error:

$ ./debian_package_manager/generate Processing packages: "arm64" "debian11" Processing packages: "amd64" "debian11" Processing packages: "ppc64le" "debian10" Processing packages: "arm" "debian10" Processing packages: "amd64" "debian10" Processing packages: "ppc64le" "debian11" Processing packages: "arm" "debian11" Processing packages: "arm64" "debian10" Processing packages: "s390x" "debian10" Processing packages: "s390x" "debian11" panic: error generating config: could not process packages for "s390x" "debian10": failed to fetch remote file: "https://snapshot.debian.org/archive/debian/20220505T211744Z/dists/buster/main/binary-s390x/Packages.xz": Get "https://snapshot.debian.org/archive/debian/20220505T211744Z/dists/buster/main/binary-s390x/Packages.xz": dial tcp 185.17.185.185:443: i/o timeout

goroutine 1 [running]: main.main() /opt/projects/distroless/debian_package_manager/cmd/generate/generate.go:28 +0xee

I tracked this down to build.go which uses errgroup.WithContext to run extractPackageInfo() in parallel. This seems to sometime aggravate the rate limiting on snapshot.debian.org. Removing the lines related to errgroup.WithContext makes the connections only happen one at a time and allows the application to run successfully.

Using internal mirrors or caching may alleviate this problem. However, I thought it would be useful to document this issue in case others have run into it as well.

[loosebazooka]

generate is mostly just for bootstrapping the config, update is the everyday usage. We can de-parallelize it though. What are you using it for?

[rcwoolley]

I wanted to experiment with adding packages to the container base images to see how one might create custom base images. My simple test involved adding "ed" and "nano" to the base (by also modifying base.bzl).

I was investigating whether it would be beneficial to create a "build-essential" type container image using Distroless that omitted any unnecessary packages present in the standard Debian container images.