Streamline Multistrap

[ashthespy]

Following up from #459 -- Cleaned up some redundant files -- Added an apt-cache option that should help speedup/reduce traffic during builds

@xipmix Let me know if this helps your use case? Should be able to use it as follows:

export APT_CACHE='http://localhost:3142'
build.sh []

This sets a cache file in /etc/apt/apt.conf.d/ that is later removed from the final image, so it should be used throughout the image building process.

[xipmix]

I tried this and have had difficultly getting it to work correctly.

The first problem is I get to a certain point during multistrap and one of the package 'downloads' fails. This is what I get with a clean package cache:

# ./build.sh -d pi -b arm -v myvers3
... things proceed normally...
[ .. ] Running multistrap for arm (armhf) [ arm.conf ]
...
W: Target Translations (main/i18n/Translation-en) is configured multiple times in /data/volumio/Build/build/arm/root/etc/apt/sources.list.d/multistrap-accessories.list:1 and /data/volumio/Build/build/arm/root/etc/apt/sources.list.d/multistrap-volumio.list:1
E: Failed to fetch http://archive.raspbian.org/raspbian/pool/main/k/keyutils/keyutils_1.6-6_armhf.deb  Connection failed [IP: 192.168.0.3 3142]
E: Some files failed to download
apt download failed. Exit value: 100

The file is actually in the cache

$ ls -lt /var/cache/apt-cacher-ng/archive.raspbian.org/raspbian/pool/main/k/keyutils/
total 52
-rw-r--r-- 1 apt-cacher-ng apt-cacher-ng 28416 Jan 25 13:35 keyutils_1.6-6_armhf.deb
-rw-r--r-- 1 apt-cacher-ng apt-cacher-ng   287 Jan 25 13:35 keyutils_1.6-6_armhf.deb.head
-rw-r--r-- 1 apt-cacher-ng apt-cacher-ng 13976 Jan 25 13:34 libkeyutils1_1.6-6_armhf.deb
-rw-r--r-- 1 apt-cacher-ng apt-cacher-ng   291 Jan 25 13:34 libkeyutils1_1.6-6_armhf.deb.head

I ran again, using the cache I'd built up so far and got past the multistrap step this time. I noticed the cached file got updated too - the file seems to have been incomplete. Perhaps multistrap is doing some parallel file grabbing going on that's too much for apt-cacher-ng?

$ ls -lt /var/cache/apt-cacher-ng/archive.raspbian.org/raspbian/pool/main/k/keyutils/
total 72
-rw-r--r-- 1 apt-cacher-ng apt-cacher-ng 46824 Jan 25 13:52 keyutils_1.6-6_armhf.deb
-rw-r--r-- 1 apt-cacher-ng apt-cacher-ng   287 Jan 25 13:52 keyutils_1.6-6_armhf.deb.head
-rw-r--r-- 1 apt-cacher-ng apt-cacher-ng 13976 Jan 25 13:34 libkeyutils1_1.6-6_armhf.deb
-rw-r--r-- 1 apt-cacher-ng apt-cacher-ng   291 Jan 25 13:34 libkeyutils1_1.6-6_armhf.deb.head

However there's a final failure.

[ -- ] Adding NodeJs lists - node_14.x 
Err:1 https://deb.nodesource.com/node_14.x buster InRelease
  Reading from proxy failed - read (115: Operation now in progress) [IP: 192.168.0.3 3142]
Reading package lists... Done
W: Failed to fetch https://deb.nodesource.com/node_14.x/dists/buster/InRelease  Reading from proxy failed - read (115: Operation now in progress) [IP: 192.168.0.3 3142]
W: Some index files failed to download. They have been ignored, or old ones used instead.
Reading package lists... Done
Building dependency tree... Done
E: Unable to locate package nodejs
[ -- ] Volumio chroot config failed [ volumioconfig.sh ]
[ error ] Build script failed!! 

So I don't think this functionality is quite ready for prime time. I'll keep poking at it to see what improvements I can come up with.

Questions:

• Have you got this to work on your own setup?
• Why does the curl check refer to APT_PROXY instead of APT_CACHE; are you testing if APT_PROXY is set?

As an aside, one must set the APT_CACHE variable in a root shell - sudo ./build.sh... will unset it. I'll mention that in a future docfix pull request.

[ashthespy]

To be honest, I've not played around much with much..

-- multistrap internally just uses host system's apt, so no parallel downloading happening here.. -- Are you setting up proper https tunnelling for repos that require it? (such as https://deb.nodesource.com/)

Why does the curl check refer to APT_PROXY instead of APT_CACHE; are you testing if APT_PROXY is set?

That is a fat fingers mistake from me.. :doh:

As an aside, one must set the APT_CACHE variable in a root shell - sudo ./build.sh... will unset it.

Run with sudo -E ./build.sh... to preserve environment..

[xipmix]

-- Are you setting up proper https tunnelling for repos that require it? (such as https://deb.nodesource.com/)

Not so far. I tried this, which got me over the line; all the other https sources seem to use wget rather than apt.

     cat <<-EOF >"${DirEtcparts}/02cache"
                Acquire::http { Proxy "${APT_CACHE}"; };
+               Acquire::HTTP::Proxy::deb.nodesource.com "DIRECT";
                EOF
   fi

[ashthespy]

I am assuming you have already setup the proper PassThroughPattern in your apt-cacher-ng/ configuration?

PassThroughPattern: ^(.*):443$

Your solution seems to be suggested in nodesource/distributions, but I was hoping they had fixed it since then..

[xipmix]

Re the PassThroughPattern- I had that turned off, because I was rewriting the https urls in the recipe config files (before, many packages were being pulled from https: sources). I turned it on and it works fine, so either method should work.