search

volumio / Build

Buildscripts for Volumio System
GNU General Public License v2.0
113 stars 102 forks source link

[volumioOS branch] build.sh -b arm -d pi fails on /etc/os-release check #459

Closed xipmix closed 3 years ago

xipmix commented 3 years ago 
$ sudo ./build.sh -b arm -d pi -v myver
...
[ .... ] Building Raspbian System for arm (armhf)  
...
[ -- ] Adding Volumio revision information to os-release 
[ -- ] Appending Volumio rev 
PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
VOLUMIO_BUILD_VERSION="794d3a66bd86c411f112ef316f5a5b4e1a59349e"
VOLUMIO_FE_VERSION="cad4eff7365bdee75d9502abc18273965a249393"
VOLUMIO_FE3_VERSION="5f4fed497a2ed5ff425b644d7034dfde99092431"
VOLUMIO_BE_VERSION="d7a5678df93a861e2c1906f5c3ad7c28fffa68ed"
VOLUMIO_ARCH="arm"
[ o.k. ] Cloned Volumio BE  [ d7a5678 Warn when `volumio pull` receives unknown args ]
[ ---- ] Configuring Volumio  
...
[ o.k. ] Finished chroot config for pi  
[ -- ] Unmounting chroot temporary devices at /mnt/volumio/rootfs 
[ o.k. ] Finished chroot image configuration  [ 25:7 min ]
[ .... ] Finalizing Rootfs (Cleaning, Stripping, Hashing)  
[ .... ] Computing Volumio folder Hash Checksum  
[ .. ] Checking os-release 
[ error ] Missing VOLUMIO_ info in /etc/os-release! 
PRETTY_NAME="Raspbian GNU/Linux 10 (buster)"
NAME="Raspbian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=raspbian
ID_LIKE=debian
HOME_URL="http://www.raspbian.org/"
SUPPORT_URL="http://www.raspbian.org/RaspbianForums"
BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"
VOLUMIO_HASH="ccd42c83999f7d084a8e8d85ca1f2eef"
$

I think this is happening because makeimage.sh unmounts the image before calling finalize.sh, but other builds I tried (build.sh -b x86 && build -d x86_i386) worked without failing like this.

ashthespy commented 3 years ago

Could you share your full log? I added the check to catch situations where base-files gets updated. This overwrites the os-release file, and leads to this.

What is strange is that you've run into this while building a fresh rootfs in the same run. That now is puzzling..

I think this is happening because makeimage.sh unmounts the image before calling finalize.sh,

Hmm, no - it should be fine, we just unmount the chroot devices, and directly write to the ROOTFSMNT. I also notice an issue in that line, will fix it when I am doing something in there..

xipmix commented 3 years ago

I dug into this a bit more. The reason appears to be that base-files gets updated during scripts/volumio/chrootconfig.sh. Things are ok up to here

[ o.k. ] Finished Kernel installation                                           
+ cat /etc/os-release                                                           
PRETTY_NAME="Debian GNU/Linux 10 (buster)"                                      
NAME="Debian GNU/Linux"                                                         
VERSION_ID="10"                                                                 
VERSION="10 (buster)"                                                           
VERSION_CODENAME=buster                                                         
ID=debian                                                                       
HOME_URL="https://www.debian.org/"                                              
SUPPORT_URL="https://www.debian.org/support"                                    
BUG_REPORT_URL="https://bugs.debian.org/"                                       
VOLUMIO_BUILD_VERSION="611458f8bcbd70180bb501d70a9d76254c2c5759"                
VOLUMIO_FE_VERSION="cad4eff7365bdee75d9502abc18273965a249393"                   
VOLUMIO_FE3_VERSION="5f4fed497a2ed5ff425b644d7034dfde99092431"                  
VOLUMIO_BE_VERSION="d7a5678df93a861e2c1906f5c3ad7c28fffa68ed"                   
VOLUMIO_ARCH="arm"                                                              
VOLUMIO_VARIANT="volumio"                                                       
VOLUMIO_TEST="FALSE"                                                            
VOLUMIO_BUILD_DATE="Sat 16 Jan 2021 07:51:26 AEDT"                              
VOLUMIO_VERSION="myvers1"                                                       
VOLUMIO_HARDWARE="pi"                                                           
VOLUMIO_DEVICENAME="Raspberry Pi"                                               
+ set +x

But then

[ dbg ]  Node version installed:  [ v8.17.0 ]                                   
+ cat /etc/os-release                                                           
PRETTY_NAME="Raspbian GNU/Linux 10 (buster)"                                    
NAME="Raspbian GNU/Linux"                                                       
VERSION_ID="10"                                                                 
VERSION="10 (buster)"                                                           
VERSION_CODENAME=buster                                                         
ID=raspbian                                                                     
ID_LIKE=debian                                                                  
HOME_URL="http://www.raspbian.org/"                                             
SUPPORT_URL="http://www.raspbian.org/RaspbianForums"                            
BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"                           
+ set +x

which puts us here (edit) /usr/lib/os-release and the /etc/os-release symlink are owned by base-files. It gets upgraded from 10.3+deb10u7 to 10.3+rpi1+deb10u7 and neither of those files are marked as 'conffiles' so the package is not expecting modifications.

xipmix commented 3 years ago

I'm going to have to take a break from this for a bit, catch you later. Hopefully this is enough to find a solution.

ashthespy commented 3 years ago

@xipmix yep, the check was put in place as I ran into the same issue ;-) But I am curious how you run into it while building a fresh base rootfs (-b arm). I only ran into it while using a rootfs that had an older version of base-files, and then gets updated, wiping out our changes. Are you sure that you run into this with a fresh rootfs?

On a side note, I've been advocating writing our bits and bobs to a different file, -- similar to /etc/debian_version

xipmix commented 3 years ago

Definitely had a clean rootfs; removed everything in build/ before starting. The initial version suggests the package is coming from a different repository. I really don't understand why packages are being pulled from both archive.raspbian.org and raspbian.raspberrypi.org, but I checked the Packages file from both and they both have the right version (with +rpi1). The initial version I have, somehow, appears to be coming from deb.debian.org.

recipes/base/VolumioBase.conf pulls base-files from deb.d.o, while recipes/base/arm-buster.conf pulls from archive.r.o. Looking into build.sh I think I see the problem. The file pointed to by RASPBIANCONF does not exist.

(edit) Hm. actually the file is not supposed to exist, it gets created right there. But perhaps the sed command is not doing the replacement correctly. But once it gets created, does it ever get used? I don't see that. Perhaps that's the problem?

ashthespy commented 3 years ago

recipes/base/VolumioBase.conf pulls base-files from deb.d.o, while recipes/base/arm-buster.conf pulls from archive.r.o. Looking into build.sh I think I see the problem. The file pointed to by RASPBIANCONF does not exist.

Packages aren't being pulled from Debian because the armhf arch of Debian won't work on early Rasberry Pis (VFP2 vs VFP3).. So instead we patch the sources "on the fly" to point to raspbian.

But once it gets created, does it ever get used? I don't see that. Perhaps that's the problem?

It's being used here. I really really doubt the packaging is the issue, as otherwise people would have reported errors while trying to boot the test images on Pi0s ;-)

Definitely had a clean rootfs; removed everything in build/ before starting.

Could you share a full build log to pastebin or something? There is definitely something fishy going on if even a full build with -b flag is failing. What makes it more interesting is that images are currently being built with no issues on other machines.. and we should all be pulling for the same (uptodate) repos..

xipmix commented 3 years ago

It's being used here.

Yeah, I found my way there eventually. The code looks ok and like it should all work. I'm puzzled about the use case for the USE_MULTISTRAP_CASCADE variable, but I do think it's selecting the correct CONF file.

Could you share a full build log to pastebin or something?

Unfortunately I can't at the moment, on the road. I'll run some experiments when I get back. Probably best to ignore this issue until I can do that and send the log.

ashthespy commented 3 years ago

I'm puzzled about the use case for the USE_MULTISTRAP_CASCADE variable, but I do think it's selecting the correct CONF file.

That was to ease with A/B testing while I was streamlining the multistrap config. Previously, each Base had a separate .conf file with all the packages. This meant we were maintaining 4 sets of configurations that essentially differed only in the architecture. So, by using the cascade option of multistrap, we avoid that by using a single VolumioBase.conf that is a central location for all packages. Each architecture specific config then pulls in this, and adds any additional packages (if required).

Now that things are working as expected, I will remove this and the redundant *-buster.conf files and this variable to reduce confusion.

xipmix commented 3 years ago

I figured this out, it's a furphy. I've been using apt-cacher-ng to reduce the amount of re-downloading of packages. To handle https sources, I have to rewrite the .conf urls like this

- source=https://deb.debian.org
+ source=http://localhost:3142/HTTPS///deb.debian.org

which obviously breaks the rewriting process above. I do this with a little bit of shell in a file that I source before running build.sh.

I had an idea about how to support this kind of usage, which might improve the code as well; instead of (re)writing files in scripts orrecipes and then having to undo everything, one could copy the files to build/(scripts|recipes) and modify as necessary, then tell multistrap to use those files. Does that sound like something that would be acceptable?

ashthespy commented 3 years ago

Ah, glad you found the issue!

I've been playing around with caching as well, and would suggest checking out the apt.conf.d/ method with the Acquire::http::Proxy instead.. That way you don't need to rewrite the sources..

But of course, with https, your server will probably just tunnel it through, making all this moot.. So we might want to relook at https repos..