search

volumio / Build

Buildscripts for Volumio System
GNU General Public License v2.0
113 stars 102 forks source link

Building for Debian buster (10) #388

Closed arthurlutz closed 3 years ago

arthurlutz commented 5 years ago

Since #259 never made it while debian stretch was the current stable, here is a follow up for Debian Buster.

Being two releases behind might be considered relatively urgent in terms of security by some, eventough there is LTS support by debian until june 2020 https://wiki.debian.org/LTS/Jessie

gkkpch commented 5 years ago

We're aware of the urgency and also welcome any support we can get. Apart from the usual package version and dependency upgrades, we seem to have hit an issue with building our initramfs. This is still under investigation.

gkkpch commented 5 years ago

The first part of the work more or less copied the efforts for Debian stretch, changing package versions in the recipe file, updating library versions in mkinitramfs-custom.sh, etc. But several new issues were discovered:

Status So far I have only modified the build for X86 as this is the one I am most familiar with. It is still WIP and the build process has not been tested on a Debian buster host yet (still using jessie). My goal is to have it running on the newest Debian and Ubuntu hosts. Also see my last comment in #151, which resolves a longstanding mkinitramfs issue.

Next steps

gkkpch commented 4 years ago

Status update First milestone reached: a booting Volumio image for X86. Currently via legacy BIOS, I had no time yet to test UEFI (which I expect should work). Basic features are functional (UI, Web Radio, play from NAS). Still a lot of work to do, but we're now further than we ever were with Stretch...

Next steps

Saiyato commented 4 years ago

Hi @gkkpch,

I'm not too savvy when it comes to compiling sources/making images, but if there's something I (or someone else from the community) can do to test/help for the Pi platform, let us know ;)

I have all Pi's in the B-family available for testing and when I find myself a cheap Zero, that one as well ;) Also several DAC's, a DIGI and AMP should it be needed.

PS: my incentive for Buster is both security and new libraries for functionality (Kodi for example :P) PPS: with some guidance I can be taught to compile/build for myself for testing purposes.

gkkpch commented 4 years ago

It is not ready for PI yet, there is still some work to be done on mkinitramfs_volumio.sh to make it suitable for the different kernels in a PI image. I would like to finish the X86 version first. This one is most urgent because of the firmware issues with newer hardware. For the PI I can use help later on. Ik hou je hier op de hoogte ;)

xipmix commented 4 years ago

@Saiyato have you tried a build? I have been able to cross-build for pi on x86 without much effort. What step do you get stuck on? Let's see if we can get you going while gkkpch is sorting out x86

gkkpch commented 4 years ago

@xipmix: and it worked? It would be very interesting to see your results (recipe file, build script, boot log) as our first attempts with PI failed miserably. Which host linux distro did you cross-compile on? X86 is working fine, tested on 6 different hw configurations so far. Still waiting for the volumio-specific packages to be compiled, but apart from that an alpha version for X86 is very close. As mentioned earlier, our customized mkinitramfs has not been extended yet with the RPi requirements.

xipmix commented 4 years ago

Sorry, I realize you are talking buster here. I just wanted to suggest building (for jessie anyway) isn't so scary. I will poke a bit at buster and let you know if I make any progress. I assume you are working on the build-buster branch?

gkkpch commented 4 years ago

nope, I use buster-new, build-buster is the failing one. This will (should) be removed. Again, note: mkinitrams-volumio.sh does not work yet for multiple kernels and the kernel versions are not detected not from within either. I specified the version because it was easier for me to play with

biva commented 4 years ago

Some encouragement for @gkkpch to succeed in killing the last bugs :) sorry for not being skilled to help here, but you have supporters :)

gkkpch commented 4 years ago

I'll push today's new version of mkinitramfs-volumio.sh, which replaces mkinitramfs-custom.sh It will be in the "buster-new" branch, I have not been able to test it (x86 only has 1 kernel), just let me know when anytime fails with RPi. No new progress for x86 yet, waiting for some volumio-specific issues to be resolved plus, most important, kernel 5.4 LTS to be released this weekend...

gkkpch commented 4 years ago

Not sure how you want to tackle the PI build, let Michelangelo know before you start, I'll help wherever I can

gkkpch commented 4 years ago

The mkinitramfs-volumio.sh script was renamed back to mkinitramfs-custom.sh to avoid confusion. It now also works for multiple kernels which PI images have. The buster recipe files for arm and armv7 have been prepared too and some minor changes to volumioconfig.sh and configure.sh have been done to allow building images. There are a number of volumioconfig script issues with buster, which need to be checked, but these do not seem to harm the initial work. The Volumio "Build" repo has been updated, please use branch "buster-new" Building an amrv7 rootfs and an image for the Odroid N2 completes without build issues, not tested yet whether it actually works. It was only to show that the basics are there.

So, any volunteers for a PI build?

Saiyato commented 4 years ago

I've been kept quite busy, but I've set apart some time to work on some plugins again this weekend. I'll try to read into building for ARM, this will be my first time; so be gentle :P

I've been planning to upgrade my main Pi for some time now, so if the build succeeds I'm willing to test on production (used daily for several hours). On a new SD card obviously ;) This will test the following plugins as well: LastFM, SnapCast, pydPiper, volspotconnect and some others

Saiyato commented 4 years ago

I tried building for the Pi, but I'm not sure I tried correctly. Anyone able to assist? :) The following steps were followed:

  1. Install requirements
  2. Patch build.sh (add the three lines)
  3. Clone build repo
  4. Build system and image in one and two go's using the following command for the single go: ./build.sh -b arm -d pi -v 2.0 (I couldn't find a reason to define a specific version for now, it's only used for the name, am I right?)

I built the image on a Ubuntu VM (using the Hyper-V hypervisor). The images were around 3GB, which is fine, but I'm missing the remoteConfig file in 1.img and my volumio_current.sqsh is 16 times smaller than on the orginal image one. So I'm pretty sure that's my problem. Sifting through the logs I see the following: "The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 9165938D90FDDD2E"

Does that mean I can't build on my VM? I'm pretty much a n00b when it comes to building anything on Linux, spoiled by IDE ease-of-use. Sorry ;)

gkkpch commented 4 years ago

Yeah, currently multistrap on Ubuntuj does not work correctly, but there is a solution. Go to https://github.com/volumio/Build/tree/buster-new README.md has a section for Building on Ubuntu and Debian Buster. The instructions for a small modification of build.sh are there...

gkkpch commented 4 years ago

and in case you get stuck, send me a forum PM when you wish to chat via Whatsapp

Saiyato commented 4 years ago

I'm getting somewhere now, in the extra instructions there's a semi-colon on the wrong side of the double quotes for the multistrap perl file.

It says

AllowUnauthenticated=true;"

but it should be: AllowUnauthenticated=true";

I was getting errors because of premature ending of function.

Saiyato commented 4 years ago

Thanks for the help so far, it looks like I'm going to succeed in building now... fingers crossed

gkkpch commented 4 years ago

A bit later than planned, there is now a restructured branch BuildBusterV1. Please use this one for the base of all further buster development. All other "buster" branches are obsolete as of today and will be removed shortly.

BuildBusterV1 allows to build for jessie as well as buster, which might come handy when comparing the buster with the jessie version. The main reason however, is to allow buster and jessie to coexist for a while in the same build process without risking regression. For that reason, the scripts and resources for "jessie" and "buster" are kept strictly separate (except build.sh). A build defaults to "jessie" unless the "-s buster" parameters are added when building the architecture with the "-b " parameters. e.g.

sudo ./build.sh -b x86 -s buster -d x86 -v -myfirst

Currently we have support for X86 with Debian buster. This is going to be alpha in an week or so (minor bugs to be ironed out).

Make sure not to modify "jessie" build scripts, only edit those in scripts-buster. Copy the script from "scripts" (jessie) to the scripts-buster folder and edit.

volumio commented 4 years ago

Great. IMHO to progress even faster, when you think it's ok to merge, we should merge to streamline work from there

gkkpch commented 4 years ago

yes, fully agree to keep the time in the branch as short as possible. As soon as alpha testing is done, we should merge. There is little risk as we still have the possibility to build jessie without causing regression. After each merge, we should create a fresh BuildBuster branch to ensure we have the latest before we start the next part.

x86 took a lot of time because of the initial mkinitramfs effort, the big gap between the old kernel and firmware resources and the custom packages. Once this is tested properly, RPI should go fairly quick, it mainly takes custom packages, the rest of the sbc's will follow with minor modifications (if at all).

ashthespy commented 4 years ago

I wonder why I didn't see the updates in this issue earlier, thought I had subscribed to it.. :open_mouth: Either-way, I have been running an extremely lean buster version on my fruity boards (opi/nanopi) for some time now - it currently uses the upstream node/mpd versions, but seems to work for my use thus far. I can clean up my local branch and merge it into BuildBusterV1 in a few days if you'd like? Of course things like volumio-remote-updater aren't included, but the basic functionality seems to work.

gkkpch commented 4 years ago

make sure you put modified scripts in their buster folder, modifications in jessie scripts or files will not be merged. Look at what we have in recipes-buster, scripts-buster and volumio-buster. You’ll see what I mean.

ashthespy commented 4 years ago

My local fork has diverged quite a bit from current master - I took some time to refactor and reorganise the build system so that it doesn't just run through and spit out an image! :sweat_smile: Further to armv7, I also have armv8 builds with buster and Volumio "working" but like I said not extensively tested. I will work towards integrating these into ~master~ this repo - but would you guys be open to a more modular build system?

The mkinitramfs-volumio.sh script was renamed back to mkinitramfs-custom.sh to avoid confusion. It now also works for multiple kernels which PI images have. The buster recipe files for arm and armv7 have been prepared too and some minor changes to volumioconfig.sh and configure.sh have been done to allow building images. There are a number of volumioconfig script issues with buster, which need to be checked, but these do not seem to harm the initial work.

@gkkpch Which branch has the right mkinitramfs-custom.sh for the multiple kernel versions for the pi? Currently the build fails (using BuildBusterV1)

Creating initramfs
Version: 4.19.86+
/bin/cat: /var/tmp/mkinitramfs_mDivPu/lib/modules/4.19.86+/modules.builtin: No such file or directory
depmod: WARNING: could not open modules.order at /var/tmp/mkinitramfs_mDivPu/lib/modules/4.19.86+: No such file or directory
depmod: WARNING: could not open modules.builtin at /var/tmp/mkinitramfs_mDivPu/lib/modules/4.19.86+: No such file or directory
Version: 4.19.86-v7+
/bin/cat: /var/tmp/mkinitramfs_ekXFlx/lib/modules/4.19.86-v7+/modules.builtin: No such file or directory
depmod: WARNING: could not open modules.order at /var/tmp/mkinitramfs_ekXFlx/lib/modules/4.19.86-v7+: No such file or directory
depmod: WARNING: could not open modules.builtin at /var/tmp/mkinitramfs_ekXFlx/lib/modules/4.19.86-v7+: No such file or directory
Version: 4.19.86-v7l+
/bin/cat: /var/tmp/mkinitramfs_ekXFlx/lib/modules/4.19.86-v7l+/modules.builtin: No such file or directory
/bin/ln: failed to create symbolic link '/var/tmp/mkinitramfs_ekXFlx/etc/mtab': File exists
/bin/ln: failed to create symbolic link '/var/tmp/mkinitramfs_ekXFlx/sbin/mount.ntfs-3g': File exists
E: /usr/share/initramfs-tools/hooks/ntfs_3g failed with return 1.
gkkpch commented 4 years ago

Sorry, your cannot push to master, I mentioned that before. You really need to use the BuildBusterV1 branch.

On 7 Feb 2020, at 21:06, Ash notifications@github.com wrote:

My local fork has diverged quite a bit from current master - I took some time to refactor and reorganise the build system so that it doesn't just run through and spit out an image! 😅 Further to armv7, I also have armv8 builds with buster and Volumio "working" but like I said not extensively tested. I will work towards integrating these into master - but would you guys be open to a more modular build system?

So, any volunteers for a PI build?

This now gets interesting -

Creating initramfs Version: 4.19.86+ /bin/cat: /var/tmp/mkinitramfs_mDivPu/lib/modules/4.19.86+/modules.builtin: No such file or directory depmod: WARNING: could not open modules.order at /var/tmp/mkinitramfs_mDivPu/lib/modules/4.19.86+: No such file or directory depmod: WARNING: could not open modules.builtin at /var/tmp/mkinitramfs_mDivPu/lib/modules/4.19.86+: No such file or directory Version: 4.19.86-v7+ /bin/cat: /var/tmp/mkinitramfs_ekXFlx/lib/modules/4.19.86-v7+/modules.builtin: No such file or directory depmod: WARNING: could not open modules.order at /var/tmp/mkinitramfs_ekXFlx/lib/modules/4.19.86-v7+: No such file or directory depmod: WARNING: could not open modules.builtin at /var/tmp/mkinitramfs_ekXFlx/lib/modules/4.19.86-v7+: No such file or directory Version: 4.19.86-v7l+ /bin/cat: /var/tmp/mkinitramfs_ekXFlx/lib/modules/4.19.86-v7l+/modules.builtin: No such file or directory /bin/ln: failed to create symbolic link '/var/tmp/mkinitramfs_ekXFlx/etc/mtab': File exists /bin/ln: failed to create symbolic link '/var/tmp/mkinitramfs_ekXFlx/sbin/mount.ntfs-3g': File exists E: /usr/share/initramfs-tools/hooks/ntfs_3g failed with return 1. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

gkkpch commented 4 years ago

@ashthespy : could you please get in touch with me at volumio(at)bluewin.ch?

ashthespy commented 4 years ago

Sorry, your cannot push to master, I mentioned that before. You really need to use the BuildBusterV1 branch.

Ooops, I mean this repo, not master in particular. Either way - after ~much~ some more tweaking 

Raspbian GNU/Linux 10 volumio ttyAMA0

volumio login: volumio
Password:
Linux volumio 4.19.86-v7+ #1283 SMP Fri Nov 29 18:32:25 GMT 2019 armv7l
                       ___
                      /\_ \                        __
         __  __    ___\//\ \    __  __    ___ ___ /\_\    ___
        /\ \/\ \  / __`\\ \ \  /\ \/\ \ /' __` __`\/\ \  / __`\
        \ \ \_/ |/\ \L\ \\_\ \_\ \ \_\ \/\ \/\ \/\ \ \ \/\ \L\ \
         \ \___/ \ \____//\____\\ \____/\ \_\ \_\ \_\ \_\ \____/
          \/__/   \/___/ \/____/ \/___/  \/_/\/_/\/_/\/_/\/___/

             Free Audiophile Linux Music Player - Version 2.0

          C 2015 Michelangelo Guarise - Volumio Team - Volumio.org

Volumio Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.

could you please get in touch with me?

Will do!

volumio commented 4 years ago

Amazing! Thanks ash!

ashthespy commented 4 years ago

I haven't been finding the time to get on with the merging into this repo, but if people want to debug/test for actual Buster issues, rather than build issues ~Raspberry Pi Volumio-3.000-2020-02-19-pi~ ~Raspberry Pi Volumio-3.001-2020-02-19-pi~ ~Raspberry Pi Volumio-3.003-2020-02-25-pi~ Raspberry Pi Volumio-3.004-2020-03-02-pi

Known issues

gkkpch commented 4 years ago

nice work! @volumio: since when is Volumio 3 default??

ashthespy commented 4 years ago

@gkkpch Since https://github.com/volumio/Volumio2/commit/a864ffd84c384cd470cf8f7eba9979c24dac9bad -- it messed with me as well. Maybe we should tag a release branch and pull from that, similar to UI to avoid issues

volumio commented 4 years ago

Since yesterday Volumio3 UI is default, but you can change it to classic one from appearance ;) Thanks for the work so far, I will use your image to compile the missing packages

gkkpch commented 4 years ago

@volumio: Is it just for testing at the moment or is a announcement upcoming? Loads of people aren't aware of Volumio 3, anyway, I'll change my numbering to 3.00x to be more specific.

volumio commented 4 years ago

Nope, it's just a new UI ;) So, see it as a new feature for Volumio rather than a Volumio3 build.. So we can keep the numbering as it is now ;)

gkkpch commented 4 years ago

Then people should be informed about it I guess...

volumio commented 4 years ago

Yep, sure. It will be in the changelog. Do you think we should do something more?

gkkpch commented 4 years ago

Not so sure if this is not working or some other regression, but buster is now missing streaming services. I don't see mine (Tidal either)

gkkpch commented 4 years ago

confirmed by Robert, he does not see them either. With version 0.001 it was still working. (Robert does Intel Atom platform testing as I don't have one)

gkkpch commented 4 years ago

I have not listed newer X86 buster Issues here, because currently we are already a step ahead of the RPi version with the beta version. A link to the x86 discussion may help, some info certainly will be x86 specific, but this may change: https://forum.volumio.org/volumio-x86-debian-buster-debugging-party-beta-t13957.html

ashthespy commented 4 years ago

Not sure what changed in volumio/Volumio2 but the UI doesn't seem to want to load with today's build. The backend seems to be up, could navigate to the test page and enable ssh, but the UI was stuck at the while loading circle - I could however navigate via a different device on the network. ~I didn't really see anything suspicious from a cursory glance at the logs or web console.~ The front end seems to want to connect to the wrong interface (wlan0 instead of the default device) ifdown wlan0 and its up and running.

PS: I agree with Gé, lets track keep this issue specific to Build system issue for Buster, and split out Volumio/OS stuff to a new thread/issue..

ashthespy commented 3 years ago

I guess we can close this -- the build system specific buster quirks were ironed out quite a while back.. :-)

Saiyato commented 3 years ago

@ashthespy nice! Does this mean we're on track to make it official? :smile: I have been a bit busy, so I missed quite some forum msgs. Because this means I would need to look into some plugins I guess :wink:

ashthespy commented 3 years ago

Ah, this issue was mainly tracking stuff in the Build system/initramfs that was preventing us building Buster images.. For the actual migration, we have #401 (which is a bit stale though). Basically, everything pretty much works -- but needs a lot of testing! 😉