Planned updates for 17.10 / 18.04?

[stacefauske]

From @kamwoods on January 19, 2018 23:28

First off, great tool. I've been using it for a while. Future development question: I've tried BodhiBuilder 2.2.7 on 17.10 and 18.04 (current daily build as of 1/19/2018), and while both the GUI front and and ISO build work as expected, Ubiquity crashes on attempted install almost immediately after stepping through the dialogs.

I'm happy to dig into this myself, but thought I'd ask (since you appear to be the sole maintainer) - have you experimented with these versions, and do you know why this might be happening?

_Copied from original issue: stacefauske/bodhibuilder2.2.x#9

[stacefauske]

I haven't tested it on those versions yet at all. Since a lot can happen during alpha and beta testing (and because of my personal time commitments ), I probably won't dig into this until February. But it's definitely on the to-do list. If you're familiar with ubiquity, please feel free to take a look and post your findings if you like. Thanks, sef

[stacefauske]

From @Austcool-Walker on February 1, 2018 0:3

I have the same issue on Ubuntu 18.04 bodhibuilder v2.2.7

[kamwoods]

Just started looking into this again. The ubiquity crash occurs as it starts to copy files over (after you've clicked through most of the dialogs. The traceback in the log includes the following:

Feb 25 20:26:57 hostname /install.py [6511]: Exception during installation: Feb 25 20:26:57 hostname /install.py [6511]: Traceback (most recent call last): Feb 25 20:26:57 hostname /install.py [6511]: File "/usr/share/ubiquity/install.py", line 757, in Feb 25 20:26:57 hostname /install.py [6511]: install.run() Feb 25 20:26:57 hostname /install.py [6511]: File "/usr/share/ubiquity/install.py", line 135, in run Feb 25 20:26:57 hostname /install.py [6511]: self.copy_all() Feb 25 20:26:57 hostname /install.py [6511]: File "/usr/share/ubiquity/install.py", line 469, in copy_all Feb 25 20:26:57 hostname /install.py [6511]: self.db, sourcepath, targetpath, md5_check) Feb 25 20:26:57 hostname /install.py [6511]: File "/usr/lib/ubiquity/ubiquity/install_misc.py", line 754, in copy_file Feb 25 20:26:57 hostname /install.py [6511]: with open(targetpath, 'wb') as targetfs: Feb 25 20:26:57 hostname /install.py [6511]: OSError: [Errno 26] Text file busy: '/target/swapfile'

This happens in both 17.10 and the current daily builds of 18.04. I assume it's an artifact of Ubuntu having moved to a swap file rather than a partition in these releases, and that this exception is a consequence of some other issue with the respin/environment setup. I did try generating a new deb with the various efi and pool files replaced with the relevant ones from the current 18.04 ISO, with the same results.

[kamwoods]

Removed my previous comment as it was inaccurate. Revised:

If, prior to respin, I disable swap on the installed 18.04LTS system with:

sudo swapoff -a sudo rm -f /swapfile

and remove the appropriate entry from /etc/fstab, the respin produces an ISO that does not cause ubiquity to crash on install. However, after booting to the live desktop with the new ISO and attempting to install, I get a non-fatal error (pop-up window message) at the end of the install about not being able to get apt packages from the CD-ROM, and the system is subsequently unbootable after dismissing this message and clicking the reboot prompt.

[kamwoods]

A simpler (temporary) approach is just to add "/swapfile" to the EXCLUDES directive in bodhibuilder.conf. But the other problem remains; after respin (using my fork which updates the various debs and script references, although this doesn't seem to matter at the time of writing), attempting to install with the ISO produces the same apt setup failure error near the end of the install, also logged in /var/log/syslog:

Mar 20 00:23:43 bcadmin ubiquity: umount: /target/cdrom: not mounted. Mar 20 00:23:43 bcadmin /plugininstall.py: log-output -t ubiquity umount /target/cdrom Mar 20 00:23:43 bcadmin /plugininstall.py: Exception during installation: Mar 20 00:23:43 bcadmin /plugininstall.py: Traceback (most recent call last): Mar 20 00:23:43 bcadmin /plugininstall.py: File "/usr/share/ubiquity/plugininstall.py", line 1729, in Mar 20 00:23:43 bcadmin /plugininstall.py: install.run() Mar 20 00:23:43 bcadmin /plugininstall.py: File "/usr/share/ubiquity/plugininstall.py", line 61, in wrapper Mar 20 00:23:43 bcadmin /plugininstall.py: func(self) Mar 20 00:23:43 bcadmin /plugininstall.py: File "/usr/share/ubiquity/plugininstall.py", line 188, in run Mar 20 00:23:43 bcadmin /plugininstall.py: self.configure_apt() Mar 20 00:23:43 bcadmin /plugininstall.py: File "/usr/share/ubiquity/plugininstall.py", line 591, in configure_apt Mar 20 00:23:43 bcadmin /plugininstall.py: "AptSetup failed with code %d" % ret) Mar 20 00:23:43 bcadmin /plugininstall.py: ubiquity.install_misc.InstallStepError: AptSetup failed with code 141

From looking around online this seems to be some cascade from the ISO build process rather than a ubiquity bug, but I haven't made it any further yet.

[kamwoods]

I now have a "fixed" version that (at least for Backup) cleanly creates a bootable ISO for 18.04LTS in my fork. The issue above seems to have been related to the fact that the dists and pool directories were not being created in ISOTMP; for the time being I just added some mkdir -p lines for them, but there's quite a bit of other legacy stuff that probably needs fixing, too. Going to continue working on it for the time being...

[muucouple]

@kamwoods, looking for trying Your fork...

So, I should first install bodhibuilder_2.2.7_all.deb and after that replace the installed files with the ones on Your repository?

Thanks in advance for your help

[kamwoods]

@muucouple Simpler just to build the fork directly if you want to test it. Run "dpkg-deb --build bodhibuilder_2.18.x_all" in a terminal after you have cloned it out, then "sudo dpkg -i bodhibuidler_2.18.x_all". Keep in mind that my fork is temporary, and work-in-progress.

[muucouple]

@kamwoods Perfect... It seems to be working (at least booting and Installing to a new machine in VirtualBox)

When installing sudo dpkg -i bodhibuidler_2.18.x_all.deb i get this warnings: chown: cant access '/var/log/dmesg': file or directory doesn't exist chmod: cant access '/var/log/dmesg': file or directory doesn't exist

In the first run it generated an ISO of 36KB, to fix this I had to do a: sudo apt-get install isolinux syslinux-utils

After that ISO generated OK

When executing bodhibuilder-gtk I got this messages: id: <<999>>: user doesnt exist id: <<999>>: user doesnt exist GID & UID 999 are not in use, OK

The install icon is missing in the live-session (I think this is my fault so don't sweat it over this) so i have to manually run: sudo ubiquity gtk_ui to install

[kamwoods]

Thanks for the feedback. I'm also testing in VirtualBox. Ubuntu doesn't keep an explicit /var/log/dmesg any more. The lines to check/write to it should presumably be removed; that's on my list. And yeah, isolinux and syslinux-utils should be installed.

Odd about the icon. It appears on my desktop, but it's not set to "trusted" (another fix needed - I'm not sure what to do about this since it requires setting with gio and is store in gvfs-metadata, which is deleted when the user is copied over for the "Backup" option). When running from the command line, those messages about id 999 are just bb checking to make sure there's no active user at 999, that's not an error.

Not sure when Stace is going to start up on this again; for now I'm just going to keep tweaking in my fork and hopefully that will be useful when they're ready to update, accept a pull request, or selectively apply some of these changes.

[stacefauske]

Thank you for your testing in backup mode! I have been testing in dist mode (as this is our highest priority) the past couple of days and have come across the same issues. I will implement some of your changes and continue with the dist mode tests.

[muucouple]

@kamwoods @stacefauske My observations where in dist mode... I have not tried installing directly in a machine but I'll let You know how it goes. At least in a new VBox machine it did install without issues.

[kamwoods]

@stacefauske I've also noticed that autologin is not restored when installing from an ISO built in Backup mode, even if it set for the appropriate user prior to building. To be clear - the live user autologs in as expected, but the backed up user does not, after install. Relevant details: in my test there's a custom.conf file in the /etc/gdm3 directory (both in the installed system and in dummysys) that has "autologin=true" for the relevant user to be backed up. After building the ISO and installing from that ISO, this file is gone. I have been digging through the code and looking around online, and for the life of me I can't figure out why this is. (Note - this definitely previously worked for 16.04-appropriate builds of bodhibuilder).

[kamwoods]

Ah, I realized that I was removing custom.conf in apt-setup.bb. Updated it in my branch. This leaves only one thing: trust for desktop launchers. I can't find any way to make "ubiquity.desktop" trusted (so that the proper icon appears and you don't get the trust and launch popup) on the ISO.

[kamwoods]

@stacefauske FYI I "solved" the desktop launcher trust issue (particularly for ubiquity.desktop) by creating a launcher in ~/.config/autostart for the live user that runs a script that on first login executes 'gio set ~/Desktop/ubiquity.desktop "metadata::trusted" yes'. The launcher is then trusted for the live user when booting from the ISO, the correct icon shows up, and no "trust this" dialog is triggered.

I've seen some references elsewhere to similar workarounds. This is ugly, but I haven't been able to get anything else to work. I'm curious to know if you have a better, more sensible solution.

[stacefauske]

Which desktop are you running? When I make a spin with the gnome 3 desktop, the ISO is so bogged down that it barely runs. I can install from the ISO, but it takes a looong time. However, when I run lubuntu 18.04, I have no trouble at all creating or installing from the ISO.

Edit: Also, which version of virtualbox are you running?

[kamwoods]

@stacefauske Right now, I'm using the current stock daily build of Ubuntu 18.04LTS (http://cdimage.ubuntu.com/daily-live/current/) in VirtualBox 5.2.10 (with Extension Pack). So, Gnome 3. I've been successfully testing the code that's in my fork for several weeks now with various daily builds, and both VirtualBox 5.2.8 and 5.2.10 on Linux and Mac.

You can find an example of an ISO I created a week or two back at:

http://distro.ibiblio.org/bitcurator/BitCurator-2.0.0-beta5.iso

if you're interested in seeing if you have the same problem with an ISO that seems to work ok for me. (Note that this ISO is was made using the current setup in my fork of bodhibuilder, the install icon has just been swapped). Also note: this ISO takes about 1 minute to boot in VirtualBox 5.2.10 on my Macbook 12" (non-Pro) i5 with 2 procs, 2GB RAM assigned and 128MB Vram assigned. So, not fast, but when it's booted everything is relatively smooth and usable, and it boots a bit faster when installed.

Some additional details that might or might not be useful:

My testing is as follows: Install VirtualBox 5.2.10 and host Extension Pack. Make a new VM in for Ubuntu 64-bit, with 2 processors assigned, 128MB video RAM assigned, and 2-4GB RAM (it seems to boot and run fine with 2GB as well, just a bit slower).

I start the VM and install from the daily ISO. Reboot as prompted after the install and do a sudo apt-get update, install dkms, and upgrade. Then I install git with apt. Then I git clone my fork, run "dpkg-deb --build bodhibuilder_2.18.x_all", run "dpkg -i bodhibuilder_2.18.x_all".

Then I customize my environment and run BB as usual and make the ISO in Backup mode. I have not tested dist mode. Installing in a new VM works fine for me with that ISO (the live ISO also seems reasonably responsive, no less responsive than the old 16.04LTS builds). FYI I have tested in VirtualBox 5.2.8 and 5.2.10 on an Intel NUC i5 with 8GB RAM running Ubuntu 18.04 (daily build, as the host) and and Macbook 12 i5 8GB RAM running 10.13.4. I've also done a bare metal install on the NUC and on a Dell OptiPlex 7050 i7 with 16GB RAM and some sort of actual video card. This is obviously all newer hardware, and they all have SSDs, but nothing too high end.

[stacefauske]

Thanks for the info and the input!

I'm not able to even boot the 18.04 daily build anymore in VB 5.2.10. Tried multiple downloads and multiple boots of each. A little close to release date for something like this to be happening. I think there is something weird with gdm3. When I replaced it with lightdm on the daily build (before installing Lubuntu) I had more success with booting the resulting bodhibuilder ISO. Only thing is, it wouldn't auto-login even the live user, but I haven't had time to troubleshoot down that path yet. Now I've installed Lubuntu and can't re-install the regular Ubuntu daily because today's ISO won't boot. Frustrating. And the only other daily ISO I have is from back in the beginning of Feb.

Anyway, here is a deb that I've spun up with some of your changes, but not all of them. Would be interested to see how it works for you in the Ubuntu daily if you would be willing to try it out. But it won't surprise me if it doesn't work with your system since I haven't been having the same issues you've been experiencing.

Specifically, I'm not having the desktop launcher issue you're talking about with ubiquity. Whenever I hit the Install icon, it just runs. But, if you want to add any custom commands that will only run once upon the initial boot of the newly created system, you can put them in the file "/etc/bodhibuilder/firstboot/file-firstboot". They'll only run once and never again. If you need to run the command on every boot of the system you could always put it in /etc/rc.local. Each new LTS release seems to have it's set of fun things to work around and get used to. If I had more time I could read the release notes a little closer, but time is a precious commodity lately. :) Thanks again.

EDIT: Are you running the Wayland or Xorg desktop?

[stacefauske]

I am still not able to get the Ubuntu 18.04 daily build to boot in VirtualBox, so I have been doing extensive testing with the Lubuntu 18.04 daily build. In the Lubuntu daily, I have had success with both the 64 and 32 bit versions, doing both online and offline installs in VirtualBox. Here is the download link for the first release of bodhibuilder_2.18.0_all. Please download and install for testing in Lubuntu. If you are able to test it in Ubuntu, please also consider doing that and reporting back the findings. As a note: the only extra software I've installed in the Lubuntu images I've tested, is what is required by bodhibuilder, and "virtualbox-guest-utils & virtualbox-guest-x11" (& their dependencies). I'll leave this issue open for comments for a few days and then consider closing based on the comments. Thanks!

[kamwoods]

@stacefauske Thanks for all the follow-up. In response to your previous comment - my installs are all default with the daily 18.04LTS ISOs, so Xorg + gdm3. I'll test with your new build and try to report back soon.

[kamwoods]

@stacefauske Got a bit held getting back to this. As of 5/14/2018, both your current release and my fork have the same (failing) behavior for me. Using a stock Ubuntu 18.04LTS install, they both generate (in Backup mode) a bootable ISO. That ISO gets part way through an install in a clean VirtualBox VM and then fails out with a "Permission Denied" error. Looking through syslog it appears to be something related to failing to write out one or more temporary files due to a permission error on a directory. I thought this might be due to a regression in ubiquity that I'd seen mentioned elsewhere, but even after shifting to xenial-proposed in a new build and using the more updated ubiquity there, the problem persists.

[kamwoods]

@stacefauske Final note - I was unable to track down all the changes required to get this working with stock Ubuntu 18.04LTS on my boxes, but the folks at Pinguy did. The current PinguyBuilder code at https://github.com/pinguy/Pinguy-Builder works for me.

[stacefauske]

I finally found a random forum thread that talked about the same permission errors with Ubiquity. He mentions the same problems until he installs an "ubiquity-slideshow-(something)" package. I installed "ubiquity-slideshow-lubuntu" and the permission errors go away, ISO installs successfully to a VBox VM. Strange. Going to have to clean up my code due to the many troubleshooting things I tried along the way then I'll upload a 2.18.1 version. Also going to have to implement a check for a "slideshow" too. Strange.

[stacefauske]

Please try version 2.18.1 here

[kamwoods]

Yes, works for me. I tested by doing a dpkg-deb build from your source, not the prebuilt, but I'm assuming they're the same.

Minor caveat: (this also applies to the PinguyBuilder variant at https://github.com/pinguy/Pinguy-Builder). If I create the ISO in Backup mode with a pre-existing user that has autologin set to true, and then install from that ISO, the autologin is subsequently disabled - that is, a password is required on first boot after install. Maybe this doesn't matter for Bodhibuilder, but retention of autologin is a behavior that used to carry over on install from a Backup-created ISO with previous iterations.

[stacefauske]

Hi Kam, Thanks for the feedback. Yes, as of right now they are the same, but won't be for long since we're trying to get offline installs working as well. Would you mind please opening another issue related to the autologin? This thread is getting a bit long, and since online (internet connected) installs are working, I'd like to close out this issue. Thanks, sef

[stacefauske]

Hi Kam, I have created a new issue "Backup mode - autologin not working on installed system" that quotes your previous entry. If any more details are required, please make an entry in that issue. Or if there are any other distinct issues, please create a new one with the details. Thanks! stace

Closing.