debian-8 lx dataset - apt-get upgrade broken

[noahmehl]

I noticed yesterday that when I provisioned a new machine with this dataset: 445d04f4-cad6-11e5-a1a0-9f6c0ce02707 that upgrading with apt-get has broken in relation to udev. https://gist.github.com/noahmehl/0a70bb20f292994d6b70a83c36670918

[david415]

For the record, I am also experiencing this problem and it's kind of a show stopper not being able to upgrade a debian linux zone.

[trentm]

@chorrell Have you seen this?

[chorrell]

No, I don't think I've seen this before.

[chorrell]

Also, it would be good to know what platform version is being used here.

[pfmooney]

@chorrell PI version is not likely to make a huge difference since, in general, we lack the necessary mechanisms to support udev in LX.

[noahmehl]

Are you looking for this? 20160303T211841Z

Also, I don't think that we want udev support, I think we want the apt process not to try and change udev. I suppose, I could always pin the udev package version? Perhaps this should be part of the original dataset?

[chorrell]

I'm not sure what the best approach is. The udev package is a dependency of systemd, so you can't just uninstall it as far as I can tell.

I also tried apt-mark hold udev but that doesn't seem to help either.

[chorrell]

Apt pinning the udev package seems to work. I created a file called udev in /etc/apt/preferences.d with the following contents:

Package: udev
Pin: release *
Pin-Priority: -1

Then I did an apt-get update; apt-get upgrade

I'm still reading up on apt pinning, but that seems to be a decent work around so far and something I should be able to incorporate into the debian-8 image.

I'm open to feedback though :)

[noahmehl]

@chorrell I'm totally fine with that. I lack a complete understanding of the LX dataset internals. Like, how does the original dataset get created, etc...? Anyways, I think pinning is fine, as it's (udev) is not really used? Anyways, this gets my support 👍

[chorrell]

OK cool, I'll work on an image update this week so that work around is backed in.

fwiw, this is how I build the lx-brand debian-8 (and debian-7) images: https://github.com/joyent/debian-lx-brand-image-builder

I create an Debian install tarball from a chroot installation in a running Debian KVM instance. I do an apt-get upgrade when creating the tarball, but since that's in a chroot it seems to not hit this issue.

[noahmehl]

Ah! I see now. Then yes, we'll probably just want to pin it before creating the dataset. I wonder if any of the ubuntu datasets have had this problem, and what they did to solve it?

[chorrell]

RE: ubuntu

This doesn't seem to affect Ubuntu 14.04 from what I can tell, but I'll most likely have to do something similar for 16.04

[noahmehl]

@chorrell you're the maintainer for all of these? I didn't realize....

[chorrell]

Yes!

[noahmehl]

@chorrell well, a general thank you then for all of your work :)

[chorrell]

Thanks! :)

[chorrell]

Hi!

I just released the following image to https://images.joyent.com that addresses this:

debian-8 20160421 257a5a32-0758-11e6-b5f1-5fea6d13e92f

https://docs.joyent.com/public-cloud/instances/infrastructure/images/debian#debian-8-20160421

[chorrell]

I made a mistake with that image. The udev fix didn't land. Curiously, my testing wasn't able to reproduce the apt-get upgrade issue...

[noahmehl]

So, sometimes, 'apt-get upgrade' is upgrading udev, and sometimes, 'apt-get full-upgrade' was doing it.

[chorrell]

Interesting.

Working on another image that has the udev apt pinning in place

[chorrell]

The debian-8 20160427 5ff8fd8a-0ca8-11e6-b8bd-cf9c395fb29d image was just released that addresses this. It has /etc/apt/preferences.d/udev to pin the udev package.

[farmergreg]

I think I have the same (or similar problem) with the current ubuntu image 675834a0-1267-11e6-81fe-2b1280d24244 released on 2016-05-05. Here's a log of what happens: https://gist.github.com/dietsche/4966450c8cb8c2db9d9d11c94c1bb71b

[chorrell]

I need to look into that. We're apt pinning udev in the Ubuntu 16.04 image but that doesn't seem to help in this case.

[chorrell]

I think removing the udev package is a possible work around

apt-get remove udev

[chorrell]

Another option is create /usr/sbin/policy-rc.d with:

#!/bin/sh  
exit 101

Make it executable:

chmod +x /usr/sbin/policy-rc.d

Do the upgrade, then:

rm -f /usr/sbin/policy-rc.d

That seemed to work for https://github.com/Microsoft/BashOnWindows/issues/143 which seems to be a pretty similar issue.

[farmergreg]

I went this route, and it seems to be working ok. The zone I tested in is a nginx / letsencrypt ssl reverse proxy with nothing else installed.

apt-get remove udev -y
apt-get autoremove

[chorrell]

Hmm, ok. Might be something I need to do in the Ubuntu 16.04 image then.

[farmergreg]

Well, just reading the bash thread, more carefully, and it sounds like Ubuntu's recommendation was what you posted. Maybe should consider going with that? I tried the one liner someone posted and it worked for me on a different vm:

cat > /usr/sbin/policy-rc.d <<EOF
#!/bin/sh
exit 101
EOF
chmod +x /usr/sbin/policy-rc.d
dpkg-divert --local --rename --add /sbin/initctl
ln -s /bin/true /sbin/initctl

[chorrell]

I think the best approach for the 16.04 image would be to just remove the udev package completely as part of the build. I'm going to test that and see how it goes.

[chorrell]

Testing looks good. I'll probably just remove udev from the debian-8 image as well

[chorrell]

Looks like we need to stick with pinning udev in Debian 8. apt-get remove udev in Debian 8 also results in removing systemd and some other things. I don't expect that to work out well...

[chorrell]

Just released a new Ubuntu 16.04 image with the udev package (and services) removed:

https://images.joyent.com/images/4edaa46a-1b83-11e6-b863-3fb0064133ee

[ismell]

does that mean systemd is removed? Was it ever used or was upstart used?

[chorrell]

No, systemd is still present in the 16.04 image I just released. The apt-get -y remove udev command didn't try to uninstall systemd, although it did remove console-setup, console-setup-linux and some other related packages.

For some reason in 16.04 that works without removing (or trying to remove systemd). In Debian 8 it will try and remove systemd and friends.

[eripa]

Any news on this for Debian 8? I'm running a couple of zones based on the image with UUID 3da6330e-388d-11e6-b41b-d766707c6c3d and they have all gone bad trying to update the udev package. I've been running apt-get update && apt-get dist-upgrade on a regular basis until they stopped working for some reason.

The package is pinned (nothing I've done, so I guess it's part of the image) as described above. So I'm not sure why this is happening.

I have a vague memory of me fixing this a couple of months ago, but I cannot seem to figure out a way to get out of this situation. Removing udev removes systemd, which then tries to add udev back when I reinstall it.

Thanks for any help on this..

Setting up udev (215-17+deb8u5) ...
addgroup: The group `input' already exists as a system group. Exiting.
udev does not support containers, not started.
insserv: Service udev has to be enabled to start service udev-finish
insserv: exiting now!
update-rc.d: error: insserv rejected the script header
dpkg: error processing package udev (--configure):
 subprocess installed post-installation script returned error exit status 1
Errors were encountered while processing:
 udev
E: Sub-process /usr/bin/dpkg returned an error code (1)

[chorrell]

Does this only happen with dist-upgrade and not upgrade?

The udev package is pinned to prevent this when doing an apt-get upgrade.

[noahmehl]

@chorrell Is there any general guidance on dist-upgrade in relation to LX zones? Just wondering if this is documented somewhere.

[chorrell]

No, not that I know of.

[razamatan]

wouldn't it be easier to just not use systemd for debian?

[noahmehl]

@razamatan that would likely be a pretty large break from the standard install for the debian version. I for one wouldn't want that change by default.

[chorrell]

Yeah, that would be a pretty significant and unexpected change for a Debian 8 image.

[razamatan]

heh, i thought i'd try the proposition. so we're stuck w/ systemd in deb8 which i'm getting bitten by this bug. is pinning the workaround for now? i thought the pinning solution is in place, but dist-update doesn't really ignore udev.

[quanah]

This is a bummer, because virtualbox requires systemd-udev. Since VBox is not compatible with KVM, an LXC image is the only option for use with virtualbox. But without systemd-udev support, you can't use LXC either. Which means there's no way to use vagrant with virtualbox images (which is what the majority of images are) when smartOS is the hypervisor. I would note that LXC images under vagrant work with udev just fine, so this limitation on udev + LXC seems to be a smartOS limitation?

[jmai444]

Latest Debian image 9a8d53c0-c15b-11e6-9c4f-3bcedc82f8e1 now broken for me. Getting messages as per eripa above when I do apt-get upgrade.

This image wasn't broken previously as I used it in another VM a few weeks ago and was able to run apt-get upgrade without issues.

[jmai444]

I spun up a throwaway Debian LX zone using the 9a8d53c0-c15b-11e6-9c4f-3bcedc82f8e1 image. I issued this command before trying to upgrade in order to stop udev upgrading:

apt-mark hold udev

I am at least able to get through apt-get upgrade without error. It won't help if you've already upgraded and I'm not sure if it will cause subsequent issues in the zone.

[plobbes]

+1 for apt-mark hold udev

[chorrell]

I discovered an issue with the /etc/apt/preferences.d/udev file. It has leading spaces which, apparently, prevents the apt pinning from working. If you remove the leading spaces, the upgrade should work and the udev package will be held back.

[chorrell]

The latest debian-8 lx-brand image should now address this: https://images.joyent.com/images/e74a9cd0-f2d0-11e6-8b69-b3acf2ef87f7

[plitc]

My workaround: apt-get autoclean; apt-get clean; apt-get update; apt-get upgrade (... udev install failed ...) sed -i '1i exit 0' /var/lib/dpkg/info/udev.postinst apt-get install -f apt-get upgrade; apt-get dist-upgrade; apt-get autoremove

(... apt debugging was with strace and shellscripts with -x ...)