[improvement] no easy way to add packages required by apt before apt stage

[ubuntu-server-builder]

This bug was originally filed in Launchpad as LP: #1813399

Launchpad details

affected_projects = []
assignee = None
assignee_name = None
date_closed = None
date_created = 2019-01-26T14:01:45.960398+00:00
date_fix_committed = None
date_fix_released = None
id = 1813399
importance = undecided
is_complete = False
lp_url = https://bugs.launchpad.net/cloud-init/+bug/1813399
milestone = None
owner = till.riedel
owner_name = Till Riedel
private = False
status = confirmed
submitter = till.riedel
submitter_name = Till Riedel
tags = ['debian']
duplicates = []

Launchpad user Till Riedel(till.riedel) wrote on 2019-01-26T14:01:45.960398+00:00

I am running cloud-init on a libvirt/virsh with this image: https://cdimage.debian.org/cdimage/openstack/archive/9.6.5-20190122/debian-9.6.5-20190122-openstack-amd64.qcow2

This is not really a bug, because I believe it is rather a packaging problem of many distros. So please close this asap if you do not feel responsible ;)

This is why I also posted this problem on the debian-cloud mailing list, but maybe this can be fixed also in cloud-init.

If apt-transport-https or dirmngr are not installed (they are no dependencies of cloud-init).

apt: sources: docker: source: 'deb [arch=amd64] https://download.docker.com/linux/debian stretch stable' keyserver: keyserver.ubuntu.com keyid: 0EBFCD88

you will run into problems like:

[ 14.011339] cloud-init[469]: E: The method driver /usr/lib/apt/methods/https could not be found. [ 14.013524] cloud-init[469]: Cloud-init v. 0.7.9 running 'modules:config' at Wed, 23 Jan 2019 16:28:06 +0000. Up 10.01 seconds. [ 14.016921] cloud-init[469]: 2019-01-23 16:28:10,921 - util.py[WARNING]: Running module apt-configure (<module 'cloudinit.config.cc_apt_configure' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_apt_configure.py'>) failed

and

[ 14.460293] cloud-init[448]: Command: ['gpg', '--keyserver', 'keyserver.ubuntu.com', '--recv', '0EBFCD88'] [ 14.463206] cloud-init[448]: Stderr: gpg: failed to start the dirmngr '/usr/bin/dirmngr': No such file or directory

Unfortunatedly I could not find a way to call anything to add those packages before apt is run so that I had to patch the images.

One solution would be to allow multistage apt install in one way or the other (I know it adds complexity), so that one could install the relevant packages before the package list is modified and updated. Actually checking if a line contains https or a keyid and installing the appropriate dependencies could also work.

Probably it would be much easier to fix the package dependency on cloud-init itself for the relevant distros. The downside of this is that it adds a bunch of packages as further dependencies, so I guess distros are not really willing to do that...

Thanks in advance,

Till

PS: I can live with having written some scripts patching the images after download...

[ubuntu-server-builder]

Launchpad user do3meli(d-info-e) wrote on 2019-04-17T09:16:15.836663+00:00

i ran in to the exact same problem. i see it the same way and believe we have to add dirmngr as package dependency for cloud-init.

i found the following mailing list for debian cloud images that discusses this issue: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=910654

the link references a merge request with more details on it: https://salsa.debian.org/cloud-team/debian-cloud-images/merge_requests/22

basically they will/have added gnupg has a hard dependency on dirmngr for debian buster. so most likely they will fix it in the image itself.

for older images like debian stretch and as a fallback logic i think we should adjust cloud-init to always install dirmngr before running apt configuration module.

@Scott Moser: Any idea how to achieve this?

[ubuntu-server-builder]

Launchpad user do3meli(d-info-e) wrote on 2019-04-17T09:40:29.307656+00:00

btw: i have fixed this with a little workaround in the cloud-init config by running a bootcmd that installs the package very early in the boot process:

bootcmd:

• apt -y install dirmngr

[ubuntu-server-builder]

Launchpad user Julian Andres Klode(juliank) wrote on 2019-04-17T09:54:27.110848+00:00

Just some words from the APT side:

So apt-transport-https is irrelevant for 18.04+ and buster, it's just a transitional package, and it should not be installed on those.

Regarding key management: keyserver.ubuntu.com talks https, it's trivial to get keys from it without involving the gpg crap. Not sure what the others do, but I'd generally advise against the use of other keyservers. Heck, I'd prefer serving a .asc or .gpg file and having a keyurl to dump that file into trusted.gpg.d

[ubuntu-server-builder]

Launchpad user Julian Andres Klode(juliank) wrote on 2019-04-17T09:58:59.670615+00:00

So in fact, HKP is quite easy to do by hand, e.g.:

http://keyserver.ubuntu.com:11371/pks/lookup?op=get&search=0x99242560

where 0x99242560 is the keyid field and keyserver.ubuntu.com is the keyserver.

It's trivial to do that yourself in cloud-init, there's no need to involve all the gnupg and dirmngr mess.

[ubuntu-server-builder]

Launchpad user Julian Andres Klode(juliank) wrote on 2019-04-17T09:59:42.005007+00:00

That said, if you do, do require HKPS (HKP using HTTPS).

[kacper-ka]

Not the same issue, but related, as I suppose:

When using ubuntu-minimal image (I am using LXD) and seed cloud-init configuration that adds new repository and fetches signing key I have observed that cloud-init forces installation of gnupg package in order to configure apt. After new repository has been configured further apt-get update invocations are ignored as it has been already run and is marked as a one-time job.

The issue does not occur if using normal ubuntu image, which has gnupg preinstalled.

[holmanb]

After new repository has been configured further apt-get update invocations are ignored as it has been already run and is marked as a one-time job.

@kacper-ka Your issue sounds like a duplicate of https://github.com/canonical/cloud-init/issues/5223 which has been fixed in version 24.2.

[kacper-ka]

After new repository has been configured further apt-get update invocations are ignored as it has been already run and is marked as a one-time job.

@kacper-ka Your issue sounds like a duplicate of https://github.com/canonical/cloud-init/issues/5223 which has been fixed in version 24.2.

That seems to be about right, thank you for pointing it out.