search

puppetlabs / puppetlabs-apt

Puppet module to help manage Apt
https://forge.puppetlabs.com/puppetlabs/apt
Apache License 2.0
215 stars 463 forks source link

Support for storing apt keys in separate files #1034

Closed dhs-rec closed 8 months ago

dhs-rec commented 2 years ago

Use Case

The apt-key command has been deprecated in Debian testing aka bookworm in favor of storing key files directly into either /etc/apt/trusted.gpg.d/ or /etc/apt/keyrings/ and (in case of the latter) refer to them directly in individual sources.list entries (like [ signed-by=/path/to/file.gpg ].

This also already works in Debian stable aka bullseye (and maybe oldstable/buster) and corresponding Ubuntu versions. So it would be nice to have this in place already before bookworm is released, esp. so as apt-get update will emit warnings for all keys still stored in the /etc/apt/trusted.gpg file.

Describe the Solution You Would Like

apt::key should have options to store keys downloaded as .gpg or .asc files in one of the directories above.

Describe Alternatives You've Considered

Only alternative I see currently is to use a file resource instead of apt::key.

Additional Context

https://manpages.debian.org/testing/apt/apt-key.8.en.html

kenyon commented 2 years ago

This is https://tickets.puppetlabs.com/browse/MODULES-9695.

github-actions[bot] commented 1 year ago

Hello! 👋

This issue has been open for a while and has had no recent activity. We've labelled it with attention-needed so that we can get a clear view of which issues need our attention.

If you are waiting on a response from us we will try and address your comments on a future Community Day.

Alternatively, if it is no longer relevant to you please close the issue with a comment.

dhs-rec commented 1 year ago

Yeah, the bot is right: Ubuntu 22.04 has already transitioned to the new style, so this should really get some attention.

A bit more information can be found here: https://wiki.debian.org/DebianRepository/UseThirdParty

jamesps-ebi commented 1 year ago

It looks like this module already supports the [ signed-by=/path/to/file.gpg ] option for apt::source resources. So that aspect is already taken care of.

I guess it's just a case of placing keys under /etc/apt/trusted.gpg.d or /usr/share/keyrings by default. Since the apt-key is deprecated in all current Debian/Ubuntu distributions, the ability to pull keys from a keyserver should probably be removed completely, which would make this a breaking change for existing deployments.

It might be possible to remove the existing apt_key provider in favour of a simple file resource to create the keyfiles on the managed node.

dhs-rec commented 1 year ago

I guess it's just a case of placing keys under /etc/apt/trusted.gpg.d or /usr/share/keyrings by default.

/etc/apt/keyrings or /usr/share/keyrings, depending on whether the keyring in question is managed by a package later on (see link above).

heini commented 1 year ago

BTW: https://discourse.ubuntu.com/t/spec-apt-deb822-sources-by-default/29333

Supporting deb822 format (like apt::deb822) would make the whole thing quite easy. Could be done with a template and two file resources (one for the repository definition file, one for the key).

See sources.list(5) for the details.

jamesps-ebi commented 1 year ago

BTW: https://discourse.ubuntu.com/t/spec-apt-deb822-sources-by-default/29333

Tested on my Ubuntu 22.04.1 LTS box. Looks like this format is already supported and works in combination with the old format, but is not currently the default for Ubuntu. As far as I can tell, it is also supported in the latest stable release of Debian (bullseye), but again is not the default.

Until both distros switch to deb822 format, there may be some compatibility issues with other programs that interact with apt sources. But it does look like efforts are being to make this format default in the next stable release of both Debian and Ubuntu.

Some more supporting docs for anyone interested in implementing this:

dhs-rec commented 1 year ago

It is supported in Debian (and thus Ubuntu) since Stretch. From sources.list (5):

This is a new format supported by apt itself since version 1.1.

AardvarKOlogY commented 1 year ago

https://github.com/puppetlabs/puppetlabs-apt/issues/1034#issuecomment-1333808499

[...] Since the apt-key is deprecated in all current Debian/Ubuntu distributions, the ability to pull keys from a keyserver should probably be removed completely, which would make this a breaking change for existing deployments.

It might be possible to remove the existing apt_key provider in favour of a simple file resource to create the keyfiles on the managed node.

Yes, a hard switch from one release to the other may break things. One could opt for continuing to support apt-key while bringing in the new way of providing a file as suggested. Meaning: defining something like "provider" defaulting to "file" and optionally answering to "aptkey". That is, if we continue to use something like apt::key. If you do separate key distribution by files, you can already do this and just ignore apt::key (e.g. apt::source's apt update will ofc work fine if the key file has been placed prior to its execution).

And as to the "deb [ signed-by=/some/file ]" inside sources.list: I'd consider this optional, since in reality it also is: there are default locations for the keys. If keys are placed there, they are being used. apt::key (or whatever) should take this into account, too.

dhs-rec commented 1 year ago

No, signed-by is NOT optional if keyring files are used. It's required to specify which keyring file to use for verifying this particular repository (and its packages).

AardvarKOlogY commented 1 year ago

Seems optional to me so far. See sources.list(5):

If no keyring files are specified the default is the trusted.gpg keyring and all keyrings in the trusted.gpg.d/ directory (see apt-key fingerprint).

I've tried this out: If you place a keyring in /etc/apt/keyrings, you need to specify it through signed-by, if you place it (yes, a keyring file) in /etc/apt/trusted.gpg.d, you don't need to do that. The files in /etc/apt/trusted.gpg.d also don't result in a deprecation warning (unlike keys stored in /etc/apt/trusted.gpg e.g. through apt::key, this triggers on upgrades and so on).

The apt-key manpage states:

   /etc/apt/trusted.gpg.d/
      File fragments for the trusted keys, additional keyrings can be stored here (by other packages or the administrator). Configuration Item Dir::Etc::TrustedParts.
  /etc/apt/keyrings/
      Place to store additional keyrings to be used with Signed-By.

I'd say this further validates my point - the only "but" here is it's the manpage of apt-key(8) which is to be deprecated.

I wonder: if everything is supposed to switch away from even trusted.gpg.d (which would be the consequence of what you write), why are the distribution keyrings (Debian or Ubuntu maintainers) still in that very folder and the default sources.list does not use Signed-By?

My ulterior motive here is adhering to all standards. One could argue if this is currently inconsistent by design, however, then Signed-By would be the (only) go-to option because whatever happens to trusted.gpg.d in the future this will work.

heini commented 1 year ago

Which OS/version? Ubuntu 22.04 already made the switch (and complains loudly about keys in trusted.gpg), Debian will do with Bookworm.

Did you read the Debian wiki page linked above? It clearly says:

The certificate MUST NOT be placed in /etc/apt/trusted.gpg.d or loaded by apt-key add.

AardvarKOlogY commented 1 year ago

Which OS/version?

Debian 11 and Ubuntu 22.04. Did no verification on a testing branch yet.

Did you read the Debian wiki page linked above? It clearly says: [...]

Actually, no, to be honest (skipped past it somehow...). The Debian Wiki on securing apt and the Debian Manual in the respective section do not, but then again, some of the information there is rather dated, and the link above makes a quite strong point about the newly added keys in trusted.gpg.d simply making every repository valid vs /etc/apt/keyring files only validating, well, the very repository files they are explicitly being referenced in, the latter option making more sense. (apt-doc for apt 2.2.4 isn't helpful at all.)

The apt changelog also complains about trusted.gpg, but not about trusted.gpg.d. That complaint is being issued by apt itself from 2.4.0 on if trusted.gpg is being used, but not for trusted.gpg.d.

I tend to believe (I know, beliefs in engineering...) now that this switch is only to move away from storing everything in trusted.gpg. Feel free to add in here, I do not see an opportunity to spin up a Debian 12 machine at least for the next 12+ hours myself to verify.

That is what caused my confusion here. Considering the wiki article cited by you and dhs-rec about Third Party Repository signing as best practice at the very least, I'd say I stand corrected and sorry for adding my confusion to this discussion.

weavage commented 1 year ago

Don't forget basic authentication params for keys stored/mirrored in protected repos like JFrog.

XSpielinbox commented 11 months ago

As per apt-key manpage in all versions of Ubuntu at least since Ubuntu 22.04 and Debian 11 apt-key(8) will last be available in Debian 11 and Ubuntu 22.04.

These releases are out for years now and with Debian 12 the first version with long term support that drops it, is out now for month too. It would be really nice if one could finally follow the way the respective repositories package their keys (no other reputable software I know still uses apt-key). To summarize, one should be able to:

  1. put the keys in /usr/share/keyrings/ or /usr/share/<package>, as e.g. MongoDB, signald, Signal, OnlyOffice, GitLab-Runner, GitLab, Element, VSCodium, Icinga, PostgreSQL, NGINX, Bazel, eduVPN, Brave or even CUDA, VirtualBox and NVIDIA Container Toolkit do it. Some of them like e.g. Element also bring there own package that manages these keys then.
  2. put the keys in /etc/apt/keyrings/, as e.g. Node.js, Grafana, FreeRADIUS, Dovecot, Docker, Podman and ROCm do it
  3. put the keys in /etc/apt/trusted.gpg.d/, as e.g. InfluxDB and Foreman do it.

I have tested all these methods manually and they all work at least since Ubuntu 20.04. They should work on all versions since Debian 9 and Ubuntu 16.04, if I am not mistaken. Releases before that only support the third method (so /etc/apt/trusted.gpg.d/).

But as far as I can tell, Puppet still does not support any one of them. Please correct my if I am wrong on that.

Are there any plans when this will be supported?

kenyon commented 11 months ago

@XSpielinbox those instructions that say to put keys under /usr are wrong, only packages should be installing keys under /usr. Keys not installed by packages should go under /etc. This was discussed in #1105. Otherwise, agreed.

dhs-rec commented 11 months ago

@XSpielinbox those instructions that say to put keys under /usr are wrong, only packages should be installing keys under /usr.

@kenyon, no, they are not wrong. Keys can initially be installed in /usr/share/keyrings any way you like, but only packages should update those keys.

kenyon commented 9 months ago

I think this can be closed, was done in #1128.

XSpielinbox commented 9 months ago

Is there any plan when there will be a new release?

kenyon commented 9 months ago

I'm seeing if I can get rid of this TODO: https://github.com/puppetlabs/puppetlabs-apt/blob/bee5ee11ebec6de7e91febd4975d67a14e57bfa4/manifests/source.pp#L192

kenyon commented 8 months ago

I'm seeing if I can get rid of this TODO:

https://github.com/puppetlabs/puppetlabs-apt/blob/bee5ee11ebec6de7e91febd4975d67a14e57bfa4/manifests/source.pp#L192

Got rid of that TODO in #1147, and that was released in #1150.

This can be closed.