search

adoptium / installer

Installer scripts for Eclipse Temurin binaries
Apache License 2.0
150 stars 75 forks source link

Complete new `linuxNew` based packaging #330

Closed karianna closed 2 years ago

karianna commented 3 years ago

Initial List before we split into an Epic etc.

karianna commented 3 years ago

From the original author:

If I remember correctly, the necessary infra for 11 and 16 is present. Doing packages for 8 is copy & paste and a little bit on top. Jenkins is able to build the packages. The only steps left to do: Pull them from Jenkins, manually sign the RPMs (create a new GPG key beforehand), and upload everything onto a server.

jerboaa commented 3 years ago

Initial metadata update: https://github.com/adoptium/installer/pull/331

Is there an email alias that can be used for RPMs/Debs? I suppose packaging@adoptium.com is wrong?

karianna commented 3 years ago

https://github.com/adoptium/installer/pull/332 for Updated README and an early attempt at adding a Java 8 temurin Debian and Red Hat packages

karianna commented 3 years ago

Initial metadata update: #331

Is there an email alias that can be used for RPMs/Debs? I suppose packaging@adoptium.com is wrong?

Good question. I suspect we aren't allowed to use the AdoptOpenJDK.net email addresses. Perhaps an EF-based mailing list?

aahlenst commented 3 years ago

Parameterize the scripts so that version numbers aren't hard-coded?

I've learnt the hard way over the past years that it's tough to reconcile OpenJDK's version numbering and Adopt's version numbering with the requirements of Debian and RPM packages. Remember the drama of 11.0.9.1. If you get it wrong, the packages won't show up on the user's computers (been there, done that). If you get it very wrong, you have to increment the epoch (been there, done that). Computing the version number correctly requires access to correct metadata (for example, because the .1 rebuilds aren't included in java -version). In the past, metadata was sometimes incomplete or wrong. Furthermore, extensive automated testing would be needed to ensure that every edge case is covered. You also have to consider that the packages are a separate "product" with their own version. If you fix a bug in the package, you have to bump the version. And you have to set it back once the next PSU comes out. Therefore, the lack of parameterization is a feature.

Fix RH Build failing due to GPG signing

I'm not sure what it means. If it means that you intend to sign the RPMs during the build, I'd reconsider. Signing RPMs during build requires sensitive key material on the build machines. Building takes time, so you need multiple machines. That was a problem in the past. So it's a deliberate choice not to sign the RPMs. I'd rather sign the RPMs just before uploading or even on the machine that distributes them (private GPG key is required on there anyway because the package indexes have to be signed).

I'd also strongly consider creating a keyring package for Debian (see https://wiki.debian.org/DebianRepository/UseThirdParty, Brave did this well in the past). This allows to rollover keys.

karianna commented 3 years ago

Fix RH Build failing due to GPG signing

I'm not sure what it means. If it means that you intend to sign the RPMs during the build, I'd reconsider. Signing RPMs during build requires sensitive key material on the build machines. The building takes time, so you need multiple machines. That was a problem in the past. So it's a deliberate choice not to sign the RPMs. I'd rather sign the RPMs just before uploading or even on the machine that distributes them (a private GPG key is required on there anyway because the package indexes have to be signed).

Something in the default Gradle tasks caused a GPG sign to be executed. I think I'd rather have that as a separate explicit step. Still digging into the code to see why it triggers (Suse and Debian don't).

I'd also strongly consider creating a keyring package for Debian (see https://wiki.debian.org/DebianRepository/UseThirdParty, Brave did this well in the past). This allows to roll over keys.

Will check with EF if they have something.

smlambert commented 3 years ago

Additional piece that we didn't pull into the linuxNew story before (back when the work from adoptium-packages-linux was pulled in, is the cacert piece, which can be found here: https://github.com/aahlenst/adoptium-packages-linux/tree/cacerts - so add a step in the description of this issue to bring that in.

jerboaa commented 3 years ago

Initial metadata update: #331 Is there an email alias that can be used for RPMs/Debs? I suppose packaging@adoptium.com is wrong?

Good question. I suspect we aren't allowed to use the AdoptOpenJDK.net email addresses. Perhaps an EF-based mailing list?

@tellison suggested temurin-dev@eclipse.org which seems fine. It's just an email address for users to report bugs/issues.

karianna commented 3 years ago

temurin-dev@eclipse.org

I've globally replaced that in my PR

aahlenst commented 3 years ago

Dockerfiles for RPM-based distros need an update across the board because the old sks-keyservers.net pool was shut down.

RUN gpg --batch --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys 595E85A6B1B4779EA4DAAEC70B588DFF0527A9B7

needs to be replaced with

RUN  gpg --batch --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 595E85A6B1B4779EA4DAAEC70B588DFF0527A9B7
aahlenst commented 3 years ago

Does anybody have any logs that shed some light on "Fix RH Build failing due to GPG signing?" I just ran an RPM build and it worked fine.

aahlenst commented 3 years ago

Would be very helpful if someone could hook it up with GitHub Actions. A workflow exists but wasn't copied over. Although it's x64 only, that alone catches some issues and helps verifying PRs.

karianna commented 3 years ago

RUN gpg --batch --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 595E85A6B1B4779EA4DAAEC70B588DFF0527A9B7

I swapped this out but get the error:

#9 [ 6/19] RUN gpg --batch --verify /tmp/gosu.asc /usr/bin/gosu
#9 sha256:3ca4c0b2d52e7d651b6e73a9d43b94d10b5f37545dfc6f2cbb752ccc2ac64eea
#9 0.232 gpg: Signature made Thu Apr 16 06:40:32 2020 UTC
#9 0.232 gpg:                using RSA key B42F6819007F00F88E364FD4036A9C25BF357DD4
#9 0.232 gpg: Can't check signature: No public key
#9 ERROR: executor failed running [/bin/sh -c gpg --batch --verify /tmp/gosu.asc /usr/bin/gosu]: exit code: 2
------
 > [ 6/19] RUN gpg --batch --verify /tmp/gosu.asc /usr/bin/gosu:
------
executor failed running [/bin/sh -c gpg --batch --verify /tmp/gosu.asc /usr/bin/gosu]: exit code: 2

On a side note, should I also swap the key server for the Suse script?

karianna commented 3 years ago

@aahlenst - Extra question. I understand you completed the ca-certs work in your repo (https://github.com/aahlenst/adoptium-packages-linux/tree/cacerts). Should I just be looking to bring across all of https://github.com/aahlenst/adoptium-packages-linux/tree/cacerts/ca-certificates and patch the scripts to refer to those as what I think you've done in your repo?

karianna commented 3 years ago

RUN gpg --batch --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 595E85A6B1B4779EA4DAAEC70B588DFF0527A9B7

I swapped this out but get the error:

#9 [ 6/19] RUN gpg --batch --verify /tmp/gosu.asc /usr/bin/gosu
#9 sha256:3ca4c0b2d52e7d651b6e73a9d43b94d10b5f37545dfc6f2cbb752ccc2ac64eea
#9 0.232 gpg: Signature made Thu Apr 16 06:40:32 2020 UTC
#9 0.232 gpg:                using RSA key B42F6819007F00F88E364FD4036A9C25BF357DD4
#9 0.232 gpg: Can't check signature: No public key
#9 ERROR: executor failed running [/bin/sh -c gpg --batch --verify /tmp/gosu.asc /usr/bin/gosu]: exit code: 2
------
 > [ 6/19] RUN gpg --batch --verify /tmp/gosu.asc /usr/bin/gosu:
------
executor failed running [/bin/sh -c gpg --batch --verify /tmp/gosu.asc /usr/bin/gosu]: exit code: 2

On a side note, should I also swap the key server for the Suse script?

I've resolved this, gosu and tini needed separate keys.

karianna commented 3 years ago

@aahlenst - Extra question. I understand you completed the ca-certs work in your repo (https://github.com/aahlenst/adoptium-packages-linux/tree/cacerts). Should I just be looking to bring across all of https://github.com/aahlenst/adoptium-packages-linux/tree/cacerts/ca-certificates and patch the scripts to refer to those as what I think you've done in your repo?

@aahlenst - Stewart and I were a little confused over this. I think the current build bundles the cacerts with the JDK itself, so is having a separate adoptium-ca-cacerts linux package actually needed?

aahlenst commented 3 years ago

I hope you have some time 😁

Having the keystore as part of the JDK (= included in the package) is a subpar experience.

So, integrating with the tooling of the OS to centrally manage the keystore and have a consistent list of certificates in the OS and JDK is the right thing to do.

On RPM based distributions, this can be achieved rather easily. The ca-certificates packages of RHEL and friends automatically generate a suitable keystore for the JDK. You just have to symlink it.

On Debian derivatives, this is more complicated. There exists a package that can generate a keystore for the JDK. Unfortunately, it has a cyclic dependency on the JDK because the program to generate the keystore is written in Java. Some script in that package acts as a tie breaker, but it can only do that for JDK it knows. And it knows only JDKs shipped as part of Debian. We tried it in the past and failed. So, I decided to roll our own without the cyclic dependency. To be clear, it does not contain a cacerts file. But it generates one and integrates with Debian's facilities to manage CA certificates. You add a CA certificate to Debian and the cacerts will automatically be regenerated. Same happens if you remove one. As a result, the package is very small and even architecture-independent. In can be used by any JDK (not only Temurin). It just needs to be installed once.

I think that provides enough value for a little price and brings Temurin's packages on equal footing as the JDKs shipped by the distros themselves. But Temurin even has the AQA/TCK stamps of approval.

karianna commented 3 years ago

Thinking out loud. I'm not 100% convinced this is what users will expect. Up until now Java users have kept the O/S keystore and Java keystore separate and I'd hypothesize for many users that's a good thing (they may not want their O/S to have all of the allowable certs by Java and vice versa).

By making this change now, we'd be changing some pretty fundamental default behaviour.

Perhaps offering to do this as an optional step would be OK (if I can figure out how to do that).

karianna commented 3 years ago

For Eclipse Signing: see https://wiki.eclipse.org/IT_Infrastructure_Doc#What_about_GPG_signing.3F looks like they have a maven target mvn gpg:sign-and-deploy-file we’ll need to find out what the cbi url is to send artefacts e.g https://cbi.eclipse.org/jarsigner/sign but for gpg

George is asking the EF on the details here.

aahlenst commented 3 years ago

We got requests to integrate with the OS facilities. And now would be a good time to fix all the weirdness. We'd also follow every (?) distro JDK package out there. RHEL, SUSE, Debian, Alpine, all do this. It was inaptness that we didn't have it from the beginning.

If you want to make this opt-in (Debian, don't know about RHEL from the top of my head):

  1. Rename cacerts to bundled_cacerts or something similar.
  2. Claim a directory/file in /etc/default. Define a variable like BUNDLED_CACERTS=Yes.
  3. Include tooling in the package to symlink cacerts to either bundled_cacerts or the file in the cacerts package depending on the value of BUNDLED_CACERTS.
  4. Invoke that tooling during postinst.

This would have to be part of the Temurin JDK package. If you included it in the cacerts package, the cacerts package would have to know about all versions. Doable, but maybe not desirable.

Should I just be looking to bring across all of https://github.com/aahlenst/adoptium-packages-linux/tree/cacerts/ca-certificates and patch the scripts to refer to those as what I think you've done in your repo?

You might succeed in cherry-picking the commit or use git patch apply to bring it over.

aahlenst commented 3 years ago

For Eclipse Signing: see https://wiki.eclipse.org/IT_Infrastructure_Doc#What_about_GPG_signing.3F looks like they have a maven target mvn gpg:sign-and-deploy-file we’ll need to find out what the cbi url is to send artefacts e.g https://cbi.eclipse.org/jarsigner/sign but for gpg

George is asking the EF on the details here.

That's what you have to do: https://access.redhat.com/articles/3359321. RPMs only. Debian packages should not be signed, but the package index instead.

sxa commented 3 years ago

We got requests to integrate with the OS facilities.

@aahlenst Makes sense but any idea why we wanted to have our own CA package instead of using the CA package supplied with each distro? e.g. ca-certificates on RHEL (Provides /etc/pki/ca-trust/extracted/java/cacerts) or ca-certificates-java (Provides /etc/ssl/certs/java/cacerts) on Ubuntu

karianna commented 3 years ago

OK, John and I were looking at this locally and I see what Debian does now (basically loops through the O/S certs and adds them via keytool for Java, thereby keeping them in sync - there's an update command as well). So I think it makes sense as @aahlenst suggests. Now to figure out the coding to make that a reality :-).

karianna commented 3 years ago

It looks like Debian already solves this with the ca-certificates-java package. I'll try setting that as a dependency.

Now, the ca-certificates-java may not have the richness of the Adopt set that we pull from Mozilla, in which case I'll go back to the fuller solution that @aahlenst proposed. Sorry for the slowness here folks, I'm learning as I go :-)

Hmmm, maybe not #105 :-)

OK fine, I'm wrong - lets do it the Andreas way - this may take me some time to code up, but will try to get it done before EOW

aahlenst commented 3 years ago

@karianna @sxa I know it's hard to keep track of everything I wrote. But https://github.com/adoptium/installer/issues/330#issuecomment-896308438 explains everything. Why ca-certificates-java doesn't work and that our cacerts package is supposed to be Debian only because RHEL has solved this differently (and better). Please re-read. I also welcome suggestions to improve my writing.

Also note that most Linux distributions effectively use Mozilla's list of certificates.

sxa commented 3 years ago

Thanks, and apologies that I hadn't read through that comment previously

karianna commented 3 years ago

Likewise - thanks for the detail - I'll give this a go tomorrow and update the PR as appropriate. This is the last step before signing to solve. Both RPM and Deb 8/11 packages have been manually tested and compared against the previous AdoptOpenJDK packages and are the same - so that's good.

karianna commented 3 years ago

@aahlenst - I'd like to fork and clone your repo into Adoptium that creates the adoptium-ca-certificates.deb file. Would that be OK?

aahlenst commented 3 years ago

@karianna Understood. I'm happy to arrange a full transfer. Or you can create a new repo and just push the old contents into it.

karianna commented 3 years ago

Thanks @aahlenst !

Instructions from Tim E ->

As it is existing work coming into the project then that should be a CQ. To have a new repo created we need to raise a Bugzilla request (https://www.eclipse.org/projects/handbook/#ip-cq)

When you log in to eclipse.org you can see the committer tools at the Temurin project page, and a link to “Create a Contribution Questionnaire”

I'll follow that now.

aahlenst commented 3 years ago

Something to keep an eye on while evaluating repository hosting options: The software responsible for generating the Debian repository needs to be capable to sign the Release and InRelease files with multiple GPG keys. Otherwise, it is impossible to securely roll over the keys. Artifactory cannot do it (at least when I checked the last time, heck it cannot even generate those files correctly). reprepro can since 4.3.

karianna commented 3 years ago

I'm back and working on this now:

@aahlenst - I'm getting the following ca-certifacates test failing:

org.opentest4j.AssertionFailedError: 
Expecting:
 <"drwxr-xr-x root/root         0 2020-07-29 13:35 ./
drwxr-xr-x root/root         0 2020-07-29 13:35 ./etc/
drwxr-xr-x root/root         0 2020-07-29 13:35 ./etc/ca-certificates/
drwxr-xr-x root/root         0 2020-07-29 13:35 ./etc/ca-certificates/update.d/
-rwxr-xr-x root/root       795 2020-07-29 13:35 ./etc/ca-certificates/update.d/adoptium-cacerts
drwxr-xr-x root/root         0 2020-07-29 13:35 ./etc/default/
-rw-r--r-- root/root       175 2020-07-29 13:35 ./etc/default/adoptium-ca-certificates
drwxr-xr-x root/root         0 2020-07-29 13:35 ./etc/ssl/
drwxr-xr-x root/root         0 2020-07-29 13:35 ./etc/ssl/certs/
drwxr-xr-x root/root         0 2020-07-29 13:35 ./etc/ssl/certs/adoptium/
drwxr-xr-x root/root         0 2020-07-29 13:35 ./usr/
drwxr-xr-x root/root         0 2020-07-29 13:35 ./usr/share/
drwxr-xr-x root/root         0 2020-07-29 13:35 ./usr/share/doc/
drwxr-xr-x root/root         0 2020-07-29 13:35 ./usr/share/doc/adoptium-ca-certificates/
-rw-r--r-- root/root       159 2020-07-29 13:35 ./usr/share/doc/adoptium-ca-certificates/changelog.Debian.gz
">
to be equal to:
 <"drwxr-xr-x root/root         0 2020-07-29 13:35 ./
drwxr-xr-x root/root         0 2020-07-29 13:35 ./etc/
drwxr-xr-x root/root         0 2020-07-29 13:35 ./etc/ca-certificates/
drwxr-xr-x root/root         0 2020-07-29 13:35 ./etc/ca-certificates/update.d/
-rwxr-xr-x root/root       795 2020-07-29 13:35 ./etc/ca-certificates/update.d/adoptium-cacerts
drwxr-xr-x root/root         0 2020-07-29 13:35 ./etc/default/
-rw-r--r-- root/root       180 2020-07-29 13:35 ./etc/default/adoptium-ca-certificates
drwxr-xr-x root/root         0 2020-07-29 13:35 ./etc/ssl/
drwxr-xr-x root/root         0 2020-07-29 13:35 ./etc/ssl/certs/
drwxr-xr-x root/root         0 2020-07-29 13:35 ./etc/ssl/certs/adoptium/
drwxr-xr-x root/root         0 2020-07-29 13:35 ./usr/
drwxr-xr-x root/root         0 2020-07-29 13:35 ./usr/share/
drwxr-xr-x root/root         0 2020-07-29 13:35 ./usr/share/doc/
drwxr-xr-x root/root         0 2020-07-29 13:35 ./usr/share/doc/adoptium-ca-certificates/
-rw-r--r-- root/root       169 2020-07-29 13:35 ./usr/share/doc/adoptium-ca-certificates/changelog.Debian.gz
">

I'll confess I'm a little stuck here - it appears that the file sizes are different which is what's causing that diff to fail, for example:

rw-r--r-- root/root 175 2020-07-29 13:35 ./etc/default/adoptium-ca-certificates

coming from the dpkg --contents <container> command

vs

-rw-r--r-- root/root 180 2020-07-29 13:35 ./etc/default/adoptium-ca-certificates

coming from the deb-contents.txt file.

Are you seeing the same issue locally. I'm wondering if it's a genuine bug or whether the file sizes differ due to host O/S differences.

aahlenst commented 3 years ago

I think updating the file should be sufficient. I altered some files while making the PR so this is expected to fail.

This test is less about the actual sizes than about the contents and the permissions. Reformatting the output of dpkg --contents by getting rid of the size column might be sensible.

aahlenst commented 3 years ago

@karianna dpkg --contents adoptium-ca-certificates_1.0.0-1_all.deb | awk '{$3=$4=$5=""; print $0}' is what you're looking for. This removes the size and time based information.

karianna commented 3 years ago

Thanks, @aahlenst! - I made a few other modifications as well and now all of those tests pass. I think we're now more at the stage of working with the Eclipse Foundation and/or other hosting providers to see where we can land the results of this work.

perlun commented 3 years ago

@karianna Any more progress on this recently? Is this blocked by https://bugs.eclipse.org/bugs/show_bug.cgi?id=575312 btw, or is there anywhere else these packages could be hosted in the meanwhile?

karianna commented 3 years ago

I just need to find time - and sadly my main work machine has pretty much died on me so I'm going to lose potentially another week. Super sorry about this I know the community has been waiting for a long time.

perlun commented 3 years ago

No worries, things like this can happen to all of us. Let us know if we can be of any help. :+1:

medb commented 3 years ago

Is there any ETA when the deb packages will be available?

karianna commented 3 years ago

Hoping to complete this by end of Sept, the Java 17 release is giving us extra impetus :-)

rpardini commented 3 years ago

Hello, it's me again! 😛 While we wait for the official packages, here's some unofficial 'installer' packages for Debian and Ubuntu. https://github.com/rpardini/adoptium-deb-installer I had a working package generator back from AOJ days, and adapted it today to use GitHub Actions and the v3 Adoptium API. It produces tiny packages that just download the tarball from official sources, check SHA256, and sets it up the way Debian and Ubuntu expects it (update-alternatives, etc). They're multi-arch, and packages for JDK 8, 11, 16 and 17 are available, both in a simple Debian APT repo and via Ubuntu PPA. Remember: unofficial, don't go using those in production.

byF commented 3 years ago

@karianna anything the community can do to help?

karianna commented 3 years ago

I'll defer to @sophia-guo and @gdams who are progressing this piece.

persicsb commented 3 years ago

Any update on this? @sophia-guo , @gdams ? What can we help to move this forward?

berndschatz commented 3 years ago

Is it possible that the adoptjdk project continuos to provide the packages in their repos until you are ready to take it over ?

sophia-guo commented 3 years ago

I think @gdams is probably working on this to push the DEB’s/RPM’s to a repo? I don't have much more information about that. We do have a jenkinsfile, which was designed to include installer and uploading job. Though that part has not been done yet due to insufficient information or decisions about the server.

sigv commented 3 years ago

@sophia-guo, if the decision cannot be made about the target location for uploads, for a full repository -- could it be possible to push the .deb and .rpm packages to Github in the meantime, similar to how it happens for Windows .msis?

For example, jdk-17+35 can be downloaded as Windows x64 MSI, actually linking to adoptium/temurin17-binaries#jdk-17+35.

perlun commented 3 years ago

Agree. The top-most important thing (for me, as an end-user) is that the .deb and .rpm packages become available (and automatically built by CI); it doesn't have to be a "proper" apt-get source as the first step.

berndschatz commented 3 years ago

Agree. The top-most important thing (for me, as an end-user) is that the .deb and .rpm packages become available (and automatically built by CI); it doesn't have to be a "proper" apt-get source as the first step.

Yes, setup an own repo, push the deb/rpms in it and change a repo URI is fine. By switching from a package to tgz , you have to adapt your scripts and concepts, e.g. ,,update-alternatives --config java'' will not work anymore.

mfriedenhagen commented 3 years ago

Completely agree with @berndschatz and @perlun . I convinced my management back then that using adoptopenjdk would be a good idea because Debian maintainers have slower update cycles, we firstly rolled our own debs but happily switched to the official ones. Please upload them to GitHub, mirroring is much easier than building debs.