Disable resigning of already signed jars

[netomi]

As you are aware, signing of jars using the Eclipse Foundation signing service is currently quite slow as the service is in degraded mode and only support 1 concurrent signing atm.

While trying to mitigate that for now till we have a more scalable solution, I have noticed in the build at https://ci.eclipse.org/platform/job/eclipse.platform.releng.aggregator/job/master/ that there are lots of jars (> 800) that are signed for each build. Many of these jars are already signed, and you will find entries like that in the log file:

14:50:16  [INFO] Signing jar: /home/jenkins/agent/workspace/latform.releng.aggregator_master/equinox/bundles/org.eclipse.equinox.device/target/org.eclipse.equinox.device-1.2.100-SNAPSHOT.jar
14:50:19  [INFO] Jar '/home/jenkins/agent/workspace/latform.releng.aggregator_master/equinox/bundles/org.eclipse.equinox.device/target/org.eclipse.equinox.device-1.2.100-SNAPSHOT-sources.jar' is already signed and will be resigned.

I was wondering if it would not make sense to change the resigningStrategy of the eclipse-jarsigner plugin to something like DO_NOT_RESIGN instead of keeping at the default which is RESIGN.

More information about the configuration can be found at: https://eclipse-cbi.github.io/org.eclipse.cbi/eclipse-jarsigner-plugin/sign-mojo.html

Disable RESIGNING would reduce the number of signing operations considerably imho.

cc @merks

[akurtakov]

This is might be doable but has to be checked deeper by someone as resigning is done before the decision to replace or not with baseline version is taken

15:56:41  [INFO] --- eclipse-jarsigner:1.4.3:sign (sign) @ org.eclipse.osgi.tests ---
15:56:41  [INFO] Signing of nested jars of '/home/jenkins/agent/workspace/latform.releng.aggregator_master/equinox/bundles/org.eclipse.osgi.tests/target/org.eclipse.osgi.tests-3.19.100-SNAPSHOT.jar' is disabled.
15:56:41  [INFO] Jar '/home/jenkins/agent/workspace/latform.releng.aggregator_master/equinox/bundles/org.eclipse.osgi.tests/target/org.eclipse.osgi.tests-3.19.100-SNAPSHOT.jar' is already signed and will be resigned.
15:56:41  [INFO] Signing jar: /home/jenkins/agent/workspace/latform.releng.aggregator_master/equinox/bundles/org.eclipse.osgi.tests/target/org.eclipse.osgi.tests-3.19.100-SNAPSHOT.jar
15:56:46  [INFO] Signing of nested jars of '/home/jenkins/agent/workspace/latform.releng.aggregator_master/equinox/bundles/org.eclipse.osgi.tests/target/org.eclipse.osgi.tests-3.19.100-SNAPSHOT-sources.jar' is disabled.
15:56:46  [INFO] Jar '/home/jenkins/agent/workspace/latform.releng.aggregator_master/equinox/bundles/org.eclipse.osgi.tests/target/org.eclipse.osgi.tests-3.19.100-SNAPSHOT-sources.jar' is already signed and will be resigned.
15:56:46  [INFO] Signing jar: /home/jenkins/agent/workspace/latform.releng.aggregator_master/equinox/bundles/org.eclipse.osgi.tests/target/org.eclipse.osgi.tests-3.19.100-SNAPSHOT-sources.jar
15:56:50  [INFO] 
15:56:50  [INFO] --- tycho-p2:4.0.8:p2-metadata (p2-metadata) @ org.eclipse.osgi.tests ---
15:56:50  [INFO] MavenProject: org.eclipse.osgi:org.eclipse.osgi.tests:3.19.100-SNAPSHOT @ /home/jenkins/agent/workspace/latform.releng.aggregator_master/equinox/bundles/org.eclipse.osgi.tests/pom.xml
15:56:50      The main artifact has been replaced with the baseline version.
15:56:50      The following attached artifacts have been replaced with the baseline version: [sources]

Thus if the bundle is changed and not resigned - a not signed version will end up in the build output. On the other side, this verification build most likely can have signing disabled entirely as it's not supposed to publish anything to downloads.

[netomi]

if builds on master would not be signed at all, that would also be quite some relief for other projects imho.

[netomi]

the eclipse-jarsigner could also be extended to have another resigningStrategy, something like ONLY_RESIGN_IF_INVALID, where the jar is only resigned, if it does not have a valid signature. That would avoid the risk that the jar is not resigned if a signature is found, but the jar was modified and the signature is not valid anymore.

[HannesWell]

This is might be doable but has to be checked deeper by someone as resigning is done before the decision to replace or not with baseline version is taken

One can exclude certain files from the comparator as it is for example done in M2E: https://github.com/eclipse-m2e/m2e-core/blob/master/m2e-parent/pom.xml#L118-L119

IIRC for MANIFEST.MF only the main section is checked any ways so all signatures are ignored.

On the other side, this verification build most likely can have signing disabled entirely as it's not supposed to publish anything to downloads.

Thinking about it I wonder why artifacts signing was activated at all on the master. These artifacts are not published, only archived as Jenkins build artifacts. But I don't think they are used for anything else than checking build results.

[laeubi]

Signatures are (as far as I know) already excluded, anyways the log seem to suggest that the artifact is already signed, so skip the signing seems obvious, at least recently @merks needed to bump versions manually if signing changes anyways....

[HannesWell]

With https://github.com/eclipse-platform/eclipse.platform.releng.aggregator/pull/2140 signing will be disabled entirely for the master/maintenance branch builds of this repository. Assuming that there is one I-build and therefore at least one master build (submodule-updates) of this repo per day, in reality there is probably more, this should already reduce the number of signings of this repo to less than the half than it used to be.

Currently I find the String Signing jar: 973 times in the build logs of a single build. Assuming that each one means an actual sign operation and on average two builds (one I-build and one master build) ran per day, this means 2*973*365=710.290 sign operations per year. In reality it is probably higher due to more I-builds in milestone/RC weeks and 'manual' changes to the master. Btw. the number of times I find is already signed and will be resigned. is 798 in the last master build. But as said elsewhere, many of them are tests that are not published.

As a follow-up I created https://github.com/eclipse-platform/eclipse.platform.releng.aggregator/pull/2143 to reduce the signings even further by not re-signing the actual jars.

[merks]

You’re awesome 😎

[netomi]

tyvm, that should reduce the number of signing operations considerably and give us more room for implementing a more scalable signing service (they are usually charged by signing operations / year and our number is really high).

[akurtakov]

tyvm, that should reduce the number of signing operations considerably and give us more room for implementing a more scalable signing service (they are usually charged by signing operations / year and our number is really high).

Why doesn't EF recommend going PGP signing wherever possible (everything but native content more or less) in this case?

[netomi]

We discussed this internally. Adding jar signing support to the Eclipse IDE was apparently a difficult process as I have been told and we do not want to open another can of worms atm that was the conclusion.

Hopefully we have a more scalable solution soon.

[HannesWell]

FYI I just submitted https://github.com/eclipse-platform/eclipse.platform.releng.aggregator/pull/2143, which should again (usually) significantly reduce the number of signing operations in I-builds. If one is interested, tonight's I-build can be checked.

That should also reduce the overall runtime of I-builds.