Open HannesWell opened 1 year ago
My only idea is to remove the offending signature from our p2-repo where we mirror parts of the SimRel.
HOW do you mirror the content? Because Nexus is known to break this because it returns invalid XML:
Should PGPSignatureVerifier.unnormalizedPGPProperty() also cover the -----BEGIN PGP MESSAGE----- case?
Seems reasonable, but wondering what PGP MESSAGE
means here, as it should actually be a signature/key? Have you tested that BC can parse this properly afterwards if you correct the error?
My only idea is to remove the offending signature from our p2-repo where we mirror parts of the SimRel.
HOW do you mirror the content? Because Nexus is known to break this because it returns invalid XML:
* [Allow PGP signature verification to be disabled eclipse-tycho/tycho#1239](https://github.com/eclipse-tycho/tycho/issues/1239)
Just build a product and copy the p2-repo created by the tycho-p2-repository-plugin:2.7.5
for it to a server.
Actually pretty bare metal. But I compared the pgp.signatures/publicKeys in the artifacts.xml
for org.antlr.runtime 3.2.0.v20220404-1927 the signatures in our repo and the SimRel 2022-12 repo and they are the same. So the mirroring seens to work.
Should PGPSignatureVerifier.unnormalizedPGPProperty() also cover the -----BEGIN PGP MESSAGE----- case?
Seems reasonable, but wondering what
PGP MESSAGE
means here, as it should actually be a signature/key? Have you tested that BC can parse this properly afterwards if you correct the error?
I'm not familiar enough with PGP to tell you what it should mean 😅
I tried but wasn't able to manipulate the string in the remote debugged VM, because after the replacement the string is not assigned to any variable before being converted to a byte-array.
What I could try was to replace the PGP MESSAGE
by PGP PUBLIC KEY BLOCK
but then BC complained about an invalid value so I assume they are not the same.
However manually removing the pgp.signature/publicKeys made the update work.
Some (recent?) versions of ~bouncycastle~ gpg seem to generate different headers/footers than the ones that are supported in p2 currently. The unnormalize should ideally cover all ------BEGIN PGP *------
cases.
Such format has been in the specification for long time:
https://www.rfc-editor.org/rfc/rfc4880#section-6.2
I would have expected bouncy castle could always consume it, even older version.
But testing the 2021-06 installer installing the Eierlegende Wollmilchsau 2022-12 I do see such errors
ERROR: org.eclipse.equinox.p2.engine code=0 session context was:(profile=D__Users_test20_all-latest-released_eclipse, phase=org.eclipse.equinox.internal.p2.engine.phases.Collect, operand=, action=).
ERROR: org.eclipse.equinox.p2.artifact.repository code=4 Result of processing steps.
OK: unknown code=0 OK
OK: unknown code=0 OK
OK: unknown code=0 OK
ERROR: org.eclipse.equinox.p2.artifact.repository code=0 Public key not found for 8122282445186051625.
OK: unknown code=0 OK
I would have expected bouncy castle could always consume it, even older version.
As mentioned by @HannesWell, the issue is more about how p2 "unnormalize" line breaks, sometimes incorrectly so the message that is passed to BouncyCastle is then incorrect. IIRC, the history behind this unnormalization is that initially, when we started playing with PGP, we were copying the keys and signatures as it in XML attributes. But by default, XML Writer will not allow line spaces. So unnormalize came as a hack to conveniently allow both formats, but this hack currently has some limitations.
Yes, though generally, as you suggest, the content is written with encoded characters so I don't think it's actually coming in in a form that needs to be re-processed to make it correct. I get more the sense that BC didn't implement everything at that point yet. But I'd need to run under debug control to verify that...
IIRC, the history behind this unnormalization is that initially, when we started playing with PGP, we were copying the keys and signatures as it in XML attributes.
That's why I'm wondering who has produced the data? Is the source faulty or the transformation or ...
@laeubi No, it works, it conforms to the specification, and is produced by BC. But apparently it just doesn't work (isn't consumable) by a really old version of BC.
I would have expected bouncy castle could always consume it, even older version.
As mentioned by @HannesWell, the issue is more about how p2 "unnormalize" line breaks, sometimes incorrectly so the message that is passed to BouncyCastle is then incorrect.
Yes that is what I suspect as well to be the root of this issue. As shown in my initial comment after the 'unnormalization' the header is split over multiple lines. And BC (correctly) complains about that invalid header.
IIRC, the history behind this unnormalization is that initially, when we started playing with PGP, we were copying the keys and signatures as it in XML attributes. But by default, XML Writer will not allow line spaces. So unnormalize came as a hack to conveniently allow both formats, but this hack currently has some limitations.
As far as I can tell, are the keys encoded into the attributes so shouldn't it be possible to restore them as they were initially and thus making the whole unnormalization obsolete?
Because in my debugging session everything looked correct until the unnormalization method was called.
On my laptop (that I don't use often for coding), I just tried to updated my Oomph provisioned Eclipse Modelling Tools 2021-12 to 22-12 and got a similar error that I think is also caused by this:
ERROR: org.eclipse.equinox.p2.engine code=4 An error occurred while collecting items to be installed
at org.eclipse.oomph.util.OomphPlugin.coreException(OomphPlugin.java:291)
at org.eclipse.oomph.p2.internal.core.ProfileTransactionImpl$3.commit(ProfileTransactionImpl.java:553)
at org.eclipse.oomph.p2.internal.core.ProfileTransactionImpl.commit(ProfileTransactionImpl.java:345)
at org.eclipse.oomph.setup.p2.impl.P2TaskImpl.perform(P2TaskImpl.java:904)
at org.eclipse.oomph.setup.internal.core.SetupTaskPerformer.doPerformNeededSetupTasks(SetupTaskPerformer.java:3851)
at org.eclipse.oomph.setup.internal.core.SetupTaskPerformer.access$1(SetupTaskPerformer.java:3794)
at org.eclipse.oomph.setup.internal.core.SetupTaskPerformer$WorkspaceUtil$1.run(SetupTaskPerformer.java:5178)
at org.eclipse.core.internal.resources.Workspace.run(Workspace.java:2313)
at org.eclipse.core.internal.resources.Workspace.run(Workspace.java:2338)
at org.eclipse.oomph.setup.internal.core.SetupTaskPerformer$WorkspaceUtil.performNeededSetupTasks(SetupTaskPerformer.java:5172)
at org.eclipse.oomph.setup.internal.core.SetupTaskPerformer$WorkspaceUtil.access$0(SetupTaskPerformer.java:5170)
at org.eclipse.oomph.setup.internal.core.SetupTaskPerformer.performNeededSetupTasks(SetupTaskPerformer.java:3785)
at org.eclipse.oomph.setup.internal.core.SetupTaskPerformer.performTriggeredSetupTasks(SetupTaskPerformer.java:3760)
at org.eclipse.oomph.setup.internal.core.SetupTaskPerformer.perform(SetupTaskPerformer.java:3638)
at org.eclipse.oomph.setup.ui.wizards.ProgressPage$9.run(ProgressPage.java:600)
at org.eclipse.oomph.setup.ui.wizards.ProgressPage$11$1.run(ProgressPage.java:727)
at org.eclipse.core.internal.jobs.Worker.run(Worker.java:63)
ERROR: org.eclipse.equinox.p2.engine code=0 session context was:(profile=C__dev_Eclipse-modelling_eclipse, phase=org.eclipse.equinox.internal.p2.engine.phases.Collect, operand=, action=).
ERROR: org.eclipse.equinox.p2.artifact.repository code=4 Result of processing steps.
OK: unknown code=0 OK
OK: unknown code=0 OK
OK: unknown code=0 OK
ERROR: org.eclipse.equinox.p2.artifact.repository code=0 Public key not found for 70b824d9a6b4ae29.
OK: unknown code=0 OK
ERROR: org.eclipse.equinox.p2.artifact.repository code=4 Result of processing steps.
OK: unknown code=0 OK
OK: unknown code=0 OK
OK: unknown code=0 OK
ERROR: org.eclipse.equinox.p2.artifact.repository code=0 Public key not found for 70b824d9a6b4ae29.
OK: unknown code=0 OK
Testing older installers, it seems anything before 2022-03 does not work well. I tested updating the 2022-03 modeling package to the latest, and that also works (but prompts from trusting bcpg because this older version didn't check for a PGP signature if the jar signature wasn't trusted). So yes, I think anything 2021-12 and older is not in good shape to handle updates. I haven't tested, but presumably they can be updated to something like 2022-03 and then can be updated to latest...
As far as I can tell, are the keys encoded into the attributes so shouldn't it be possible to restore them as they were initially and thus making the whole unnormalization obsolete?
In the early stage of PGP development, I was copy/pasting signatures and keys directly in artifacts.xml files. It was pretty convenient!
Is there anything that needs to be done for this issue?
@HannesWell
I'm still not sure if you want something to be fixed or if something needs to be fixed....
@HannesWell
I'm still not sure if you want something to be fixed or if something needs to be fixed....
Sorry for not replying before. I'm unsure. At the moment this issue does not happen anymore because of the additional check at https://github.com/eclipse-equinox/p2/blob/806b8b8940c5301bf39a0adcb97e693aad7ce2e4/bundles/org.eclipse.equinox.p2.artifact.repository/src/org/eclipse/equinox/internal/p2/artifact/processors/pgp/PGPSignatureVerifier.java#L174-L176 and because the PGP-keys are usually encoded in a p2-repos metadata and then decoded like in an ASCII-armored file, before being passed to this method. So it is fixed for the usual case, but maybe the following replacements should be enhanced to also match the other described cases, just in case the keys are not properly decoded for any scenario? But I cannot judge that well.
A year almost has passed - should this one be closed?
A year almost has passed - should this one be closed?
We had the described workaround in place and as also mentioned with the current implementation the problem does not occur anymore. I don't plan to work this at the moment and as time passes this probably becomes more and more irrelevant. So yes, let's close this.
@HannesWell @akurtakov @merks I sadly now encountered this problem also and it seems there is a -----BEGIN PGP MESSAGE-----
variant that is not handled in unnormalizedPGPProperty
Testing older installers, it seems anything before 2022-03 does not work well.
@merks do you mean that 2022-03 including is not working? Or should it work for 2022-03 onwards?
My comment was about updating finding keys not about invalid armor.
<artifact classifier="osgi.bundle" id="org.osgi.service.event" version="1.4.1.202109301733">
<properties size="14">
<property name="maven-groupId" value="org.osgi"/>
<property name="maven-artifactId" value="org.osgi.service.event"/>
<property name="maven-version" value="1.4.1"/>
<property name="maven-repository" value="eclipse.maven.central.mirror"/>
<property name="maven-type" value="jar"/>
<property name="download.checksum.sha-1" value="0f6d98f06834a911d04be512b751de3563d6b909"/>
<property name="download.size" value="47710"/>
<property name="artifact.size" value="47710"/>
<property name="download.md5" value="f4ff633129209e607b8cc02cfb248275"/>
<property name="download.checksum.sha-512" value="3c324d94e2852c31410ebdecd9a16845652d080fc0fc6fc708d58227b9708ed4c34de5d6d17aa841c430cab706b5f9df926a6bf1c0c313575533d2fd37f619ab"/>
<property name="download.checksum.md5" value="f4ff633129209e607b8cc02cfb248275"/>
<property name="download.checksum.sha-256" value="9f11c68980dbd931850f3f27fc7c35e1d710ecc728fe872bd6d786cd7e4b7b5e"/>
<property name="pgp.signatures" value="-----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEEnjBEBxt1jry35FZzcA5PObwFNksFAmQAZjcACgkQcA5PObwF NksYKQ//Yk/NSRwdVIZG25UjUxItLLvwGHiDd9bjEfp/K/LY9vtRS1LtCPLSQ6DM D+FQmxmQH6HLKHBQzaxptuEP/E428MuDiNlBUx96hIi8OeIwldV1M+4yJzTa42Ln CSk4Nto/bMK6Gw6fHCULk0Oxq7Ft6cesIay+j7zF23UP+uZaBdgs2evmiqz90Hx5 wkaVYS9OjA3O+n5A32gBu1t4Q4bPB1A7MobfpPKy0fmtqTgOIQfpYI8as0u+c08b e77+uz9CBXacPDmHpwqUfqXDwVi6yLifblCFM4kHkcPodBvuqnT1FQLPjmEl3zzZ RsIkm4A91hkQWFWfGa+wExPEztGikwlPrHfjDzvTo0Zq1iOS2vBdC9BTpMdYaEIo HQOuq0LBDLR8aukhuF0Iay4/eJIa7/5WbdIe7auyh/97o0PExfSJtWGI07QX/57T DtQ+wc0Kd71P9NiJGaORjICL08oz2bpF8DcId9xrcDI0XCqHmvyRhGzyieO9CwDK AppW0BFQ/pzC6K59EkwgF4imgrSnPrOjXIcGFcJhsNwgRM0djlOKTEzEzqDqe4L9 J7ZG8nRoUQvJZ/0/tbGsCxL4jlofjjyyp6LaGL8fN8Phy9zDFLQtQ5FhAY6e+nlZ BqhniC660TkgzE3ZJEGKzlPdAMOIzakjeoXHnXg/0GrrUF8VSyM= =iLKp -----END PGP SIGNATURE-----"/>
<property name="pgp.publicKeys" value="-----BEGIN PGP MESSAGE----- uQINBFhaXPsBEAC3bR7f5euHbpIDDTuFYHPI0+S5X0DhuqcGBUL2HSFhWMwIlfsA aO+pt7GyfXLUkTmzugwmwO+sOW2QmwEZQcK2z3BrcjytZophZ9AUajbAjnadSH6U XCMmfExVVnaYSfl/+Uub42szQE/r3gCRIz6M6clVVAjpFv4G/mumfQUV/XzLoUEY XTgwTokFJ97R+hDbHvBEBrUT8M6zHP5DhN3EBug3qb6wZVOa/+HEX3M+7k4jVT/p pNumw0acg0DDoSNQ13VsRV6sV0XE4zr3Zfs84f8xCgXpEMs4U6DZGqs3iJVVtbRf 0oL0fgcxNgRrmbCrBfbXYfrS4u+fJ0vB+Wrflv9eNA3i6TtVL6uYpZy9uO2B1olK VzfEhsgB3QrULB4jVHZjIXGe4ILn45ndMtAeY4M91wyobgG99Xl+1vPHrxV0+2zR P66J3puyxiKE2B7gd7hib54CB3lYyrG1S+K1kZGCI1IFKCnqmTJXY0tKoLAASS3v tDcknXenzR5RVSpWTDuxtusekfL0Bw8pCBoz9L4Hex8Q1j//D5CZlqcg1NKFfmBZ 7ta9PTuJcpOsz/LaPG/0VHYt/QAv5o4eeZESl7iZyM4/0NFh2s/rq0R8Z9yVSSkI vvO8d8XGZ65NTm3T4NFuEihn+AEm+zg4KiGdYBEZvs8QQoW9e1+MMN8xnwARAQAB iQREBBgBCAAPAhsCBQJhuzR9BQkSxtkCAinBXSAEGQEIAAYFAlhaXPsACgkQcA5P ObwFNkunSw//SRR1tGS1pDj2jonLpR0wPilCphS6ANv895yvlg6rHG4nKi4hQ0Jz ZxhGCwkgxEkRaKiyLfEiTihETkF161AqLPhyvE8LuQ1AG+A+tUnR8/T3gKE8t/m2 /UtScZwN1QEQVc/uG7MTrbZ2ngXfH65k3fzhjy95AnJHAswu2vic1hzDi77HlQpN 0O3adJuU/jfdu1RxNE0MRt8MFEjsTFwSBVm6lDxgcZV+qjRLGQznTyLF5/AyCI7Z 4z9xHZPKFq1eHzqevifNiqfb8KX22sHKOSdnVBzBq/UxbT5jIbNSRhD91FjtZD7Z 6wi3POsB/9RWZBldCov4ZEajmxFzxpx4RAqYOSIkEor9ZtRGbZuWvTie4vFIur7T f543mE6nxKcggExNp4MTyOd1scMc9oyczH561OTdHOCYEyoCwpG9N2Hb1/MDnWSi HKG451CvdrE5FHcPZKjp/nHUcRw/WQC3bgj6ScAay64EKC5S9tW+Wp85Oyyvj+M7 lBzOxp19nESpfC++fzBAQPMxtD8EvrZTxqFSJxMOH9bhzB8+MFt08tmYb5SwoYi4 C8JJ+wZgNetJKK+j07fvyMUChH/SbkCVszMiiSEjHA2Kk0LMVYKS/OLJU7i7tZXV aJ078QEeTDy5hSzsutd+orlFkR9+mgr1HUh0UgYlofTfEi7bLDeSr0cJELbTq5vM ZBKCicIP/irazYBVKw0SluhHtjzRcs5WIdH5bVPsEE87+iUc4daONWdVIhLdokxt OWlrEmZFLKqq9Z8fzvlf5LAQMOBkMAkl0z2ej4KG7zrjWyqDgysEI2WBlqTAFSeL +89Kc9BzJE9heYW8EfpXbNfOnKnAYWsbhcomSxVQ/jBIuyLBg/0gYKpBNx8HC6v9 xNH0Ja+wM/7w3JC1aIwMYJn1yF2ykUYS+BoTCU7TA8r43pHg4I4Fz+Y2P5RLk+RJ I4kJezDNiJOpIcr/nKTPxMGUzMtWlGyAJ7LkyOZCtQXhtXwaT8grjtHzlwlGrpgD Rtf7wWjzEWeaQSegTFM9Mid+09kCp0PkJvveg8wJCuoVboNOto0O5rQsUczjXxiW kXYlHGeQL4rWc1zP7F1n4DEwDbVZC7jOn/80l3x4LcKuhc86gP4L5HKbdjn5GcQ0 3RVLl1WVTQCdpr0+am28hl9XpyHdlWwSEmqqoUnjGv5B8RClocBRS4ECPPZCVSBl yK8eDgRww9Fu1EFq4xkq5fGj4YUOAIm756iW41NQ3VnPYbom/J27iFFN8+h92CSb KAqhmRwQh+GGo0eGCXmPHyQ/KCHTvnTZCFBUvabm3rVNFaDO+RvmwPwNCRz0DYzG paeMOGo4nMMGbzdhgfJ/X5Ed1/Mqz8egHhGIO94ebKEN5ZtJjAOK =xtSl -----END PGP MESSAGE-----"/>
</properties>
</artifact>
Such artifact properties might look like this.
The pgp.signatures
is decoded correctly but the pgp.publicKeys
not leading to numerous validation errors in case such items are mirrored / used.
We have updated our product from the Eclipse 2021-06 to 2022-12 and now notice that product updates fail with the following message:
In the error log the following exception is associated to that error:
Because of the stack-trace I remote debugged at
https://github.com/eclipse-equinox/p2/blob/343acf530ca293f624fb70cc5710dac5a5933890/bundles/org.eclipse.equinox.p2.artifact.repository/src/org/eclipse/equinox/internal/p2/artifact/processors/pgp/PGPSignatureVerifier.java#L117-L125
and noticed that in the context of
canonical: osgi.bundle,org.antlr.runtime,3.2.0.v20220404-1927
the signature in thepgp.publicKeys
property is converted inunnormalizedPGPProperty()
fromto (left out the equal inner part!)
And I assume that the header is afterwards malformed and therefore causing the exception.
Today the method looks like this and I assume the additional check is the reason that this does not occur if one updates from a more recent version: https://github.com/eclipse-equinox/p2/blob/806b8b8940c5301bf39a0adcb97e693aad7ce2e4/bundles/org.eclipse.equinox.p2.artifact.repository/src/org/eclipse/equinox/internal/p2/artifact/processors/pgp/PGPSignatureVerifier.java#L170-L183
Should
PGPSignatureVerifier.unnormalizedPGPProperty()
also cover the-----BEGIN PGP MESSAGE-----
case? I searched the 2022-12 SimRelartifacts.xml
and there indeed 84 hits for-----BEGIN PGP MESSAGE-----
.Does anybody has a good idea how to solve this in our case where we update from 2021-06 to 2022-12? My only idea is to remove the offending signature from our p2-repo where we mirror parts of the SimRel. Luckily org.antlr.runtime,3.2.0.v20220404-1927 is the only one we mirror with such header.