search

bcgit / bc-java

Bouncy Castle Java Distribution (Mirror)
https://www.bouncycastle.org/java.html
MIT License
2.3k stars 1.13k forks source link

Upgrading to 1.78 results in class not found errors when running in OSGI containers #1621

Open bpaunescu opened 6 months ago

bpaunescu commented 6 months ago

We've recently upgraded from 1.77 to 1.78 and are observing the following error:

Caused by: java.lang.NoClassDefFoundError: org/bouncycastle/util/io/pem/PemWriter at net.corda.crypto.impl.CipherSchemeMetadataProvider.objectToPem(CipherSchemeMetadataProvider.kt:239) at net.corda.crypto.impl.CipherSchemeMetadataProvider.encodeAsString(CipherSchemeMetadataProvider.kt:210) ... 116 more Caused by: java.lang.ClassNotFoundException: org.bouncycastle.util.io.pem.PemWriter not found by bcpkix-jdk18on [3] ... 118 more

Our platform uses OSGI. From what we can see, bcpkix-jdk18on library is missing a substantial amount of entries from the Import-Package section of its Manifest file. In 1.77 there are explicit import entries for bcprov packages which have now been removed and are causing these issues when using the libraries in OSGI environments.

tomaswolf commented 6 months ago

These broken manifests make BC 1.78 unusable in OSGi environments. Would be nice if this could be fixed and a patch release be published.

dghgit commented 6 months ago

Try what's on https://downloads.bouncycastle.org/betas/ if that works I'll put together a proper one upload it to Maven Central.

bpaunescu commented 6 months ago

@tomaswolf have you been able to successfully use the 1.78.1 versions linked above? We've installed those in our local Maven repo and the project is now failing to compile altogether.

Looking at the bcpkix manifest file, it's still missing the required import-package entries.

tomaswolf commented 6 months ago

@tomaswolf have you been able to successfully use the 1.78.1 versions linked above?

No, but I didn't try. I did download them and looked at the manifests, and they still have the same problem.

I guess the reason (for bcpg) is this manifest setting. This is missing something for importing "org.bouncycastle.*", possibly with a version (range). Same for the other bundles.

dghgit commented 6 months ago

Ah, thanks @tomaswolf - I seem to have fixed only one of 2 problems. I've uploaded a new set of 1.78.1 jars to https://downloads.bouncycastle.org/betas this one adds the missing import - they appear to activate in my local version of kharaf.

In case of caching, sha256 checksums for the latest:

6635bcdafb39922aef0245ecf291963db63e3d716e8176d7ebefb1d91609a4d0 core/build/libs/core-1.78.1.jar a1e8f245595c0b9ef2220d8940cfabb46d398702a53aa4e488a74f60a33b0c99 jmail/build/libs/bcjmail-jdk18on-1.78.1.jar 356041517799fed719c5bb22a5d145af3b20d97d16df6fe818acb454a4c81bf7 mail/build/libs/bcmail-jdk18on-1.78.1.jar ddfa5e85f74d335e241bc2bded9aebbf34ec36edb4c8f736f60d3c9723c513f2 mls/build/libs/bcmls-jdk18on-1.78.1.jar 1c699c555e548155d8e69800a3f3b6dd59c2409d0cd5cad2aa9bd52af1bb3ee2 pg/build/libs/bcpg-jdk18on-1.78.1.jar fac0fd1d4dbb8ffa1eaab0282a5ae6e87d122021316995ccbf01964efb272da2 pkix/build/libs/bcpkix-jdk18on-1.78.1.jar abaca1a1b71b9554c9d8afc0f7bcf39b3a767b622168173cdaa291c17d5b2770 prov/build/libs/bcprov-jdk18on-1.78.1.jar 272b6576d3742e53901e9221d5c968fb60d09e3086479f9e5694a9c1f2381e9d test/build/libs/test-1.78.1.jar 9fdd1bd605042d198757376bdb748ea2f878a535695734587d99d0a596926ae1 tls/build/libs/bctls-jdk18on-1.78.1.jar 388286e9823df9dd59606eb7183d861eae02bb432d453b285ae3dcfd2abf0ed5 util/build/libs/bcutil-jdk18on-1.78.1.jar

tomaswolf commented 6 months ago

The manifests look better, but this doesn't work yet.

In Eclipse we use bcprov, bcpkix, bcpg, and bcutil. With these 1.78.1 JARs I get:

java.lang.NoClassDefFoundError: org/bouncycastle/asn1/gnu/GNUObjectIdentifiers
    at org.bouncycastle.openpgp.PGPPublicKey.init(PGPPublicKey.java:109)
    at org.bouncycastle.openpgp.PGPPublicKey.<init>(PGPPublicKey.java:227)
    at org.bouncycastle.openpgp.PGPPublicKeyRing.<init>(PGPPublicKeyRing.java:122)
    at org.bouncycastle.openpgp.PGPObjectFactory.nextObject(PGPObjectFactory.java:130)
    at org.bouncycastle.openpgp.PGPPublicKeyRingCollection.<init>(PGPPublicKeyRingCollection.java:62)
    ...
Caused by: java.lang.ClassNotFoundException: org.bouncycastle.asn1.gnu.GNUObjectIdentifiers cannot be found by bcpg-jdk18on_1.78.1
    at org.eclipse.osgi.internal.loader.BundleLoader.generateException(BundleLoader.java:541)
    at org.eclipse.osgi.internal.loader.BundleLoader.findClass0(BundleLoader.java:536)
    at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:416)
    at org.eclipse.osgi.internal.loader.ModuleClassLoader.loadClass(ModuleClassLoader.java:168)
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:520)
    ... 65 more

bcpg imports org.bouncycastle.asn1.gnu, but bcutil doesn't export it (it's marked as a private package).

(The class also is twice in the git rpo, once here and once there.)

Apart from that:

precoder commented 6 months ago

I also tried with following 5 JARs from the betas site: bcutil-jdk18on-1.78.1.jar bcprov-jdk18on-1.78.1.jar bcpkix-jdk18on-1.78.1.jar bcpg-jdk18on-1.78.1.jar bcmail-jdk18on-1.78.1.jar

I get also NoClassDefFoundError for: org.bouncycastle.asn1.oiw.OIWObjectIdentifiers

`Caused by: java.lang.ExceptionInInitializerError: Exception java.lang.NoClassDefFoundError: org/bouncycastle/asn1/oiw/OIWObjectIdentifiers [in thread "FelixStartLevel"]

Thanks for your effort 👍

dghgit commented 6 months ago

Thanks for the additional info, I think this time I've actually managed to get the plugin and myself in agreement Have been able to set the bundle names back to what they were for 1.77 as well as well as deal with the versioning and the odd import issue. I found a couple more missing exports as well.

3c82b92198957dd6c1bba6ab16e131497257186187199158c33855bbbccc26c9 core/build/libs/core-1.78.1.jar 98857b92c8b8e6cafc86cb6612dd666f536a5dc900b155a003a50d0964ee25ac jmail/build/libs/bcjmail-jdk18on-1.78.1.jar 3a8706b961ca08da1f3ddf78f160d328f62bac1a2fe89fc73dd747f9de90f91b mail/build/libs/bcmail-jdk18on-1.78.1.jar 39c4df3975449e06cf60a3b665c0ca3164e9234fb2b95ad0bf58f27ca4548b40 mls/build/libs/bcmls-jdk18on-1.78.1.jar 0bc016838129e10c05035025994489f2d5ccf3658a975e0f6537ca6edf64e8b6 pg/build/libs/bcpg-jdk18on-1.78.1.jar 2a34524199f1c1d9cd79814f0e7c07e45f89d28431f220715a9b058dfb13435a pkix/build/libs/bcpkix-jdk18on-1.78.1.jar 3a53ca3b8048e561eef1151016fc5e07769a67d6cd405207239599fd5f77b17c prov/build/libs/bcprov-jdk18on-1.78.1.jar 77a93965d64e9bf6eb8f7711eb7dd0d8196d2d4efef5f91643e2ea0c637b4aa9 test/build/libs/test-1.78.1.jar 7447eee6e0a6cc4bd7d4f41ee590071079c27787cc58ab6a4e7bc160f66ba563 tls/build/libs/bctls-jdk18on-1.78.1.jar 9ac67614089053a0ebba86c7df330124ff2dff5cae13f2070df1f97c6c3a85c0 util/build/libs/bcutil-jdk18on-1.78.1.jar

Let me know, it it checks out, I'll put this one together as a release.

tomaswolf commented 6 months ago

Thanks for the update. In particular, thanks for changing back the Bundle-SymbolicNames.

Getting closer. This now works in Eclipse, but: the import version range is now "1.78.1". In OSGi, this means any version >= 1.78.1 is acceptable, even a future version 4.0.0. This is worse than "[1.78,2)".

This should be changed to "[1.78.1,1.79)".

tomaswolf commented 6 months ago

However, I run into troubles running unit tests. Might be a JUnit problem. In any case I get

java.lang.NoClassDefFoundError: org/bouncycastle/asn1/cryptlib/CryptlibObjectIdentifiers
    at org.bouncycastle.openpgp.PGPUtil$2.<init>(PGPUtil.java:71)
    at org.bouncycastle.openpgp.PGPUtil.<clinit>(PGPUtil.java:68)
    at org.eclipse.jgit.gpg.bc.internal.keys.KeyGripTest.readAsc(KeyGripTest.java:109)
    at org.eclipse.jgit.gpg.bc.internal.keys.KeyGripTest.testGrip(KeyGripTest.java:127)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:568)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
    at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
    at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
    at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
    at org.junit.runners.Suite.runChild(Suite.java:128)
    at org.junit.runners.Suite.runChild(Suite.java:27)
    at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
    at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
    at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
    at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
    at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:93)
    at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:40)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:529)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:756)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:452)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:210)
Caused by: java.lang.ClassNotFoundException: org.bouncycastle.asn1.cryptlib.CryptlibObjectIdentifiers
    at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
    at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:520)
    ... 39 more

To resolve this I have to add bcutil as an extra dependency to the bundle containing the tests.

Unclear where this comes from, but I do notice that the bcpg manifest has in its export packages:

org.bouncycastle.openpgp;uses:="org.bouncycastle.asn1,org.bouncycastle.bcpg,org.bouncycastle.bcpg.attr,org.bouncycastle.bcpg.sig,org.bouncycastle.gpg,org.bouncycastle.openpgp.operator,org.bouncycastle.util";version="1.78.1"

Class PGPUtil also imports various subpackages of asn1. Shouldn't those also appear in the "uses" clause?

dghgit commented 6 months ago

I've done another update which should properly fix the version issue. Here's the checksums:

865d1692fd412079dc3828c827df35286ea20897c5a5439e885c82612bd7c501 core/build/libs/core-1.78.1.jar bd17cb42bb2faf7103c707e06662f07fa6cec3d2c805bd3b4f19d5d61fa92413 jmail/build/libs/bcjmail-jdk18on-1.78.1.jar f9b2ea4b3e3f4774140143cf19539297ef313e96a34d9a2b649697897ed15711 mail/build/libs/bcmail-jdk18on-1.78.1.jar 2cec4266c1771a5d81c3752b53e1ee5d981399edb83fcb7194290df01f5d33d5 mls/build/libs/bcmls-jdk18on-1.78.1.jar 27afc2ff05b47be2878de2c934eed548919e42eb6325d6f7d0936dddd221ec90 pg/build/libs/bcpg-jdk18on-1.78.1.jar b7da20ed53cab20362706cce3df8093309e74bbe0e1ac166a1171f654acd6163 pkix/build/libs/bcpkix-jdk18on-1.78.1.jar c842e97caf4bd23085f04b749c2be88065134e2a29fbc297181770b0f6c73ae7 prov/build/libs/bcprov-jdk18on-1.78.1.jar ff65d110088461c4124d5fd53640c0b741275f47e74a4e6c478b2c4433e6bac9 test/build/libs/test-1.78.1.jar 080af1a6216e097253cd53ea3a2d47c2f2fd6d359511d8a64b9546029505de53 tls/build/libs/bctls-jdk18on-1.78.1.jar 2a0856f5fc5b0ca7aca922b3750a0b5997a8832419c805c5af724ae018d99480 util/build/libs/bcutil-jdk18on-1.78.1.jar

With bcpg, there is a dependency on it with bcutil now (the pom was missing this is as well, that's been fixed also). It's only a couple of classes at the moment (previously based in bcprov), but with the latest revisions to the PGP specification it's become obvious it's just going to get worse so we decided it was time to add the dependency.

Let me know how it goes. Thanks.

tomaswolf commented 6 months ago

Thanks; this now looks OK and works as far as I can test it. @precoder what about your problem case?

dghgit commented 6 months ago

Thanks @tomaswolf yes @precoder let me know if you're issue is also solved with the latest (I'm confident it is but would like to be sure) and I'll get the update out the door and on Maven Central.

precoder commented 6 months ago

Now I tested my setup again and it works without any ClassNotFoundExceptions. Thanks for your fast response time and fix :)

dghgit commented 6 months ago

1.78.1 is now starting to appear on Maven Central and bouncycastle.org.

yschimke commented 6 months ago

This is broken for me.

Trusted-Library: true
Application-Library-Allowable-Codebase: *
Bundle-Version: 1..78.1
Bnd-LastModified: 1713490297960
Bundle-ManifestVersion: 2
Import-Package: org.bouncycastle.asn1;version="",org.bouncycastle.asn1

Causing this build failure https://github.com/square/okhttp/actions/runs/8751257626/job/24016483153?pr=8364#step:7:762

 OsgiTest > testMainModuleWithSiblings() FAILED
    java.lang.IllegalArgumentException: Invalid version 1..78.1 in file build/resources/test/okhttp3/osgi/workspace/cnf/repo/put2137661645660471094.bnd
        at aQute.bnd.deployer.repository.LocalIndexedRepo.putArtifact(LocalIndexedRepo.java:378)
        at aQute.bnd.deployer.repository.LocalIndexedRepo.put(LocalIndexedRepo.java:448)
        at okhttp3.osgi.OsgiTest.deployFile(OsgiTest.kt:131)
        at okhttp3.osgi.OsgiTest.deployClassPath(OsgiTest.kt:123)
        at okhttp3.osgi.OsgiTest.prepareWorkspace(OsgiTest.kt:86)
        at okhttp3.osgi.OsgiTest.createWorkspace(OsgiTest.kt:77)
        at okhttp3.osgi.OsgiTest.testMainModuleWithSiblings(OsgiTest.kt:54)
dghgit commented 6 months ago

Oh boy... okay, I've fixed the ones available on the BC website, while dealing with that error I've also made the change @tomaswolf suggested of using "[1.78.1,1.79)" for imports as well.

If for some reason you're stuck on Maven Central, it's safe to keep using 1.78, there's 2 minor changes in 1.78.1, they're really edge conditions though, so annoying if you run into them, but if you don't, they simply don't matter. We can't replace the artifacts on Maven Central.

Let me know how you go.

yschimke commented 6 months ago

I'm not blocked, I'll just skip that bump

dghgit commented 6 months ago

Just so everyone is clear - this second issue applies to the jdk15to18 builds - they use a second, as it turns out, fragile, tool chain. The original issue reported here is still fixed. As things would have it, the jdk15to18 builds are actually okay for 1.78, the .1 release resulted in a script error which caused the invalid 1..78 to appear. People who must use Maven and are unaffected by the edge case that was also fixed in 1.78.1 can use the 1.78 release on Maven Central without OSGi issues - it's only if the rejecting invalid X.509 extensions issue affects you that you need 1.78.1 (which is now corrected for jdk15to18 on the BC website).

cheeyi commented 4 months ago

@dghgit Thanks for the update. I'm updating from 1.77 -> 1.78.1 and saw OSGI-related errors on Gradle that I did not see on 1.77 in an Android library project (see https://github.com/bcgit/bc-java/issues/1685). This issue also arose in 1.78 — is there a possibility some of these fixes may have introduced an issue with the Android build environment specifically?

dghgit commented 4 months ago

I've replied on #1685.

mmoloneysendsafely commented 3 months ago

I'm still having this issue as well. I'm trying to use the org.bouncycastle.openpgp.PGPUtil class from bcpg-jdk15to18:1.78.1, and that class tries to import org.bouncycastle.asn1.ASN1ObjectIdentifier from bcprov-jdk18on:1.78.1. However, that (and several other packages and classes) seem to have been removed from that release causing the NoClassDefFoundError.

Note: I'm not running in OSGI container, just simply trying to run my project in intelliJ.