search

bcgit / bc-java

Bouncy Castle Java Distribution (Mirror)
https://www.bouncycastle.org/java.html
MIT License
2.29k stars 1.13k forks source link

TlsFatalAlert: internal_error(80) at JcaTlsRSASigner.generateRawSignature with custom RSA provider #1722

Open pedrolamarao opened 3 months ago

pedrolamarao commented 3 months ago

During integration testing of our custom JCA security provider with Bouncy Castle JSSE provider, under certain circumstances, we encounter this exception:

Caused by: org.bouncycastle.tls.TlsFatalAlert: internal_error(80)
    at org.bouncycastle.tls.crypto.impl.jcajce.JcaTlsRSASigner.generateRawSignature(Unknown Source)
    at org.bouncycastle.tls.DefaultTlsCredentialedSigner.generateRawSignature(Unknown Source)
    at org.bouncycastle.tls.TlsUtils.generateServerKeyExchangeSignature(Unknown Source)
    at org.bouncycastle.tls.TlsECDHEKeyExchange.generateServerKeyExchange(Unknown Source)
    at org.bouncycastle.tls.TlsServerProtocol.handleHandshakeMessage(Unknown Source)
    at org.bouncycastle.tls.TlsProtocol.processHandshakeQueue(Unknown Source)
    at org.bouncycastle.tls.TlsProtocol.processRecord(Unknown Source)
    at org.bouncycastle.tls.RecordStream.readFullRecord(Unknown Source)
    at org.bouncycastle.tls.TlsProtocol.safeReadFullRecord(Unknown Source)
    at org.bouncycastle.tls.TlsProtocol.offerInput(Unknown Source)
    ... 154 more

cause by this:

Caused by: java.security.InvalidKeyException
    at dev.prodist.jca.RsaPkcsCipherImpl.engineInit(RsaPkcsCipherImpl.java:78)
    at java.base/javax.crypto.Cipher.init(Cipher.java:1283)
    at java.base/java.security.Signature$CipherAdapter.engineInitSign(Signature.java:1487)
    at java.base/java.security.Signature$Delegate.tryOperation(Signature.java:1311)
    at java.base/java.security.Signature$Delegate.chooseProvider(Signature.java:1257)
    at java.base/java.security.Signature$Delegate.engineInitSign(Signature.java:1368)
    at java.base/java.security.Signature.initSign(Signature.java:659)
    at org.bouncycastle.tls.crypto.impl.jcajce.JcaTlsRSASigner.getRawSigner(Unknown Source)
    ... 164 more

where dev.prodist.jca.RsaPkcsCipherImpl is our custom service provider.

Our providers operate with externalized keys through RPCs.

Inspecting generateRawSignature, it seems that a Signature service acquired for our PrivateKey implementation is subsequently being reinitialized with a generic memory-based PublicKey in order to do what appears to be a sanity check on key parity.

However useful it may be to anticipate key misconfiguration, we believe that JCA does not require our provider to be able to handle incompatible keys, and therefore this code is incorrectly reusing our Signature service with an incompatible key.

peterdettman commented 2 weeks ago

Thanks for the report. We've now removed the offending verification step, which should resolve this.