search

Azure / azure-sdk-for-java

This repository is for active development of the Azure SDK for Java. For consumers of the SDK we recommend visiting our public developer docs at https://docs.microsoft.com/java/azure/ or our versioned developer docs at https://azure.github.io/azure-sdk-for-java.
MIT License
2.35k stars 1.99k forks source link

KV cryptographyClient.sign(SignatureAlgorithm.ES256,...) produces broken signature #36393

Open yongxinglai-netrust opened 1 year ago

yongxinglai-netrust commented 1 year ago

Describe the bug An invalid digital signature is created when using the CryptographyClient.sign() to sign the signature digest with ECC keys. The same function works when using RSA keys, i.e. digests signed using RSA produce a valid digital signature. Other than the difference in key type, the code used to create both signatures are the same.

To Reproduce Use CryptographyClient.sign() with ECC certificates.

Code Snippet // Produces a broken signature SignResult signResult = cryptographyClient.sign(SignatureAlgorithm.ES256, rawHash); // Produces a valid signature SignResult signResult = cryptographyClient.sign(SignatureAlgorithm.RS256, rawHash);

Expected behavior Using either ECC or RSA keys should produce a valid digital signature hash.

Setup (please complete the following information):

Additional context The key used in Keyvault had a certificate tied to it and is not a standalone key.

joshfree commented 1 year ago

@vcolin7 PTAL

vcolin7 commented 1 year ago

Hi @yongxinglai-netrust, thanks for reporting this and sorry for the late response. I've gone ahead and tried reproducing this issue unsuccessfully. Could you share the code you used to try to sign your data?

This is what I used in my test, by the way:

Provider[] providers = Security.getProviders();
Provider provider = null;

for (Provider currentProvider : providers) {
    if (currentProvider.containsValue("EC")) {
        provider = currentProvider;

        break;
    }
}

if (provider == null) {
    for (Provider currentProvider : providers) {
        System.out.println(currentProvider.getName());
    }

    fail(String.format("No suitable security provider for algorithm 'EC' was found."));
}

KeyPairGenerator generator = KeyPairGenerator.getInstance("EC", provider);
ECGenParameterSpec spec = new ECGenParameterSpec("secp256r1");

generator.initialize(spec);

KeyPair keyPair = generator.generateKeyPair();
JsonWebKey key = JsonWebKey.fromEc(keyPair, provider);
KeyVaultKey importedKey = client.importKey("testEcKey", key);
CryptographyClient cryptoClient = new CryptographyClientBuilder()
    .keyIdentifier(importedKey.getId())
    .credential(new DefaultAzureCredentialBuilder().build())
    .buildClient();

byte[] data = new byte[100];

new Random(0x1234567L).nextBytes(data);

MessageDigest md = MessageDigest.getInstance("SHA-256");
md.update(data);

byte[] digest = md.digest();

SignResult signResult = cryptoClient.sign(SignatureAlgorithm.ES256, digest);

Boolean verifyStatus =
    cryptoClient.verify(SignatureAlgorithm.ES256, digest, signResult.getSignature()).isValid();

assertTrue(verifyStatus);
yongxinglai-netrust commented 1 year ago

Hi @vcolin7,

This is the relevant portion of the code.

// This is initialized with the ID of the RSA/ECC key depending on the current signing use casse.
CryptographyClient cryptographyClient = new CryptographyClientBuilder()
        .credential(clientSecretCredential)
        .keyIdentifier(keyvaultID)
        .buildClient();

// base64Hash is the BASE64 encoded digest sent in from another application
byte[] rawHash = Base64.getDecoder().decode(base64Hash);

// This line is triggered for ECC signing only.
SignResult signResult = cryptographyClient.sign(SignatureAlgorithm.ES256, rawHash);
<OR>
// This line is triggered for RSA signing only.
SignResult signResult = cryptographyClient.sign(SignatureAlgorithm.RS256, rawHash);

// The signed digest is BASE64 encoded and returned to the calling application.
String base64SignedHash = Base64.getEncoder().encodeToString(signResult.getSignature());

My application uses Keyvault to sign the digest, while the actual digest generation and signature construction is done by the calling application. I have attached sample signatures AKV_ES256.pdf and AKV_RS256.pdf which are created through this process and Keyvault.

To eliminate other possibilities, I did a POC using regular Java code. The digest signing portion is replaced with this block.

byte[] rawHash = Base64.getDecoder().decode(base64Hash);

Signature signature = Signature.getInstance("NONEwithECDSA");
// privateKey is initalized from a hardcoded ECC key/cert.
signature.initSign(privateKey);
signature.update(rawHash);

byte[] signatureBytes = signature.sign();   
String base64SignedHash = Base64.getEncoder().encodeToString(signatureBytes);

The sample signature Local_NONEwithECDSA.pdf produced by this block is valid as you can see. Other elements such as the rest of the code or the certificate issuing CA, the application that assembles the signature etc are the same.

Please let me know if you require more information. Thank you.

vcolin7 commented 1 year ago

Hey @yongxinglai-netrust, I know we talked about you providing a POC in a separate conversation about this issue, but after re-reading this I think I might not have understood what the problem is exactly or what I'm supposed to look for in the PDF samples you shared. Maybe I should take a step back: what do you mean when you say a signature is not valid? Have you tried using the CryprographyClient.verify() operation using the signature from SignResult and your digest? The result of that call should tell you if the signature is valid. How are you checking for validity?

yongxinglai-netrust commented 1 year ago

Hi @vcolin7, apologies for the delay.

Essentially the expected output for CryprographyClient.sign(SignatureAlgorithm.ES256, ..) is different when compared to the output with CryprographyClient.sign(SignatureAlgorithm.RS256, ..). I am not referring to the difference in algorithms, but rather with SignatureAlgorithm.ES256 it appears to be doing more. I suspect the output is similar or the same as CryprographyClient.signData(SignatureAlgorithm.ES256, ..). CryprographyClient.verify() should work as intended, i.e. verification should check out if we verify using the same algorithm and that's not what I am raising the issue for.

I have made some progress on the POC, perhaps it would be best for wait for that to let me demonstrate the difference as even I am not sure the paragraph above describes the issue adequately.

vcolin7 commented 1 year ago

Since you are using two different signature algorithms in RS256 and ES256, the resulting signatures are expected to be different in both content and size. In fact, the RSA signature will always be larger (256-bit vs. 64-bit). Additionally, RSA signing is an idempotent operation, meaning that you will always get the same resulting signature from signing a given piece of data with a specific key multiple times. On the other hand, this is not true for ECDSA signing where every signature is generated using a random safe integer, so you shouldn't expect signatures generated with this method to ever be the same.

Also, I don't understand what you mean when you say the client "is doing more" with algorithm ES256, what do you mean by this? If the verify() and verifyData() operations return true for a given signature, then that means you can trust the data it was generated with came from a trusted party.

I tested signing and verifying multiple times with keys generated by different RSA and ECDSA algorithms, both locally and using the Key Vault service, and could not find any reason to think the signatures or verification results were incorrect.

yongxinglai-netrust commented 1 year ago

Hi @vcolin7,

Very sorry for the long delay. I had issues importing EC keys to Keyvault, in the end I resorted to generating one and exporting it out. That and a couple of urgent issues to sort out at work. In any case, here is the POC code.


public class POC {

    // Hardcoded keys and cert so that the Java implementation would use the same key when ran from AKV.
    private String rsaKeyBase64 = "MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQCi3Ayk1lfFTNw43eKUWTRMLxBpZZPb2tJr9VrjSdKWu40PLwr//G1DK/8mbmFamCeUFGaAgNhKzYiU5E0meUQuOIIglXrabN8Kd4D10itTei/87MjYsPSGQGrzAClR0UhR1FrngpKNYhL1nFmGVneQt8G6JBkN9byz9mDTyRcDPAbAOllgg+OkNjZ06vLgrabFhCDahMB/otMxLaBtz/LcIG7o8qpW4/TfQh8QlHK8E6WjtKtYfe8wS8H3m8PTX5MpBN19q4C4lmJkxodL28/IYUX0lv6Khj+QvquSnNeX2ddo+vKJ2JlL4z+B2LunWC+W8oW2hFzhDK4XpzVqyj8lAgMBAAECggEAFW+FMoUL1GpDajC6S/BZ09U2Acle5scsfw0yQ0xKLZDOBjuY5QyijP7TfIl2wufX/cOZojxTO7VFzuMlzHSVpsfFSOij3PALg2MrVC+viDNC8bVw0iSoaqs76HQS/DPhovOia2piQ72u5SkJF4d2AVROaWOq8I1xovZA1F9ZahFcyNri9DMdZAgc5Bv08i2rrjmZacetpgLeL1LFBPzPTfT0vcmHOAeAh4U6r5JQv0k9tw4QSVI/7eY1vXis+INs4uumZ15U29heH7QdFTA1YkoE36m6+hXWFtb3xWx8KXwzF6CKAC2OIfDPogDqJ6unZIF4r4gSdQ+2uWdFs3LYmQKBgQDJuVIfTgOgLwJZ7x/3oOdFivjQN9Mfh52oxsrTyGKZfWui4oG4M27zBcStEyXd24UAfdutPTQFlfvZgdePWMA2depnUPNYHX4CcJIYI6+meZFPdLcIwBM4SF8CWPvZzShJcHu25UORlo/baeIaM1eFNI0dinVhQnmGuxBwvFoTUwKBgQDOrcPY/m/Vjo/k8ctsysRdq6M+901k+D4Fum2Xh8EmesYMdo480h5kHKDQo1wsAgi1o85yZvN0MXx+o4RigOcFnWw2z+zM1kuoqYdUlh6rsm2x9BMqNQcORD8s7FcqXlvyZ71y0noiBImJKqTkvq1q7byu9BBZMHck9E0xsilMpwKBgDHH3wFEslZcY3soLL5iYdBinEZeEGbOg9S7wiADanmwXtiihxqa7r1gZgnEGOsoGill5clZujXKro0dosGISF7oOIFok7TiCk+10gfM/rLZe0edaPHq3JNeILwmk3hLTa7Z6F3ZNBSAkF24QojutKF1t4Fbwt3Olsjt7v+zK5NrAoGBAJ1meqvsglE6xF2Qw+LzvX6085tFjfyijeGg7EulyLS8dc4SEBtDzn9hyht6/fxKBEYJHXXE8A4CjhSIBY9bKhX9oOdRjL8nSPGHQphpx6vtNFt9TYcsYVY9JXvZ9jw/JnlDBmXG49q6Z+A9yNyoVrwDmFoJZ9X08hgDsjJxw8ArAoGBAJKYujTOOxTUGkp3IKwUHB605idPGJ8F15JXbaXWwBotw+6KgKEXQWxTkAXo9OvhTDoOfRlaMpv+pTz6bHFo9DlAao/rwZjhFOZ7U/xtU6rEjEmlCcDuL6XktRs7Kzdq1FyCpFaseHLJONhN0SaRhknhZMEh03Y0ZTOigoKXQM05";
    private String rsaCertBase64 = "MIIDKjCCAhKgAwIBAgIQEe+xQN2vQGiK2LF6ES8HZDANBgkqhkiG9w0BAQsFADASMRAwDgYDVQQDEwd0ZXN0UlNBMB4XDTIzMDkyMjA5MTcxNVoXDTI0MDkyMjA5MjcxNVowEjEQMA4GA1UEAxMHdGVzdFJTQTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKLcDKTWV8VM3Djd4pRZNEwvEGllk9va0mv1WuNJ0pa7jQ8vCv/8bUMr/yZuYVqYJ5QUZoCA2ErNiJTkTSZ5RC44giCVetps3wp3gPXSK1N6L/zsyNiw9IZAavMAKVHRSFHUWueCko1iEvWcWYZWd5C3wbokGQ31vLP2YNPJFwM8BsA6WWCD46Q2NnTq8uCtpsWEINqEwH+i0zEtoG3P8twgbujyqlbj9N9CHxCUcrwTpaO0q1h97zBLwfebw9NfkykE3X2rgLiWYmTGh0vbz8hhRfSW/oqGP5C+q5Kc15fZ12j68onYmUvjP4HYu6dYL5byhbaEXOEMrhenNWrKPyUCAwEAAaN8MHowDgYDVR0PAQH/BAQDAgWgMAkGA1UdEwQCMAAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMB8GA1UdIwQYMBaAFGBcScNbQVmzYU+7rWegpc3Z91p6MB0GA1UdDgQWBBRgXEnDW0FZs2FPu61noKXN2fdaejANBgkqhkiG9w0BAQsFAAOCAQEATx382lz8npuZFs1BXAzUljBnNzWk8VZHJUJ80a2OXvzGP7esukN1AmPhE0w+i8Ap4xM/ICOQsaAWbt70rkXc6PHQ9ccMliyqWrTqe+xPc8HGmQyDZzhTRNhcyL/Z64f7xKcON8t/g9l4qbsWcCAiv4aHs+YmEy4XuXKCTIQAof76pXzmkDW41zMH8n2qNNazoL1wW+jF1VTEeJrpIBEkfTVPNax5o9/ttw1ItVHjZTDmvC5CyBM/9QDYPqNOozkNFkvoQbqdgOhh0XdL/eNZI/EDSlS1kfdUEAcqUPqo8YaR3lFBBGu0OF+73ojIz9U6jzLUYJGHbhIQgJQLF5PORA==";
    private PrivateKey rsaKey;
    private X509Certificate rsaCert;    
    private String rsaKeyId = "<RSA Key ID in Keyvault>";

    private String eccKeyBase64 = "MIGTAgEAMBMGByqGSM49AgEGCCqGSM49AwEHBHkwdwIBAQQgjwvjnrrmcR3a9P7C5Ku31te8tpfjetFXds+1PFxPi0qgCgYIKoZIzj0DAQehRANCAAReqczQ2ejzgNUEeW1kTVJL3XJVkl2BDF10z395BYnE8hz/XnED8kTF3ExnxnDLiXe4Kdy1DFey7nGr3SfiKf+G";   
    private String eccCertBase64 = "MIIBnzCCAUSgAwIBAgIQIV+aj53gSWqyzq/PvxV28zAKBggqhkjOPQQDAjASMRAwDgYDVQQDEwd0ZXN0RUNDMB4XDTIzMDkyMjA5MTgwN1oXDTI0MDkyMjA5MjgwN1owEjEQMA4GA1UEAxMHdGVzdEVDQzBZMBMGByqGSM49AgEGCCqGSM49AwEHA0IABF6pzNDZ6POA1QR5bWRNUkvdclWSXYEMXXTPf3kFicTyHP9ecQPyRMXcTGfGcMuJd7gp3LUMV7LucavdJ+Ip/4ajfDB6MA4GA1UdDwEB/wQEAwIHgDAJBgNVHRMEAjAAMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAfBgNVHSMEGDAWgBTMxkPzTdI5btsQKwTEB3mMs/0DizAdBgNVHQ4EFgQUzMZD803SOW7bECsExAd5jLP9A4swCgYIKoZIzj0EAwIDSQAwRgIhAK04dHJ24rPJ5nWi5/DTeMx+XRGZIDu4Dd2wXR9dEdALAiEAoxRzz1dvRBLV+oI5uUvHtogUERk0+dm/GhU91Vte6iY=";
    private PrivateKey eccKey;
    private X509Certificate eccCert;
    private String eccKeyId = "<ECC Key ID in Keyvault>";

    // AKV credentials
    private String keyVaultClientId = "<Client ID>";
    private String keyVaultClientSecret = "<Client Secret>";
    private String keyVaultTenantId = "<Tenant ID>";

    // Source data, assume this is the document to be digitally signed on.
    private byte[] plaintext = "plaintext".getBytes();
    // SHA256 of the source data. This is effectively the document hash.
    private byte[] digest;
    // DER encapsulated data for RSA signing. ECC does not require this and uses digest instead.
    private byte[] derDigest;

    // Raw signature output without further processing.
    private String SIGN_NO_HASH_RAW = "SIGN_NO_HASH_RAW";
    private String SIGN_WITH_HASH_RAW = "SIGN_WITH_HASH_RAW";
    // Signature output with DER encapsulation for AKV raw sig or DER removal for Java raw sig.
    private String SIGN_NO_HASH_DER_WRAP = "SIGN_NO_HASH_DER_WRAP";
    private String SIGN_WITH_HASH_DER_WRAP = "SIGN_WITH_HASH_DER_WRAP";
    private String SIGN_NO_HASH_DER_UNWRAP = "SIGN_NO_HASH_DER_UNWRAP";
    private String SIGN_WITH_HASH_DER_UNWRAP = "SIGN_WITH_HASH_DER_UNWRAP";

    public void cryptAndCompare() throws Exception {

        // Populate the data to be signed.
        digest = hashData(plaintext);
        derDigest = derEncapDataForRSA(digest);
        createKeyCert("RSA");
        createKeyCert("ECC");

        // RSA comparison
        crossCompare("RSA", signJava("RSA"), signAKV("RSA"));

        // ECC comparison       
        crossCompare("ECC", signJava("ECC"), signAKV("ECC"));
    }

    public Map<String, String> signJava(String algorithm) throws Exception {

        Map<String, String> outputMap = new HashMap<String, String>();

        System.out.println("==================================================");
        System.out.println("Processing " + algorithm + " algorithm with Java Crypto implementation");
        System.out.println();

        // This block signs the given value with the algorithm without hashing in the algorithm.
        Signature signature = Signature.getInstance((algorithm.equals("RSA") ? "NONEwithRSA" : "NONEwithECDSA"));
        signature.initSign((algorithm.equals("RSA") ? rsaKey : eccKey));    
        signature.update((algorithm.equals("RSA") ? derDigest : digest));   
        byte[] signatureNoHash = signature.sign();
        outputMap.put(SIGN_NO_HASH_RAW, Base64.getEncoder().encodeToString(signatureNoHash));
        // If ECC, produce a DER wrapped or unwrapped signature as well for cross verification.
        if (!algorithm.equals("RSA"))
            outputMap.put(SIGN_NO_HASH_DER_UNWRAP,
                    Base64.getEncoder().encodeToString(derDeencapSignatureForECC(signatureNoHash)));

        signature.initVerify((algorithm.equals("RSA") ? rsaCert : eccCert).getPublicKey());
        signature.update((algorithm.equals("RSA") ? derDigest : digest));
        System.out.println("Algorithm " + (algorithm.equals("RSA") ? "NONEwithRSA" : "NONEwithECDSA")
                + " self sign and verify success = " + signature.verify(signatureNoHash));

        // This block signs the given value with the algorithm after implicit hashing and DER encapsulation.
        signature = Signature.getInstance((algorithm.equals("RSA") ? "SHA256withRSA" : "SHA256withECDSA"));
        signature.initSign((algorithm.equals("RSA") ? rsaKey : eccKey));
        signature.update(plaintext);        
        byte[] signatureWithHash = signature.sign();
        outputMap.put(SIGN_WITH_HASH_RAW, Base64.getEncoder().encodeToString(signatureWithHash));
        // If ECC, produce a DER wrapped or unwrapped signature as well for cross verification.
        if (!algorithm.equals("RSA"))
            outputMap.put(SIGN_WITH_HASH_DER_UNWRAP,
                    Base64.getEncoder().encodeToString(derDeencapSignatureForECC(signatureWithHash)));

        signature.initVerify((algorithm.equals("RSA") ? rsaCert : eccCert).getPublicKey());
        signature.update(plaintext);
        System.out.println("Algorithm " + (algorithm.equals("RSA") ? "SHA256withRSA" : "SHA256withECDSA")
                + " self sign and verify success = " + signature.verify(signatureWithHash));

        // This block cross verifies the signatures to show that all steps in signature generation is shown.
        signature = Signature.getInstance((algorithm.equals("RSA") ? "NONEwithRSA" : "NONEwithECDSA"));
        signature.initVerify((algorithm.equals("RSA") ? rsaCert : eccCert).getPublicKey());
        signature.update((algorithm.equals("RSA") ? derDigest : digest));
        System.out.println("Algorithm manual hash and " + (algorithm.equals("RSA") ? "NONEwithRSA" : "NONEwithECDSA") + 
                " against " + (algorithm.equals("RSA") ? "SHA256withRSA" : "SHA256withECDSA")
                + " signature cross verify success = " + signature.verify(signatureWithHash));

        signature = Signature.getInstance((algorithm.equals("RSA") ? "SHA256withRSA" : "SHA256withECDSA"));
        signature.initVerify((algorithm.equals("RSA") ? rsaCert : eccCert).getPublicKey());
        signature.update(plaintext);
        System.out.println("Algorithm " + (algorithm.equals("RSA") ? "SHA256withRSA" : "SHA256withECDSA") + 
                " against manual hash and " + (algorithm.equals("RSA") ? "NONEwithRSA" : "NONEwithECDSA")
                + " signature cross verify success = " + signature.verify(signatureNoHash));
        System.out.println("--------------------------------------------------");

        System.out.println();
        System.out.println("SIGN_NO_HASH_RAW signature output = " + outputMap.get(SIGN_NO_HASH_RAW));
        System.out.println("SIGN_WITH_HASH_RAW signature output = " + outputMap.get(SIGN_WITH_HASH_RAW));
        if (!algorithm.equals("RSA")) {
            System.out.println("SIGN_NO_HASH_DER_UNWRAP signature output = " + outputMap.get(SIGN_NO_HASH_DER_UNWRAP));
            System.out.println("SIGN_WITH_HASH_DER_UNWRAP signature output = " + outputMap.get(SIGN_WITH_HASH_DER_UNWRAP));
        }

        System.out.println("==================================================");
        System.out.println();
        System.out.println();

        return outputMap;
    }

    public Map<String, String> signAKV(String algorithm) throws Exception {

        Map<String, String> outputMap = new HashMap<String, String>();

        ClientSecretCredential clientSecretCredential = new ClientSecretCredentialBuilder().clientId(keyVaultClientId)
                .clientSecret(keyVaultClientSecret).tenantId(keyVaultTenantId).build();

        CryptographyClient cryptographyClient = new CryptographyClientBuilder()
                .credential(clientSecretCredential)
                .keyIdentifier((algorithm.equals("RSA") ? rsaKeyId : eccKeyId))
                .buildClient();

        System.out.println("==================================================");
        System.out.println("Processing " + algorithm + " algorithm with AKV Crypto implementation");
        System.out.println();

        // This block signs the given value with the algorithm without hashing in the algorithm.    
        // AKV RSA does the DER encoding, and expects a hash.
        SignResult signResult = cryptographyClient
                .sign((algorithm.equals("RSA") ? SignatureAlgorithm.RS256 : SignatureAlgorithm.ES256), digest);
        byte[] signatureNoHash = signResult.getSignature();
        outputMap.put(SIGN_NO_HASH_RAW, Base64.getEncoder().encodeToString(signatureNoHash));
        // If ECC, produce a DER wrapped or unwrapped signature as well for cross verification.
        if (!algorithm.equals("RSA"))
            outputMap.put(SIGN_NO_HASH_DER_WRAP,
                    Base64.getEncoder().encodeToString(derEncapSignatureForECC(signatureNoHash)));

        VerifyResult verifyResult = cryptographyClient.verify(
                (algorithm.equals("RSA") ? SignatureAlgorithm.RS256 : SignatureAlgorithm.ES256), digest,
                signatureNoHash);
        System.out.println("CryptographyClient"
                + (algorithm.equals("RSA") ? ".sign(SignatureAlgorithm.RS256)" : ".sign(SignatureAlgorithm.ES256)")
                + " self sign and verify success = " + verifyResult.isValid());

        // This block signs the given value with the algorithm after implicit hashing and DER encapsulation.
        signResult = cryptographyClient
                .signData((algorithm.equals("RSA") ? SignatureAlgorithm.RS256 : SignatureAlgorithm.ES256), plaintext);
        byte[] signatureWithHash = signResult.getSignature();
        outputMap.put(SIGN_WITH_HASH_RAW, Base64.getEncoder().encodeToString(signatureWithHash));
        // If ECC, produce a DER wrapped or unwrapped signature as well for cross verification.
        if (!algorithm.equals("RSA"))
            outputMap.put(SIGN_WITH_HASH_DER_WRAP,
                    Base64.getEncoder().encodeToString(derEncapSignatureForECC(signatureWithHash)));

        verifyResult = cryptographyClient.verifyData(
                (algorithm.equals("RSA") ? SignatureAlgorithm.RS256 : SignatureAlgorithm.ES256), plaintext,
                signatureWithHash);
        System.out.println("CryptographyClient"
                + (algorithm.equals("RSA") ? ".signData(SignatureAlgorithm.RS256)"
                        : ".signData(SignatureAlgorithm.ES256)")
                + " self sign and verify success = " + verifyResult.isValid());

        // This block cross verifies the signatures to show that all steps in signature generation is shown.
        verifyResult = cryptographyClient.verify(
                (algorithm.equals("RSA") ? SignatureAlgorithm.RS256 : SignatureAlgorithm.ES256), digest,
                signatureWithHash);
        System.out.println("CryptographyClient"
                + (algorithm.equals("RSA") ? ".verify(SignatureAlgorithm.RS256)" : ".verify(SignatureAlgorithm.ES256)")
                + " against CryptographyClient" + (algorithm.equals("RSA") ? ".signData(SignatureAlgorithm.RS256)"
                        : ".signData(SignatureAlgorithm.ES256)")
                + " cross verify success = " + verifyResult.isValid());

        verifyResult = cryptographyClient.verifyData(
                (algorithm.equals("RSA") ? SignatureAlgorithm.RS256 : SignatureAlgorithm.ES256), plaintext,
                signatureNoHash);
        System.out.println("CryptographyClient"
                + (algorithm.equals("RSA") ? ".verifyData(SignatureAlgorithm.RS256)" : ".verifyData(SignatureAlgorithm.ES256)")
                + " against CryptographyClient" + (algorithm.equals("RSA") ? ".sign(SignatureAlgorithm.RS256)"
                        : ".sign(SignatureAlgorithm.ES256)")
                + " cross verify success = " + verifyResult.isValid());
        System.out.println("--------------------------------------------------");

        System.out.println();
        System.out.println("SIGN_NO_HASH_RAW signature output = " + outputMap.get(SIGN_NO_HASH_RAW));
        System.out.println("SIGN_WITH_HASH_RAW signature output = " + outputMap.get(SIGN_WITH_HASH_RAW));       
        if (!algorithm.equals("RSA")) {
            System.out.println("SIGN_NO_HASH_DER_WRAP signature output = " + outputMap.get(SIGN_NO_HASH_DER_WRAP));
            System.out.println("SIGN_WITH_HASH_DER_WRAP signature output = " + outputMap.get(SIGN_WITH_HASH_DER_WRAP));
        }       

        System.out.println("==================================================");
        System.out.println();
        System.out.println();

        return outputMap;
    }

    public void crossCompare(String algorithm, Map<String, String> javaOutputMap, Map<String, String> akvOutputMap) throws Exception {

        ClientSecretCredential clientSecretCredential = new ClientSecretCredentialBuilder().clientId(keyVaultClientId)
                .clientSecret(keyVaultClientSecret).tenantId(keyVaultTenantId).build();

        CryptographyClient cryptographyClient = new CryptographyClientBuilder()
                .credential(clientSecretCredential)
                .keyIdentifier((algorithm.equals("RSA") ? rsaKeyId : eccKeyId))
                .buildClient();

        System.out.println("==================================================");
        System.out.println("Processing " + algorithm + " algorithm cross comparison");
        System.out.println();

        // This block cross verifies the signatures generated by AKV with Java implementation
        Signature signature = Signature.getInstance((algorithm.equals("RSA") ? "NONEwithRSA" : "NONEwithECDSA"));
        signature.initVerify((algorithm.equals("RSA") ? rsaCert : eccCert).getPublicKey());
        signature.update((algorithm.equals("RSA") ? derDigest : digest));
        System.out.println("Algorithm manual hash and " + (algorithm.equals("RSA") ? "NONEwithRSA" : "NONEwithECDSA")
                + " against CryptographyClient"
                + (algorithm.equals("RSA") ? ".sign(SignatureAlgorithm.RS256)" : ".sign(SignatureAlgorithm.ES256)")
                + "cross verify success = " + signature.verify(Base64.getDecoder().decode(
                        akvOutputMap.get((algorithm.equals("RSA") ? SIGN_NO_HASH_RAW : SIGN_NO_HASH_DER_WRAP)))));

        signature = Signature.getInstance((algorithm.equals("RSA") ? "NONEwithRSA" : "NONEwithECDSA"));
        signature.initVerify((algorithm.equals("RSA") ? rsaCert : eccCert).getPublicKey());
        signature.update((algorithm.equals("RSA") ? derDigest : digest));
        System.out.println("Algorithm manual hash and " + (algorithm.equals("RSA") ? "NONEwithRSA" : "NONEwithECDSA")
                + " against CryptographyClient"
                + (algorithm.equals("RSA") ? ".signData(SignatureAlgorithm.RS256)"
                        : ".signData(SignatureAlgorithm.ES256)")
                + "cross verify success = " + signature.verify(Base64.getDecoder().decode(
                        akvOutputMap.get((algorithm.equals("RSA") ? SIGN_WITH_HASH_RAW : SIGN_WITH_HASH_DER_WRAP)))));

        signature = Signature.getInstance((algorithm.equals("RSA") ? "SHA256withRSA" : "SHA256withECDSA"));
        signature.initVerify((algorithm.equals("RSA") ? rsaCert : eccCert).getPublicKey());
        signature.update(plaintext);
        System.out.println("Algorithm " + (algorithm.equals("RSA") ? "SHA256withRSA" : "SHA256withECDSA")
                + " against CryptographyClient"
                + (algorithm.equals("RSA") ? ".sign(SignatureAlgorithm.RS256)" : ".sign(SignatureAlgorithm.ES256)")
                + "cross verify success = " + signature.verify(Base64.getDecoder().decode(
                        akvOutputMap.get((algorithm.equals("RSA") ? SIGN_NO_HASH_RAW : SIGN_NO_HASH_DER_WRAP)))));

        signature = Signature.getInstance((algorithm.equals("RSA") ? "SHA256withRSA" : "SHA256withECDSA"));
        signature.initVerify((algorithm.equals("RSA") ? rsaCert : eccCert).getPublicKey());
        signature.update(plaintext);
        System.out.println("Algorithm " + (algorithm.equals("RSA") ? "SHA256withRSA" : "SHA256withECDSA")
                + " against CryptographyClient"
                + (algorithm.equals("RSA") ? ".signData(SignatureAlgorithm.RS256)"
                        : ".signData(SignatureAlgorithm.ES256)")
                + "cross verify success = " + signature.verify(Base64.getDecoder().decode(
                        akvOutputMap.get((algorithm.equals("RSA") ? SIGN_WITH_HASH_RAW : SIGN_WITH_HASH_DER_WRAP)))));

        // This block cross verifies the signatures generated by Java with AKV implementation
        VerifyResult verifyResult = cryptographyClient.verify(
                (algorithm.equals("RSA") ? SignatureAlgorithm.RS256 : SignatureAlgorithm.ES256), digest,
                Base64.getDecoder().decode(
                        javaOutputMap.get((algorithm.equals("RSA") ? SIGN_NO_HASH_RAW : SIGN_NO_HASH_DER_UNWRAP))));
        System.out.println("CryptographyClient"
                + (algorithm.equals("RSA") ? ".verify(SignatureAlgorithm.RS256)" : ".verify(SignatureAlgorithm.ES256)")
                + " against Algorithm manual hash and " + (algorithm.equals("RSA") ? "NONEwithRSA" : "NONEwithECDSA")
                + " cross verify success = " + verifyResult.isValid());

        verifyResult = cryptographyClient.verify(
                (algorithm.equals("RSA") ? SignatureAlgorithm.RS256 : SignatureAlgorithm.ES256), digest,
                Base64.getDecoder().decode(
                        javaOutputMap.get((algorithm.equals("RSA") ? SIGN_WITH_HASH_RAW : SIGN_WITH_HASH_DER_UNWRAP))));
        System.out.println("CryptographyClient"
                + (algorithm.equals("RSA") ? ".verify(SignatureAlgorithm.RS256)" : ".verify(SignatureAlgorithm.ES256)")
                + " against Algorithm " + (algorithm.equals("RSA") ? "SHA256withRSA" : "SHA256withECDSA")
                + " cross verify success = " + verifyResult.isValid());

        verifyResult = cryptographyClient.verifyData(
                (algorithm.equals("RSA") ? SignatureAlgorithm.RS256 : SignatureAlgorithm.ES256), plaintext,
                Base64.getDecoder().decode(
                        javaOutputMap.get((algorithm.equals("RSA") ? SIGN_NO_HASH_RAW : SIGN_NO_HASH_DER_UNWRAP))));
        System.out.println("CryptographyClient"
                + (algorithm.equals("RSA") ? ".verifyData(SignatureAlgorithm.RS256)"
                        : ".verifyData(SignatureAlgorithm.ES256)")
                + " against Algorithm manual hash and " + (algorithm.equals("RSA") ? "NONEwithRSA" : "NONEwithECDSA")
                + " cross verify success = " + verifyResult.isValid());

        verifyResult = cryptographyClient.verifyData(
                (algorithm.equals("RSA") ? SignatureAlgorithm.RS256 : SignatureAlgorithm.ES256), plaintext,
                Base64.getDecoder().decode(
                        javaOutputMap.get((algorithm.equals("RSA") ? SIGN_WITH_HASH_RAW : SIGN_WITH_HASH_DER_UNWRAP))));
        System.out.println("CryptographyClient"
                + (algorithm.equals("RSA") ? ".verifyData(SignatureAlgorithm.RS256)"
                        : ".verifyData(SignatureAlgorithm.ES256)")
                + " against Algorithm "
                + (algorithm.equals("RSA") ? "SHA256withRSA" : "SHA256withECDSA")
                + " cross verify success = " + verifyResult.isValid());

        System.out.println("==================================================");
        System.out.println();
        System.out.println();
    }

    private void createKeyCert(String algorithm) throws Exception {

        // Create the keys and cert instances.
        KeyFactory keyFactory = KeyFactory.getInstance((algorithm.equals("RSA") ? "RSA" : "EC"));
        PKCS8EncodedKeySpec keySpec = new PKCS8EncodedKeySpec(
                Base64.getDecoder().decode((algorithm.equals("RSA") ? rsaKeyBase64 : eccKeyBase64)));
        if (algorithm.equals("RSA")) 
            rsaKey = (RSAPrivateKey) keyFactory.generatePrivate(keySpec);
        else
            eccKey = (ECPrivateKey) keyFactory.generatePrivate(keySpec);

        InputStream targetStream = new ByteArrayInputStream(
                Base64.getDecoder().decode((algorithm.equals("RSA") ? rsaCertBase64 : eccCertBase64)));
        if (algorithm.equals("RSA")) 
            rsaCert = (X509Certificate) CertificateFactory.getInstance("X509").generateCertificate(targetStream);
        else
            eccCert = (X509Certificate) CertificateFactory.getInstance("X509").generateCertificate(targetStream);       
    }   

    private byte[] hashData(byte[] bytesToHash) throws Exception {
        return MessageDigest.getInstance("SHA-256").digest(bytesToHash);
    }

    private byte[] derEncapDataForRSA(byte[] bytesToEncap) throws Exception {               

        // Construct the DigestInfo with the hashed value of plaintext. 
        // This is done implicitly in algorithms like SHA256withRSA, SHA256withECDSA and on AKV's signData(SignatureAlgorithm.RS256) function.
        // https://stackoverflow.com/questions/69750026/create-sha256withrsa-in-two-steps
        // https://datatracker.ietf.org/doc/html/rfc8017#page-47
        byte[] id = new byte[] { 0x30, 0x31, 0x30, 0x0d, 0x06, 0x09, 0x60, (byte) 0x86, 0x48, 0x01, 0x65, 0x03, 0x04, 0x02, 0x01, 0x05, 0x00, 0x04, 0x20 };
        byte[] derDigestInfo = new byte[id.length + bytesToEncap.length];
        System.arraycopy(id, 0, derDigestInfo, 0, id.length);
        System.arraycopy(bytesToEncap, 0, derDigestInfo, id.length, bytesToEncap.length);

        return derDigestInfo;
    }

    private byte[] derEncapSignatureForECC(byte[] signature) throws Exception {             

        byte[] id1 = new byte[] { 0x30, 0x44, 0x02, 0x20 };
        byte[] id2 = new byte[] { 0x02, 0x20 };
        byte[] derDigestInfo = new byte[id1.length + id2.length + signature.length];
        System.arraycopy(id1, 0, derDigestInfo, 0, id1.length);
        System.arraycopy(signature, 0, derDigestInfo, id1.length, 32);
        System.arraycopy(id2, 0, derDigestInfo, (id1.length + 32), id2.length);
        System.arraycopy(signature, 32, derDigestInfo, (id1.length + 32 + id2.length), 32);

        return derDigestInfo;
    }

    private byte[] derDeencapSignatureForECC(byte[] derDigestInfo) throws Exception {               

        byte[] rawSignature = new byte[64];
        Boolean firstPositionLeadingNull = false, secondPositionLeadingNull = false;
        // First 3 bytes are always removed.
        // If 4th byte is 0x20, expect 32 data bytes. If x021, expect 33 bytes with a leading 0x00 to be removed.
        if (derDigestInfo[3] == 0x21) 
            firstPositionLeadingNull = true;                
        System.arraycopy(derDigestInfo, ((firstPositionLeadingNull) ? 5 : 4), rawSignature, 0, 32);

        // If 2nd byte after above is 0x20, expect 32 data bytes. If x021, expect 33 bytes with a leading 0x00 to be removed.
        int midPointPos = ((firstPositionLeadingNull) ? 5 + 32 : 4 + 32);
        if (derDigestInfo[midPointPos + 1] == 0x21)  
            secondPositionLeadingNull = true;
        System.arraycopy(derDigestInfo, ((secondPositionLeadingNull) ? 3 + midPointPos : 2 + midPointPos), rawSignature, 32, 32);

        return rawSignature;
    }
}

The main function is cryptAndCompare(), so you can begin reading from there. The approach is to generate signatures in Java and Keyvault then cross verify them. The aim is to identify the extra bits of what I thought CryptographyClient.sign() was doing. Turns out instead it was not doing something required instead.

The POC runs for RSA as well, this serves as a control group to prove the concept works. The fact that RSA signatures are always the same helps to prove that too. Also critical is the key used for Java and Keyvault must be same, attached here. No passwords. The hardcoded key/cert string in the code are based off these PFXs. Keys_PFX.zip

Output for RSA (Keyvault SDK logs stripped)

==================================================
Processing RSA algorithm with Java Crypto implementation

Algorithm NONEwithRSA self sign and verify success = true
Algorithm SHA256withRSA self sign and verify success = true
Algorithm manual hash and NONEwithRSA against SHA256withRSA signature cross verify success = true
Algorithm SHA256withRSA against manual hash and NONEwithRSA signature cross verify success = true
--------------------------------------------------

SIGN_NO_HASH_RAW signature output = fFbfLMngLQU/k7/DT0Axm3DJpal2SThV6zC02ppIvg1wuA2rj2AYR9P0qGDg9FVXGioi0AB06QCSzJIfQB7mMaJBkCdqIlsCu9o1nxex8eJhTHtbm5Q1FqD/qQebOBdRMncAfox2MDON9gkwxZ6kYTgwbWrczF7jaIeehFpYBNeO5hWE8kmWAGh94jJkylPxPCdNWHLB++nntlKwP+5pJ5lJa1lE1Cdqf5aLiuRxss6tSuVyyRSM2GZUjgy4nKCG9/F9DugT0Zn+76YYxGNESUbtLERO8lwY98r8e0lDrM9CX74+V12ss03uE0RNrFegnCdwHSCZwye64gL8ZMVBiA==
SIGN_WITH_HASH_RAW signature output = fFbfLMngLQU/k7/DT0Axm3DJpal2SThV6zC02ppIvg1wuA2rj2AYR9P0qGDg9FVXGioi0AB06QCSzJIfQB7mMaJBkCdqIlsCu9o1nxex8eJhTHtbm5Q1FqD/qQebOBdRMncAfox2MDON9gkwxZ6kYTgwbWrczF7jaIeehFpYBNeO5hWE8kmWAGh94jJkylPxPCdNWHLB++nntlKwP+5pJ5lJa1lE1Cdqf5aLiuRxss6tSuVyyRSM2GZUjgy4nKCG9/F9DugT0Zn+76YYxGNESUbtLERO8lwY98r8e0lDrM9CX74+V12ss03uE0RNrFegnCdwHSCZwye64gL8ZMVBiA==
==================================================

==================================================
Processing RSA algorithm with AKV Crypto implementation

CryptographyClient.sign(SignatureAlgorithm.RS256) self sign and verify success = true
CryptographyClient.signData(SignatureAlgorithm.RS256) self sign and verify success = true
CryptographyClient.verify(SignatureAlgorithm.RS256) against CryptographyClient.signData(SignatureAlgorithm.RS256) cross verify success = true
CryptographyClient.verifyData(SignatureAlgorithm.RS256) against CryptographyClient.sign(SignatureAlgorithm.RS256) cross verify success = true
--------------------------------------------------

SIGN_NO_HASH_RAW signature output = fFbfLMngLQU/k7/DT0Axm3DJpal2SThV6zC02ppIvg1wuA2rj2AYR9P0qGDg9FVXGioi0AB06QCSzJIfQB7mMaJBkCdqIlsCu9o1nxex8eJhTHtbm5Q1FqD/qQebOBdRMncAfox2MDON9gkwxZ6kYTgwbWrczF7jaIeehFpYBNeO5hWE8kmWAGh94jJkylPxPCdNWHLB++nntlKwP+5pJ5lJa1lE1Cdqf5aLiuRxss6tSuVyyRSM2GZUjgy4nKCG9/F9DugT0Zn+76YYxGNESUbtLERO8lwY98r8e0lDrM9CX74+V12ss03uE0RNrFegnCdwHSCZwye64gL8ZMVBiA==
SIGN_WITH_HASH_RAW signature output = fFbfLMngLQU/k7/DT0Axm3DJpal2SThV6zC02ppIvg1wuA2rj2AYR9P0qGDg9FVXGioi0AB06QCSzJIfQB7mMaJBkCdqIlsCu9o1nxex8eJhTHtbm5Q1FqD/qQebOBdRMncAfox2MDON9gkwxZ6kYTgwbWrczF7jaIeehFpYBNeO5hWE8kmWAGh94jJkylPxPCdNWHLB++nntlKwP+5pJ5lJa1lE1Cdqf5aLiuRxss6tSuVyyRSM2GZUjgy4nKCG9/F9DugT0Zn+76YYxGNESUbtLERO8lwY98r8e0lDrM9CX74+V12ss03uE0RNrFegnCdwHSCZwye64gL8ZMVBiA==
==================================================

==================================================
Processing RSA algorithm cross comparison

Algorithm manual hash and NONEwithRSA against CryptographyClient.sign(SignatureAlgorithm.RS256)cross verify success = true
Algorithm manual hash and NONEwithRSA against CryptographyClient.signData(SignatureAlgorithm.RS256)cross verify success = true
Algorithm SHA256withRSA against CryptographyClient.sign(SignatureAlgorithm.RS256)cross verify success = true
Algorithm SHA256withRSA against CryptographyClient.signData(SignatureAlgorithm.RS256)cross verify success = true
CryptographyClient.verify(SignatureAlgorithm.RS256) against Algorithm manual hash and NONEwithRSA cross verify success = true
CryptographyClient.verify(SignatureAlgorithm.RS256) against Algorithm SHA256withRSA cross verify success = true
CryptographyClient.verifyData(SignatureAlgorithm.RS256) against Algorithm manual hash and NONEwithRSA cross verify success = true
CryptographyClient.verifyData(SignatureAlgorithm.RS256) against Algorithm SHA256withRSA cross verify success = true
==================================================

So as you can see, the signatures produced are cross verifiable since we know the difference for the expected input data.

Output for ECC(Keyvault SDK logs stripped)


==================================================
Processing ECC algorithm with Java Crypto implementation

Algorithm NONEwithECDSA self sign and verify success = true
Algorithm SHA256withECDSA self sign and verify success = true
Algorithm manual hash and NONEwithECDSA against SHA256withECDSA signature cross verify success = true
Algorithm SHA256withECDSA against manual hash and NONEwithECDSA signature cross verify success = true
--------------------------------------------------

SIGN_NO_HASH_RAW signature output = MEUCIEtUB+Jf/icrtKrDO+IljuW51a+wiASrwulQMvRDXrEkAiEAvpVaxxdzn9qo7+9+P/82utFAhHpSyiNKVAjBEZvhvJc=
SIGN_WITH_HASH_RAW signature output = MEYCIQDo47WpqxpqwtXns0qoH6tRFOdZ91PoTfqTFKAW5Kr51AIhANZnke9mZmgWIXSoA+vTBPzw/cr06JEEPV7xiOvjAyi5
SIGN_NO_HASH_DER_UNWRAP signature output = S1QH4l/+Jyu0qsM74iWO5bnVr7CIBKvC6VAy9ENesSS+lVrHF3Of2qjv734//za60UCEelLKI0pUCMERm+G8lw==
SIGN_WITH_HASH_DER_UNWRAP signature output = 6OO1qasaasLV57NKqB+rURTnWfdT6E36kxSgFuSq+dTWZ5HvZmZoFiF0qAPr0wT88P3K9OiRBD1e8Yjr4wMouQ==
==================================================

==================================================
Processing ECC algorithm with AKV Crypto implementation

CryptographyClient.sign(SignatureAlgorithm.ES256) self sign and verify success = true
CryptographyClient.signData(SignatureAlgorithm.ES256) self sign and verify success = true
CryptographyClient.verify(SignatureAlgorithm.ES256) against CryptographyClient.signData(SignatureAlgorithm.ES256) cross verify success = true
CryptographyClient.verifyData(SignatureAlgorithm.ES256) against CryptographyClient.sign(SignatureAlgorithm.ES256) cross verify success = true
--------------------------------------------------

SIGN_NO_HASH_RAW signature output = PHeVJwS4c7Pb3ZMs0fW9iJMAYgnygxDkmOq/LqgFzRruXjBVGJmCFFphVxHezFyj7H12goiI4oGiXoPkTdXw8g==
SIGN_WITH_HASH_RAW signature output = sTiBfyN/uK9dDjsRAlLUZebyfbwYGHMF2GEAfzUAi+v5D4WluJqJEApfs1MARuk0DCcAgEaHT84HPjUSPjfyMw==
SIGN_NO_HASH_DER_WRAP signature output = MEQCIDx3lScEuHOz292TLNH1vYiTAGIJ8oMQ5Jjqvy6oBc0aAiDuXjBVGJmCFFphVxHezFyj7H12goiI4oGiXoPkTdXw8g==
SIGN_WITH_HASH_DER_WRAP signature output = MEQCILE4gX8jf7ivXQ47EQJS1GXm8n28GBhzBdhhAH81AIvrAiD5D4WluJqJEApfs1MARuk0DCcAgEaHT84HPjUSPjfyMw==
==================================================

==================================================
Processing ECC algorithm cross comparison

Algorithm manual hash and NONEwithECDSA against CryptographyClient.sign(SignatureAlgorithm.ES256)cross verify success = true
Algorithm manual hash and NONEwithECDSA against CryptographyClient.signData(SignatureAlgorithm.ES256)cross verify success = true
Algorithm SHA256withECDSA against CryptographyClient.sign(SignatureAlgorithm.ES256)cross verify success = true
Algorithm SHA256withECDSA against CryptographyClient.signData(SignatureAlgorithm.ES256)cross verify success = true
CryptographyClient.verify(SignatureAlgorithm.ES256) against Algorithm manual hash and NONEwithECDSA cross verify success = true
CryptographyClient.verify(SignatureAlgorithm.ES256) against Algorithm SHA256withECDSA cross verify success = true
CryptographyClient.verifyData(SignatureAlgorithm.ES256) against Algorithm manual hash and NONEwithECDSA cross verify success = true
CryptographyClient.verifyData(SignatureAlgorithm.ES256) against Algorithm SHA256withECDSA cross verify success = true
==================================================

Going with the same approach we can see the signatures are cross verifiable. The missing bits that Keyvault wasnt doing is the produced signatures are not in a ASN1 (Or DER as I call them in the code) structure. Whereas the signatures produced by Java are in the correct structure already. To prove thats the secret sauce, I manually wrapped/unwrapped the ASN1 structures in derEncapSignatureForECC() and derDeencapSignatureForECC(). The results of that is cross verifiable.

So after going this far I am not even sure if this is a SDK bug at all, or has this evolved to a query or Keyvault feature request. In any case, I see some inconsistency between CryptographyClient.sign(SignatureAlgorithm.RS256) which returns output in an ASN1 struct vs CryptographyClient.sign(SignatureAlgorithm.ES256) which does not. What I would need is for CryptographyClient.sign() to produce output similar to whats returned with Java NONEwithECDSA.

yongxinglai-netrust commented 1 year ago

Hi @vcolin7,

Wondering if you had the time to take a look at the POC code above so far.