OS X 10.9, Clang 6.0 and RSA signature failures when using libcryptopp.dylib

[noloader]

It looks like RSA broke somewhere along the line on OS X 10.9 with Apple Clang 6.0 on a Core2 Duo MacBook (x86_64). The issue is present with libcryptopp.dylib but not with libcryptopp.a.

I went back to Issue 517,Add SHA3 OIDs for signature schemes (actually, 1 commit prior), which seems like the natural place to break it. The issue was present there, too.

Using the static archive is OK. Crypto++ 5.6.5 is OK. MacPort compilers on the same machine are OK. Other machines, like Linux and Solaris, are OK.

---

This may be related: Explicit specialization of member function template in source file. We had a lot of trouble trying to squash Clang warnings for padding decoration functions and their definitions. It was one of the first things we worked on after 5.6.5 was released.

---

Here's what it looks like running the test vector in question.

$ ./cryptest.exe tv rsa_pkcs1_1_5
Using seed: 1510528431

Testing Signature algorithm RSA/PKCS1-1.5(MD2).
.AlgorithmType: Signature
KeyFormat: DER
Message: 45 76 65 72 79 6f 6e 65 20 67 65 74 73 20 46 72 69 64 61 79 20 6f 66 66 2e
Name: RSA/PKCS1-1.5(MD2)
PrivateKey: 30 82 01 50 02 01 00 30 0d 06 09 2a 86 48 86 f7 0d 01 01 01 05 00 04 82 01 3a 30 82 01 36 02 01 00 02 40 0a 66 79 1d c6 98 81 68 de 7a b7 74 19 bb 7f b0 c0 01 c6 27 10 27 00 75 14 29 42 e1 9a 8d 8c 51 d0 53 b3 e3 78 2a 1d e5 dc 5a f4 eb e9 94 68 17 01 14 a1 df e6 7c dc 9a 9a f5 5d 65 56 20 bb ab 02 03 01 00 01 02 40 01 23 c5 b6 1b a3 6e db 1d 36 79 90 41 99 a8 9e a8 0c 09 b9 12 2e 14 00 c0 9a dc f7 78 46 76 d0 1d 23 35 6a 7d 44 d6 bd 8b d5 0e 94 bf c7 23 fa 87 d8 86 2b 75 17 76 91 c1 1d 75 76 92 df 88 81 02 20 33 d4 84 45 c8 59 e5 23 40 de 70 4b cd da 06 5f bb 40 58 d7 40 bd 1d 67 d2 9e 9c 14 6c 11 cf 61 02 20 33 5e 84 08 86 6b 0f d3 8d c7 00 2d 3f 97 2c 67 38 9a 65 d5 d8 30 65 66 d5 c4 f2 a5 aa 52 62 8b 02 20 04 5e c9 00 71 52 53 25 d3 d4 6d b7 96 95 e9 af ac c4 52 39 64 36 0e 02 b1 19 ba a3 66 31 62 41 02 20 15 eb 32 73 60 c7 b6 0d 12 e5 e2 d1 6b dc d9 79 81 d1 7f ba 6b 70 db 13 b2 0b 43 6e 24 ea da 59 02 20 2c a6 36 6d 72 78 1d fa 24 d3 4a 9a 24 cb c2 ae 92 7a 99 58 af 42 65 63 ff 63 fb 11 65 8a 46 1d
PublicKey: 30 5b 30 0d 06 09 2a 86 48 86 f7 0d 01 01 01 05 00 03 4a 00 30 47 02 40 0a 66 79 1d c6 98 81 68 de 7a b7 74 19 bb 7f b0 c0 01 c6 27 10 27 00 75 14 29 42 e1 9a 8d 8c 51 d0 53 b3 e3 78 2a 1d e5 dc 5a f4 eb e9 94 68 17 01 14 a1 df e6 7c dc 9a 9a f5 5d 65 56 20 bb ab 02 03 01 00 01
Signature: 05fa6a812fc7df8bf4f2542509e03e84 6e11b9c620be2009efb440efbcc66921 6994ac04f341b57d05202d428fb2a27b 5c77dfd9b15bfc3d559353503410c1e1
Source: http://www.rsasecurity.com/rsalabs/pkcs/index.html, Some Examples of the PKCS Standards
Test: Verify

Test failed.
Skipping to next test.

Testing Signature algorithm RSA/PKCS1-1.5(SHA-1).
AlgorithmType: Signature
KeyFormat: DER
Message: 45 76 65 72 79 6f 6e 65 20 67 65 74 73 20 46 72 69 64 61 79 20 6f 66 66 2e
Name: RSA/PKCS1-1.5(SHA-1)
PrivateKey: 30 82 01 50 02 01 00 30 0d 06 09 2a 86 48 86 f7 0d 01 01 01 05 00 04 82 01 3a 30 82 01 36 02 01 00 02 40 0a 66 79 1d c6 98 81 68 de 7a b7 74 19 bb 7f b0 c0 01 c6 27 10 27 00 75 14 29 42 e1 9a 8d 8c 51 d0 53 b3 e3 78 2a 1d e5 dc 5a f4 eb e9 94 68 17 01 14 a1 df e6 7c dc 9a 9a f5 5d 65 56 20 bb ab 02 03 01 00 01 02 40 01 23 c5 b6 1b a3 6e db 1d 36 79 90 41 99 a8 9e a8 0c 09 b9 12 2e 14 00 c0 9a dc f7 78 46 76 d0 1d 23 35 6a 7d 44 d6 bd 8b d5 0e 94 bf c7 23 fa 87 d8 86 2b 75 17 76 91 c1 1d 75 76 92 df 88 81 02 20 33 d4 84 45 c8 59 e5 23 40 de 70 4b cd da 06 5f bb 40 58 d7 40 bd 1d 67 d2 9e 9c 14 6c 11 cf 61 02 20 33 5e 84 08 86 6b 0f d3 8d c7 00 2d 3f 97 2c 67 38 9a 65 d5 d8 30 65 66 d5 c4 f2 a5 aa 52 62 8b 02 20 04 5e c9 00 71 52 53 25 d3 d4 6d b7 96 95 e9 af ac c4 52 39 64 36 0e 02 b1 19 ba a3 66 31 62 41 02 20 15 eb 32 73 60 c7 b6 0d 12 e5 e2 d1 6b dc d9 79 81 d1 7f ba 6b 70 db 13 b2 0b 43 6e 24 ea da 59 02 20 2c a6 36 6d 72 78 1d fa 24 d3 4a 9a 24 cb c2 ae 92 7a 99 58 af 42 65 63 ff 63 fb 11 65 8a 46 1d
PublicKey: 30 5b 30 0d 06 09 2a 86 48 86 f7 0d 01 01 01 05 00 03 4a 00 30 47 02 40 0a 66 79 1d c6 98 81 68 de 7a b7 74 19 bb 7f b0 c0 01 c6 27 10 27 00 75 14 29 42 e1 9a 8d 8c 51 d0 53 b3 e3 78 2a 1d e5 dc 5a f4 eb e9 94 68 17 01 14 a1 df e6 7c dc 9a 9a f5 5d 65 56 20 bb ab 02 03 01 00 01
Signature: 0610761F95FFD1B8F29DA34212947EC2AA0E358866A722F03CC3C41487ADC604A48FF54F5C6BEDB9FB7BD59F82D6E55D8F3174BA361B2214B2D74E8825E04E81
Source: generated by Wei Dai using Crypto++ 5.0
Test: Verify

Test failed.
Skipping to next test.
..AlgorithmType: Signature
KeyFormat: Component
Message: 74657374
ModPrime1PrivateExponent: 5D8EA4C8AF83A70634D5920C3DB66D908AC3AF57A597FD75BC9BBB856181C185
ModPrime2PrivateExponent: C598E54DAEC8ABC1E907769A6C2BD01653ED0C9960E1EDB7E186FDA922883A99
Modulus: A885B6F851A8079AB8A281DB0297148511EE0D8C07C0D4AE6D6FED461488E0D41E3FF8F281B06A3240B5007A5C2AB4FB6BE8AF88F119DB998368DDDC9710ABED
MultiplicativeInverseOfPrime2ModPrime1: 7C6F27B5B51B78AD80FB36E700990CF307866F2943124CBD93D97C137794C104
Name: RSA/PKCS1-1.5(SHA-1)
Prime1: D7103CD676E39824E2BE50B8E6533FE7CB7484348E283802AD2B8D00C80D19DF
Prime2: C89996DC169CEB3F227958275968804D4BE9FC4012C3219662F1A438C9950BB3
PrivateExponent: 2B259D2CA3DF851EE891F6F4678BDDFD9A131C95D3305C63D2723B4A5B9C960F5EC8BB7DCDDBEBD8B6A38767D64AD451E9383E0891E4EE7506100481F2B49323
PublicExponent: 010001
Signature: A7E00CE4391F914D82158D9B732759808E25A1C6383FE87A5199157650D4296CF612E9FF809E686A0AF328238306E79965F6D0138138829D9A1A22764306F6CE
Source: generated by Wei Dai using Crypto++ 5.0
Test: Verify

Test failed.
Skipping to next test.

Tests complete. Total tests = 6. Failed tests = 3.

[noloader]

Adding CPPFLAGS=-DCRYPTOPP_IS_DLL cleared the issue. It is definitely related to template instantiations.

[noloader]

Fixed this issue at Commit 4232cfd40b69 (the fix) and Commit c0a8dab984d1 (missing header file).