Keystore errors on GKE

[GMafra]

Hi,

I'm following the steps to deploy nexus with the proxy on GKE and it is failing due to issues with the Java Key store:

java.lang.RuntimeException: java.io.IOException: Keystore was tampered with, or password was incorrect at io.vertx.ext.auth.jwt.impl.JWTAuthProviderImpl.<init>(JWTAuthProviderImpl.java:78) at io.vertx.ext.auth.jwt.JWTAuth.create(JWTAuth.java:41) at com.travelaudience.nexus.proxy.JwtAuth.<init>(JwtAuth.java:29) at com.travelaudience.nexus.proxy.JwtAuth.create(JwtAuth.java:53) at com.travelaudience.nexus.proxy.CloudIamAuthNexusProxyVerticle.init(CloudIamAuthNexusProxyVerticle.java:106) at io.vertx.core.impl.DeploymentManager.lambda$doDeploy$8(DeploymentManager.java:432) at io.vertx.core.impl.ContextImpl.lambda$wrapTask$2(ContextImpl.java:337) at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:445) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Keystore was tampered with, or password was incorrect at com.sun.crypto.provider.JceKeyStore.engineLoad(JceKeyStore.java:879) at java.security.KeyStore.load(KeyStore.java:1445) at io.vertx.ext.auth.jwt.impl.JWTAuthProviderImpl.<init>(JWTAuthProviderImpl.java:66)

Expand all | Collapse all {
 insertId:  "ibljtdg4agry3a"  
 labels: {…}  
 logName:  "projects/kubernetes-dmafra/logs/nexus-proxy"  
 receiveTimestamp:  "2019-04-09T19:02:02.356485455Z"  
 resource: {…}  
 severity:  "INFO"  
 textPayload:  "java.lang.RuntimeException: java.io.IOException: Keystore was tampered with, or password was incorrect
    at io.vertx.ext.auth.jwt.impl.JWTAuthProviderImpl.<init>(JWTAuthProviderImpl.java:78)
    at io.vertx.ext.auth.jwt.JWTAuth.create(JWTAuth.java:41)
    at com.travelaudience.nexus.proxy.JwtAuth.<init>(JwtAuth.java:29)
    at com.travelaudience.nexus.proxy.JwtAuth.create(JwtAuth.java:53)
    at com.travelaudience.nexus.proxy.CloudIamAuthNexusProxyVerticle.init(CloudIamAuthNexusProxyVerticle.java:106)
    at io.vertx.core.impl.DeploymentManager.lambda$doDeploy$8(DeploymentManager.java:432)
    at io.vertx.core.impl.ContextImpl.lambda$wrapTask$2(ContextImpl.java:337)
    at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:445)
    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Keystore was tampered with, or password was incorrect
    at com.sun.crypto.provider.JceKeyStore.engineLoad(JceKeyStore.java:879)
    at java.security.KeyStore.load(KeyStore.java:1445)
    at io.vertx.ext.auth.jwt.impl.JWTAuthProviderImpl.<init>(JWTAuthProviderImpl.java:66)
"  
 timestamp:  "2019-04-09T19:01:15.668490300Z"  
}

and also some other errors such as "java.security.UnrecoverableKeyException: Password verification failed" or "Invalid keystore format at io.vertx.ext.auth.jwt.impl.JWTAuthProviderImpl."

I created the keystore as explained on the how to with the following code

keytool -genkey \
          -keystore keystore.jceks \
          -storetype jceks \
          -keyalg RSA \
          -keysize 2048 \
          -alias RS256 \
          -sigalg SHA256withRSA \
          -dname "CN=,OU=,O=,L=,ST=,C=" \
          -validity 3651 \

and was able to validate it with keytool -list -v -keystore keystore.jceks -storetype jceks

Not sure how to proceed here and get it working.

[pires]

Possibly the password is not properly defined or the file uploaded to the secret is not valid, as defined here.

Did you replace the corresponding values accordingly when running:

$ cat << EOF > nexus-proxy-ks-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: nexus-proxy-ks
type: Opaque
data:
  keystore: $(cat /path/to/keystore.jceks | base64)
  password: $(echo -n "KEYSTORE_PASSWORD" | base64)
EOF

?

Do re-read the section above, because that's most probably where your problem is.

[GMafra]

Hi Pires,

Yes, the keystore I pointed to where it was located and the password I typed in there and it encoded successfully (I tried decoding it and got the correct value). Is password complexity a thing?

The file generated by the code you posted is:

apiVersion: v1
kind: Secret
metadata:
  name: nexus-proxy-ks
type: Opaque
data:
  keystore: zs7OzgAAAAIAAAABAAAAAQAFcnMyNTYAAAFqA2zfowAABO4wggTqMBwGCSsGAQQBKgITATAPBAhIX52BBrIm4wIDAw1ABIIEyAPIMsuRhF2if+Ukm+dWISvEo7u82r7vpM9w6tEBLQUrxjrPMKceuAlnL0EfabfuS0d0XKoIDU3+m4RQOcN1xeLGdP1diuSxeITSabeTAdZFLeNlPqmJmnfHIAPgbxgW3D1xbNR4m7nQAokvGFxMmnC5KO7pbggk7UjCrQQZONrx5rbjRzGEiM8XLEVbl6+ffK22pn+Flhr6F5FdLN8Bd2VEobRomcBTfsuVns6BDEm1aMQf6RVCYy8pjUaue0z6ZMbAEkBfMAnk9A+lnI9Ou5G6xH9MbLKonCpJK9WhlqvUQ0ls3oGzdrIxVnqf7r2listk8EXterh0INj0eitxV6/9Mv6HXbutIVJXFAH2hxyDRi1tW92uRXsTyiG/OqJXCnWIA3JCnn7lIm9SZNaPDfq+OjfNkCozxvMm0Un6imdTEgJj+GbCQOcW64YR6FmdXtFmQk8IYldgxioD/q4pd7bG0sRJ3p4F9cf8lzHX9FykgCCENwsg0K2D15T0bFtIST7ExXA7bcZlo7GbOAVDu9zpeVEyIIAYsLTUYsdrhZYkQkoQX1PRMg0rl97SB1DmKd2liDMp0JYPB80Go70xtdtbgmhxx7kany3tC6t0RU87kx01WlqKzxtch4k0k0snDV2BOueIfFyX0N3iUFD7KX0xEQOV5FdtjCdlQ6dJqiPuQMulqmcbF54OJyZc8HTYW5m2gmGH17rDxtYuPlYSBXRwnGW+iXHqvQOn1lNLMCH9RB02qYYUsBg200h5TMcwIglJ5wQVPpXw7u4T4o63MYlsnLDp6sxOIoUNL4fXP+LHX8XoIDDyOPgbJCvGwDFUoGL+I4guXCZtQwehHSIsPyUDqxegOC1CMN6CVUEYvU6Qsz7rN2Ucb3bipXsF5FBGj54Yj9537vWAnqUbH2AkMggiBrBHejG1P8UmKTNHhgPr35R2uj3jfqBz8XJ5BzMgqBHQfKatgiGf5ZoQ7MeBIbVUpw7PgLfC8rNdCGEYusfCZ4SfZKe2W5+rCIpusQAvTR3rWXqCTpxgefcX9cXNPJIaVMEUxknJjZkmcasv+aLFAlUXMU6kwveJQLwY9AmDMkdw4WGA4iEbqHnjkf41rRUSaMNzY1oPSy8KvcfoQlSdTXD/uYWCch4/orPYB7SVMfObMjFDHyI2KFc4zyI2zh2Nr/mczvdG3QogZr8wHYdH6yBidScVAqrCMRi7FKE60FbkQvARYAhVxsf4JmTzjBZAQiLncy+X/Sh6Peb/nK16cuPWkLD4epYvx+HutEs/CXi8+A23bOF6Il1l5Giw9fmuufrQlQekrGfyiS0EpIfyYYXS2sFRUnZAAxfrIosJRkFcEjIrXca/xy5OJ6FenuwWSXdeI8xYvmKtHIGZjvA7fnClxaeqc5POhAhnRao9bpHCSt83cKSesB2FHichwN95GKMZOL6CSX/hRGMdq79qyEiLfqCfhKBFh1yRntObv5P+xW05oO4fTI2U9OVUZDIw673jcJDOfiPqFElRwZHSt1cBAGvuACGwfmuT+qfmMYFseNMPdPO0mxlGiViCwZyVxR11ohccWFzC60BSHjpWeUDyI/LCKY8VMcgpeybp8W/7Fez3VFMQlZFiMA1hflUmEdLD/jhtpAAAAAEABVguNTA5AAADJzCCAyMwggILoAMCAQICBC7tChowDQYJKoZIhvcNAQELBQAwQjEJMAcGA1UEBhMAMQkwBwYDVQQIEwAxCTAHBgNVBAcTADEJMAcGA1UEChMAMQkwBwYDVQQLEwAxCTAHBgNVBAMTADAeFw0xOTA0MDkxODQ3MDNaFw0yOTA0MDcxODQ3MDNaMEIxCTAHBgNVBAYTADEJMAcGA1UECBMAMQkwBwYDVQQHEwAxCTAHBgNVBAoTADEJMAcGA1UECxMAMQkwBwYDVQQDEwAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCA305Hhxm4GZg2XbAYk9eYXOe5BpMA9PeJIVPycweBxdDbjzYbyGFgf9B96VZaOH8qH+jNrodFfnGkCph3RRwOevM2iY0131vk/B13b08NiwMkH+iKWr4oQTytnOcjQR8LJYYaHoUofHfMvkwy3iW2uMQEJrW0h6vkdtoHHLHIcn3DBAikYDWWPXCvEulio5EfFwaIlPI9u2J+ZkIegq68zSMtjALs6rTWc/bVAScPGL4EB93dFrOjaDQbhmgXJnwaNPwODng2VpYq8g6Q5XrtRslvPzFoylaZcm7HUutvExtPN9ZesVKCr49pLVU9NxmDSSynsTuCTyGYj1WvQGo9AgMBAAGjITAfMB0GA1UdDgQWBBQEmnLY5AVdCGzTCCwKS4uXDM3bnjANBgkqhkiG9w0BAQsFAAOCAQEAMUgOIUcmhyX9vaezBG6OYa/b15Gd9eV4HXa/2Ln1nJeHecACF39rRAqFNNeh9AlBgcDW8GF/wfroyQq7HCwmVBcPRMC7LYJxSYSC4H6V8YwsBpdQ57KKXX67NLex13UGyEWvkE69HrWaD4iRZ7h24bNVTzYB+ccAWGyCY/gkMEee2WeHbsv2mDqsR0ggngO8Olz+YZmtheSHEnDK5qTGU1UbzKwIUcz3lQ32dKziERyVLhUdBPV83mDlfU2Mh//9LyHZBHHrBi27WPxd8OI8CHq++fs8SVOcQRBv+RmQkXet8jlKvkXfs/Ql+7xcvh1UkBK+AHNCxQ+JLbzQ4jKf2ykrr7Y2qPjHB0+OiWDEf5uuYRHW
  password: 4oCcMTIzNDU24oCd

It looks good to me. Also, this is the content of my Keystore:

Keystore type: JCEKS
Keystore provider: SunJCE

Your keystore contains 1 entry

Alias name: rs256
Creation date: Apr 9, 2019
Entry type: PrivateKeyEntry
Certificate chain length: 1
Certificate[1]:
Owner: CN=, OU=, O=, L=, ST=, C=
Issuer: CN=, OU=, O=, L=, ST=, C=
Serial number: 2eed0a1a
Valid from: Tue Apr 09 15:47:03 BRT 2019 until: Sat Apr 07 15:47:03 BRT 2029
Certificate fingerprints:
     MD5:  39:C9:0F:88:BF:BE:0F:8C:A5:09:4B:74:29:C7:BE:98
     SHA1: D3:EA:47:46:78:E3:96:A7:4E:E9:42:1B:3F:FA:E1:87:07:03:4C:E2
     SHA256: 65:12:D8:BE:F8:DC:F8:DC:B3:D8:43:9D:97:D6:65:5C:84:FC:59:80:FE:4E:38:44:72:5F:70:78:15:94:DC:CD
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 2048-bit RSA key
Version: 3

Extensions:

#1: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: 04 9A 72 D8 E4 05 5D 08   6C D3 08 2C 0A 4B 8B 97  ..r...].l..,.K..
0010: 0C CD DB 9E                                        ....
]
]

*******************************************
*******************************************

[pires]

I'm out of ideas at this point :(

[pires]

Can you please start a debug pod with this secret mounted and run the validation manually?

[GMafra]

Sure, I will do that and let you know the results

[GMafra]

Hi Pires,

This is the result:

Password looks good as well:

[GMafra]

Found the issue!

On the documentation it is said to replace KEYSTORE_PASSWORD with the password and encode it. The problem is that it is written between "" on the script, which makes it encode the value with "" as well!

So after doing this: password: $(echo -n Start123! | base64) Instead of this: password: $(echo -n "Start123!" | base64)

It worked!

[pires]

Ah! I'll fix it, thanks.