Closed edburns closed 3 years ago
Hi Ed @edburns,
I tested using the keystore commands provided in the issue. Though the deployment went through, the SSL setup is not working as expected due to wrong trust.jks file being used.
Below are the correct keystore commands that needs to be used to create the identity.jks and trust.jks files.
Create identity.jks File: keytool -genkey -alias servercert -keyalg RSA -keysize 2048 -sigalg SHA256withRSA -validity 365 -keystore identity.jks -keypass Gumby12340987 -storepass Gumby12340987
Export identity.jks File and generate root cert file: keytool -export -alias servercert -noprompt -file root.cert -keystore identity.jks -storepass Gumby12340987
Import root.cert and create trust.jks file keytool -import -alias servercert -noprompt -file root.cert -keystore trust.jks -storepass Gumby12340987
The identity and trust keystore files are related to each other and we cannot use independently created keystores and use it as identity and trust keystores.
I am working on including few validations in the script to check if the provided files is a valid keystore file. I will also try to find out if there is a way to validate, if the identity and trust keystores provided are related.
The following commands can be used to validate the identity & trust.jks keystore files:
Validate Identity Keystore
keytool -list -v -keystore identity.jks -storepass Gumby12340987 -storetype JKS |grep "Entry type:"|grep 'PrivateKeyEntry'
Returns 0 on successful validation and 1 on validation failure
Validate Trust Keystore
keytool -list -v -keystore trust.jks -storepass Gumby12340987 -storetype JKS |grep "Entry type:"|grep 'trustedCertEntry'
Returns 0 on successful validation and 1 on validation failure
These validations will be added to the SSL Configuration logic in Domain Setup scripts for all the three offers.
Implemented validation for SSL Keystores during deployment and add node features.
Created pull requests for all the three offers.
https://github.com/wls-eng/arm-oraclelinux-wls-admin/pull/96
https://github.com/wls-eng/arm-oraclelinux-wls-cluster/pull/136
https://github.com/wls-eng/arm-oraclelinux-wls-dynamic-cluster/pull/123
Hi @edburns,
Thanks for identifying the issue. The issue occurred due to a typo issue in configuring SSL parameters in nodemanager.properties for the Managed Servers.
The reason I couldn't identity this issue during my testing, is because, I was using the same passphrase for all the required passphrases.
I have made the fix, tested and found them working fine for Cluster & Dynamic Cluster offers. I have created the following Pull Requests for merging the fix.
https://github.com/wls-eng/arm-oraclelinux-wls-cluster/pull/139 https://github.com/wls-eng/arm-oraclelinux-wls-dynamic-cluster/pull/125
Can access the console over http going directly to DNS name of admin server.
Observe the cluster did form.
Observe managed servers are all running.
Observe certificate is as expected.
Deploy weblogic-kubernetes-operator/kubernetes/samples/charts/application/testwebapp.war
hit ohsVM directly.
$ curl --verbose http://wls-5d2ee46dcb-ohsstandalonedomain.eastus.cloudapp.azure.com:7777/testwebapp/index.jsp
* Trying 52.249.186.110...
* TCP_NODELAY set
* Connected to wls-5d2ee46dcb-ohsstandalonedomain.eastus.cloudapp.azure.com (52.249.186.110) port 7777 (#0)
> GET /testwebapp/index.jsp HTTP/1.1
> Host: wls-5d2ee46dcb-ohsstandalonedomain.eastus.cloudapp.azure.com:7777
> User-Agent: curl/7.58.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Wed, 10 Mar 2021 02:52:03 GMT
< X-Content-Type-Options: nosniff
< Content-Length: 436
< X-ORACLE-DMS-ECID: 005jIN0Rx5fFw005zzh8iW0000wR00006E
< X-ORACLE-DMS-RID: 0:1
< Set-Cookie: JSESSIONID=mEoaDGZgxLU8GYkp9-dbtzweYc49DJyYOcs9zD8yDI4itk6TEPQ-!-434689887; path=/; HttpOnly
< X-XSS-Protection: 1; mode=block
< Content-Type: text/html; charset=UTF-8
<
<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<link rel="stylesheet" href="/testwebapp/res/styles.css;jsessionid=mEoaDGZgxLU8GYkp9-dbtzweYc49DJyYOcs9zD8yDI4itk6TEPQ-!-434689887" type="text/css">
<title>Test WebApp</title>
</head>
<body>
<li>InetAddress: mspVM2/10.0.0.4
<li>InetAddress.hostname: mspVM2
</body>
</html>
* Connection #0 to host wls-5d2ee46dcb-ohsstandalonedomain.eastus.cloudapp.azure.com left intact
$ curl --verbose --insecure https://wls-5d2ee46dcb-ohsstandalonedomain.eastus.cloudapp.azure.com:4444/testwebapp/index.jsp
* Trying 52.249.186.110...
* TCP_NODELAY set
* Connected to wls-5d2ee46dcb-ohsstandalonedomain.eastus.cloudapp.azure.com (52.249.186.110) port 4444 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: C=US; ST=FL; L=Orlando; O=Microsoft; OU=Azure; CN=Ed Burns
* start date: Mar 8 20:39:00 2021 GMT
* expire date: Mar 3 20:39:00 2022 GMT
* issuer: C=US; ST=FL; L=Orlando; O=Microsoft; OU=Azure; CN=Ed Burns
* SSL certificate verify result: self signed certificate (18), continuing anyway.
> GET /testwebapp/index.jsp HTTP/1.1
> Host: wls-5d2ee46dcb-ohsstandalonedomain.eastus.cloudapp.azure.com:4444
> User-Agent: curl/7.58.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Wed, 10 Mar 2021 02:52:48 GMT
< X-Content-Type-Options: nosniff
< Content-Length: 436
< X-ORACLE-DMS-ECID: 005jIN38df^Fw005zzh8iW0000wR00006H
< X-ORACLE-DMS-RID: 0:1
< Set-Cookie: JSESSIONID=j7caDRc1NmkioUuzoNTHv7nmgyzlNjfdbLea4DHGEFCnthED0kSb!-434689887; path=/; HttpOnly
< X-XSS-Protection: 1; mode=block
< Content-Type: text/html; charset=UTF-8
<
<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<link rel="stylesheet" href="/testwebapp/res/styles.css;jsessionid=j7caDRc1NmkioUuzoNTHv7nmgyzlNjfdbLea4DHGEFCnthED0kSb!-434689887" type="text/css">
<title>Test WebApp</title>
</head>
<body>
<li>InetAddress: mspVM2/10.0.0.4
<li>InetAddress.hostname: mspVM2
</body>
</html>
* Connection #0 to host wls-5d2ee46dcb-ohsstandalonedomain.eastus.cloudapp.azure.com left intact
Verify certificate is as expected.
DNS
Perform cheapo DNS delegation. Modify WSL2 resolv.conf so that
ns1-02.azure-dns.com
is included in the list of nameserver
entries.
https://superuser.com/questions/1533291/how-do-i-change-the-dns-settings-for-wsl2
New /etc/resolv.conf
nameserver 172.23.240.1
nameserver 40.90.4.2
curl --verbose --head http://www.ejb030901d.com:7777/testwebapp/index.jsp
$ curl --verbose --head http://www.ejb030901d.com:7777/testwebapp/index.jsp
* Trying 52.249.186.110...
* TCP_NODELAY set
* Connected to www.ejb030901d.com (52.249.186.110) port 7777 (#0)
> HEAD /testwebapp/index.jsp HTTP/1.1
> Host: www.ejb030901d.com:7777
> User-Agent: curl/7.58.0
> Accept: */*
>
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Date: Wed, 10 Mar 2021 02:54:25 GMT
Date: Wed, 10 Mar 2021 02:54:25 GMT
< X-Content-Type-Options: nosniff
X-Content-Type-Options: nosniff
< X-ORACLE-DMS-ECID: 005jIN8tyGeFw005zzh8iW0000wR00006L
X-ORACLE-DMS-ECID: 005jIN8tyGeFw005zzh8iW0000wR00006L
< X-ORACLE-DMS-RID: 0:1
X-ORACLE-DMS-RID: 0:1
< Set-Cookie: JSESSIONID=WYgaDpFkEEvs8C210ibnsRMxI1hE8wl_YPMu4zBz6k1cBk9aLq8w!-185493146; path=/; HttpOnly
Set-Cookie: JSESSIONID=WYgaDpFkEEvs8C210ibnsRMxI1hE8wl_YPMu4zBz6k1cBk9aLq8w!-185493146; path=/; HttpOnly
< X-XSS-Protection: 1; mode=block
X-XSS-Protection: 1; mode=block
< Content-Type: text/html; charset=UTF-8
Content-Type: text/html; charset=UTF-8
<
* Connection #0 to host www.ejb030901d.com left intact
$ curl --verbose --head --insecure https://www.ejb030901d.com:4444/testwebapp/index.jsp
* Trying 52.249.186.110...
* TCP_NODELAY set
* Connected to www.ejb030901d.com (52.249.186.110) port 4444 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: C=US; ST=FL; L=Orlando; O=Microsoft; OU=Azure; CN=Ed Burns
* start date: Mar 8 20:39:00 2021 GMT
* expire date: Mar 3 20:39:00 2022 GMT
* issuer: C=US; ST=FL; L=Orlando; O=Microsoft; OU=Azure; CN=Ed Burns
* SSL certificate verify result: self signed certificate (18), continuing anyway.
> HEAD /testwebapp/index.jsp HTTP/1.1
> Host: www.ejb030901d.com:4444
> User-Agent: curl/7.58.0
> Accept: */*
>
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Date: Wed, 10 Mar 2021 02:55:07 GMT
Date: Wed, 10 Mar 2021 02:55:07 GMT
< X-Content-Type-Options: nosniff
X-Content-Type-Options: nosniff
< X-ORACLE-DMS-ECID: 005jINBO3cdFw005zzh8iW0000wR00006O
X-ORACLE-DMS-ECID: 005jINBO3cdFw005zzh8iW0000wR00006O
< X-ORACLE-DMS-RID: 0:1
X-ORACLE-DMS-RID: 0:1
< Set-Cookie: JSESSIONID=pJQaDzNGXHmP2AbLufgSymPdAIIzNSEaG4bthQZQzXfd6dXXcMfA!-185493146; path=/; HttpOnly
Set-Cookie: JSESSIONID=pJQaDzNGXHmP2AbLufgSymPdAIIzNSEaG4bthQZQzXfd6dXXcMfA!-185493146; path=/; HttpOnly
< X-XSS-Protection: 1; mode=block
X-XSS-Protection: 1; mode=block
< Content-Type: text/html; charset=UTF-8
Content-Type: text/html; charset=UTF-8
<
* Connection #0 to host www.ejb030901d.com left intact
* Server certificate:
* subject: C=US; ST=FL; L=Orlando; O=Microsoft; OU=Azure; CN=Ed Burns
* start date: Mar 8 20:39:00 2021 GMT
* expire date: Mar 3 20:39:00 2022 GMT
* issuer: C=US; ST=FL; L=Orlando; O=Microsoft; OU=Azure; CN=Ed Burns
* SSL certificate verify result: self signed certificate (18), continuing anyway.
curl --verbose --head http://admin.ejb030901d.com:7001/console
* Trying 40.76.48.36...
* TCP_NODELAY set
* Connected to admin.ejb030901d.com (40.76.48.36) port 7001 (#0)
> HEAD /console HTTP/1.1
> Host: admin.ejb030901d.com:7001
> User-Agent: curl/7.58.0
> Accept: */*
>
< HTTP/1.1 302 Moved Temporarily
HTTP/1.1 302 Moved Temporarily
< Date: Wed, 10 Mar 2021 02:56:23 GMT
Date: Wed, 10 Mar 2021 02:56:23 GMT
< Location: http://admin.ejb030901d.com:7001/console/
Location: http://admin.ejb030901d.com:7001/console/
< Content-Type: text/html
Content-Type: text/html
< X-ORACLE-DMS-ECID: fa6b968b-aac7-4261-9bd7-0cb888a0b9fc-0000013f
X-ORACLE-DMS-ECID: fa6b968b-aac7-4261-9bd7-0cb888a0b9fc-0000013f
< X-ORACLE-DMS-RID: 0
X-ORACLE-DMS-RID: 0
<
* Connection #0 to host admin.ejb030901d.com left intact
$ curl --verbose --head --insecure https://admin.ejb030901d.com:7002/console
* Trying 40.76.48.36...
* TCP_NODELAY set
* Connected to admin.ejb030901d.com (40.76.48.36) port 7002 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (OUT), TLS change cipher, Client hello (1):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Unknown (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS Unknown, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
* subject: C=US; ST=MyState; L=MyTown; O=MyOrganization; OU=FOR TESTING ONLY; CN=adminVM
* start date: Mar 7 19:37:02 2021 GMT
* expire date: Mar 8 19:37:02 2036 GMT
* issuer: C=US; ST=MyState; L=MyTown; O=MyOrganization; OU=FOR TESTING ONLY; CN=CertGenCA
* SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
* Using Stream ID: 1 (easy handle 0x5615091cb580)
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
> HEAD /console HTTP/2
> Host: admin.ejb030901d.com:7002
> User-Agent: curl/7.58.0
> Accept: */*
>
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
< HTTP/2 302
HTTP/2 302
< date: Wed, 10 Mar 2021 02:56:57 GMT
date: Wed, 10 Mar 2021 02:56:57 GMT
< location: https://admin.ejb030901d.com:7002/console/
location: https://admin.ejb030901d.com:7002/console/
< content-type: text/html
content-type: text/html
< x-oracle-dms-ecid: fa6b968b-aac7-4261-9bd7-0cb888a0b9fc-00000140
x-oracle-dms-ecid: fa6b968b-aac7-4261-9bd7-0cb888a0b9fc-00000140
< x-oracle-dms-rid: 0
x-oracle-dms-rid: 0
<
* Connection #0 to host admin.ejb030901d.com left intact
* Server certificate:
* subject: C=US; ST=MyState; L=MyTown; O=MyOrganization; OU=FOR TESTING ONLY; CN=adminVM
* start date: Mar 7 19:37:02 2021 GMT
* expire date: Mar 8 19:37:02 2036 GMT
* issuer: C=US; ST=MyState; L=MyTown; O=MyOrganization; OU=FOR TESTING ONLY; CN=CertGenCA
* SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
Using Dynamic Cluster offer version 1.0.32 2021-03-05 13:52 EST
https://ms.portal.azure.com/#create/oracle.20210202-test-01-previewarm-oraclelinux-wls-dynamic-cluster
With the following options filled out
Basics
TLS/SSL
OHS
DNS -> Create+Review
Deploying the cluster in this way causes the managed servers to not run.