edburns commented 3 years ago

Using Dynamic Cluster offer version 1.0.32 2021-03-05 13:52 EST

https://ms.portal.azure.com/#create/oracle.20210202-test-01-previewarm-oraclelinux-wls-dynamic-cluster

With the following options filled out

Basics

Nothing special

TLS/SSL

keytool -genkey -keyalg RSA -alias servercert -keystore keyStoreIdentity.jks -storepass Gumby12340987 -validity 360 -keysize 2048 -keypass Gumby12340987 -storetype jks

keytool -genkey -keyalg RSA -alias servercert -keystore keyStoreTrust.jks -storepass Gumby12340987 -validity 360 -keysize 2048 -keypass Gumby12340987 -storetype jks

OHS

Upload existing key stores

keytool -genkey -keyalg RSA -alias servercert -keystore keyStoreOhs.jks -storepass Gumby12340987 -validity 360 -keysize 2048 -keypass Gumby12340987 -storetype jks

DNS -> Create+Review

No

Deploying the cluster in this way causes the managed servers to not run.

gnsuryan commented 3 years ago

Hi Ed @edburns,

I tested using the keystore commands provided in the issue. Though the deployment went through, the SSL setup is not working as expected due to wrong trust.jks file being used.

Below are the correct keystore commands that needs to be used to create the identity.jks and trust.jks files.

Create identity.jks File: keytool -genkey -alias servercert -keyalg RSA -keysize 2048 -sigalg SHA256withRSA -validity 365 -keystore identity.jks -keypass Gumby12340987 -storepass Gumby12340987

Export identity.jks File and generate root cert file: keytool -export -alias servercert -noprompt -file root.cert -keystore identity.jks -storepass Gumby12340987

Import root.cert and create trust.jks file keytool -import -alias servercert -noprompt -file root.cert -keystore trust.jks -storepass Gumby12340987

gnsuryan commented 3 years ago

The identity and trust keystore files are related to each other and we cannot use independently created keystores and use it as identity and trust keystores.

I am working on including few validations in the script to check if the provided files is a valid keystore file. I will also try to find out if there is a way to validate, if the identity and trust keystores provided are related.

gnsuryan commented 3 years ago

The following commands can be used to validate the identity & trust.jks keystore files:

Validate Identity Keystore keytool -list -v -keystore identity.jks -storepass Gumby12340987 -storetype JKS |grep "Entry type:"|grep 'PrivateKeyEntry' Returns 0 on successful validation and 1 on validation failure

Validate Trust Keystore keytool -list -v -keystore trust.jks -storepass Gumby12340987 -storetype JKS |grep "Entry type:"|grep 'trustedCertEntry' Returns 0 on successful validation and 1 on validation failure

These validations will be added to the SSL Configuration logic in Domain Setup scripts for all the three offers.

gnsuryan commented 3 years ago

Implemented validation for SSL Keystores during deployment and add node features.

Created pull requests for all the three offers.

https://github.com/wls-eng/arm-oraclelinux-wls-admin/pull/96

https://github.com/wls-eng/arm-oraclelinux-wls-cluster/pull/136

https://github.com/wls-eng/arm-oraclelinux-wls-dynamic-cluster/pull/123

gnsuryan commented 3 years ago

Hi @edburns,

Thanks for identifying the issue. The issue occurred due to a typo issue in configuring SSL parameters in nodemanager.properties for the Managed Servers.

The reason I couldn't identity this issue during my testing, is because, I was using the same passphrase for all the required passphrases.

I have made the fix, tested and found them working fine for Cluster & Dynamic Cluster offers. I have created the following Pull Requests for merging the fix.

https://github.com/wls-eng/arm-oraclelinux-wls-cluster/pull/139 https://github.com/wls-eng/arm-oraclelinux-wls-dynamic-cluster/pull/125

edburns commented 3 years ago

Can access the console over http going directly to DNS name of admin server.
Observe the cluster did form.
Observe managed servers are all running.
Observe certificate is as expected.

Deploy weblogic-kubernetes-operator/kubernetes/samples/charts/application/testwebapp.war

hit ohsVM directly.

curl --verbose http://wls-5d2ee46dcb-ohsstandalonedomain.eastus.cloudapp.azure.com:7777/testwebapp/index.jsp

$ curl --verbose http://wls-5d2ee46dcb-ohsstandalonedomain.eastus.cloudapp.azure.com:7777/testwebapp/index.jsp
*   Trying 52.249.186.110...
* TCP_NODELAY set
* Connected to wls-5d2ee46dcb-ohsstandalonedomain.eastus.cloudapp.azure.com (52.249.186.110) port 7777 (#0)
> GET /testwebapp/index.jsp HTTP/1.1
> Host: wls-5d2ee46dcb-ohsstandalonedomain.eastus.cloudapp.azure.com:7777
> User-Agent: curl/7.58.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Date: Wed, 10 Mar 2021 02:52:03 GMT
< X-Content-Type-Options: nosniff
< Content-Length: 436
< X-ORACLE-DMS-ECID: 005jIN0Rx5fFw005zzh8iW0000wR00006E
< X-ORACLE-DMS-RID: 0:1
< Set-Cookie: JSESSIONID=mEoaDGZgxLU8GYkp9-dbtzweYc49DJyYOcs9zD8yDI4itk6TEPQ-!-434689887; path=/; HttpOnly
< X-XSS-Protection: 1; mode=block
< Content-Type: text/html; charset=UTF-8
< 

<!DOCTYPE html>
<html>
<head>
  <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">

  <link rel="stylesheet" href="/testwebapp/res/styles.css;jsessionid=mEoaDGZgxLU8GYkp9-dbtzweYc49DJyYOcs9zD8yDI4itk6TEPQ-!-434689887" type="text/css"> 
  <title>Test WebApp</title>
</head>
<body>

  <li>InetAddress: mspVM2/10.0.0.4
  <li>InetAddress.hostname: mspVM2

</body>
</html>
* Connection #0 to host wls-5d2ee46dcb-ohsstandalonedomain.eastus.cloudapp.azure.com left intact

curl --verbose --insecure https://wls-5d2ee46dcb-ohsstandalonedomain.eastus.cloudapp.azure.com:4444/testwebapp/index.jsp

$ curl --verbose --insecure https://wls-5d2ee46dcb-ohsstandalonedomain.eastus.cloudapp.azure.com:4444/testwebapp/index.jsp
*   Trying 52.249.186.110...
* TCP_NODELAY set
* Connected to wls-5d2ee46dcb-ohsstandalonedomain.eastus.cloudapp.azure.com (52.249.186.110) port 4444 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: C=US; ST=FL; L=Orlando; O=Microsoft; OU=Azure; CN=Ed Burns
*  start date: Mar  8 20:39:00 2021 GMT
*  expire date: Mar  3 20:39:00 2022 GMT
*  issuer: C=US; ST=FL; L=Orlando; O=Microsoft; OU=Azure; CN=Ed Burns
*  SSL certificate verify result: self signed certificate (18), continuing anyway.
> GET /testwebapp/index.jsp HTTP/1.1
> Host: wls-5d2ee46dcb-ohsstandalonedomain.eastus.cloudapp.azure.com:4444
> User-Agent: curl/7.58.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Date: Wed, 10 Mar 2021 02:52:48 GMT
< X-Content-Type-Options: nosniff
< Content-Length: 436
< X-ORACLE-DMS-ECID: 005jIN38df^Fw005zzh8iW0000wR00006H
< X-ORACLE-DMS-RID: 0:1
< Set-Cookie: JSESSIONID=j7caDRc1NmkioUuzoNTHv7nmgyzlNjfdbLea4DHGEFCnthED0kSb!-434689887; path=/; HttpOnly
< X-XSS-Protection: 1; mode=block
< Content-Type: text/html; charset=UTF-8
< 

<!DOCTYPE html>
<html>
<head>
  <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">

  <link rel="stylesheet" href="/testwebapp/res/styles.css;jsessionid=j7caDRc1NmkioUuzoNTHv7nmgyzlNjfdbLea4DHGEFCnthED0kSb!-434689887" type="text/css"> 
  <title>Test WebApp</title>
</head>
<body>

  <li>InetAddress: mspVM2/10.0.0.4
  <li>InetAddress.hostname: mspVM2

</body>
</html>
* Connection #0 to host wls-5d2ee46dcb-ohsstandalonedomain.eastus.cloudapp.azure.com left intact

Verify certificate is as expected.
Server certificate:
subject: C=US; ST=FL; L=Orlando; O=Microsoft; OU=Azure; CN=Ed Burns
start date: Mar 8 20:39:00 2021 GMT
expire date: Mar 3 20:39:00 2022 GMT
issuer: C=US; ST=FL; L=Orlando; O=Microsoft; OU=Azure; CN=Ed Burns
SSL certificate verify result: self signed certificate (18), continuing anywa

DNS

Perform cheapo DNS delegation. Modify WSL2 resolv.conf so that ns1-02.azure-dns.com is included in the list of nameserver entries.

https://superuser.com/questions/1533291/how-do-i-change-the-dns-settings-for-wsl2

New /etc/resolv.conf
```
nameserver 172.23.240.1
nameserver 40.90.4.2
```
curl --verbose --head http://www.ejb030901d.com:7777/testwebapp/index.jsp

$ curl --verbose --head http://www.ejb030901d.com:7777/testwebapp/index.jsp
*   Trying 52.249.186.110...
* TCP_NODELAY set
* Connected to www.ejb030901d.com (52.249.186.110) port 7777 (#0)
> HEAD /testwebapp/index.jsp HTTP/1.1
> Host: www.ejb030901d.com:7777
> User-Agent: curl/7.58.0
> Accept: */*
> 
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Date: Wed, 10 Mar 2021 02:54:25 GMT
Date: Wed, 10 Mar 2021 02:54:25 GMT
< X-Content-Type-Options: nosniff
X-Content-Type-Options: nosniff
< X-ORACLE-DMS-ECID: 005jIN8tyGeFw005zzh8iW0000wR00006L
X-ORACLE-DMS-ECID: 005jIN8tyGeFw005zzh8iW0000wR00006L
< X-ORACLE-DMS-RID: 0:1
X-ORACLE-DMS-RID: 0:1
< Set-Cookie: JSESSIONID=WYgaDpFkEEvs8C210ibnsRMxI1hE8wl_YPMu4zBz6k1cBk9aLq8w!-185493146; path=/; HttpOnly
Set-Cookie: JSESSIONID=WYgaDpFkEEvs8C210ibnsRMxI1hE8wl_YPMu4zBz6k1cBk9aLq8w!-185493146; path=/; HttpOnly
< X-XSS-Protection: 1; mode=block
X-XSS-Protection: 1; mode=block
< Content-Type: text/html; charset=UTF-8
Content-Type: text/html; charset=UTF-8

< 
* Connection #0 to host www.ejb030901d.com left intact

curl --verbose --head --insecure https://www.ejb030901d.com:4444/testwebapp/index.jsp

$ curl --verbose --head --insecure https://www.ejb030901d.com:4444/testwebapp/index.jsp
*   Trying 52.249.186.110...
* TCP_NODELAY set
* Connected to www.ejb030901d.com (52.249.186.110) port 4444 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
 CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: C=US; ST=FL; L=Orlando; O=Microsoft; OU=Azure; CN=Ed Burns
*  start date: Mar  8 20:39:00 2021 GMT
*  expire date: Mar  3 20:39:00 2022 GMT
*  issuer: C=US; ST=FL; L=Orlando; O=Microsoft; OU=Azure; CN=Ed Burns
*  SSL certificate verify result: self signed certificate (18), continuing anyway.
> HEAD /testwebapp/index.jsp HTTP/1.1
> Host: www.ejb030901d.com:4444
> User-Agent: curl/7.58.0
> Accept: */*
> 
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Date: Wed, 10 Mar 2021 02:55:07 GMT
Date: Wed, 10 Mar 2021 02:55:07 GMT
< X-Content-Type-Options: nosniff
X-Content-Type-Options: nosniff
< X-ORACLE-DMS-ECID: 005jINBO3cdFw005zzh8iW0000wR00006O
X-ORACLE-DMS-ECID: 005jINBO3cdFw005zzh8iW0000wR00006O
< X-ORACLE-DMS-RID: 0:1
X-ORACLE-DMS-RID: 0:1
< Set-Cookie: JSESSIONID=pJQaDzNGXHmP2AbLufgSymPdAIIzNSEaG4bthQZQzXfd6dXXcMfA!-185493146; path=/; HttpOnly
Set-Cookie: JSESSIONID=pJQaDzNGXHmP2AbLufgSymPdAIIzNSEaG4bthQZQzXfd6dXXcMfA!-185493146; path=/; HttpOnly
< X-XSS-Protection: 1; mode=block
X-XSS-Protection: 1; mode=block
< Content-Type: text/html; charset=UTF-8
Content-Type: text/html; charset=UTF-8

< 
* Connection #0 to host www.ejb030901d.com left intact

Verify certificate is as expected.

* Server certificate:
*  subject: C=US; ST=FL; L=Orlando; O=Microsoft; OU=Azure; CN=Ed Burns
*  start date: Mar  8 20:39:00 2021 GMT
*  expire date: Mar  3 20:39:00 2022 GMT
*  issuer: C=US; ST=FL; L=Orlando; O=Microsoft; OU=Azure; CN=Ed Burns
*  SSL certificate verify result: self signed certificate (18), continuing anyway.

curl --verbose --head http://admin.ejb030901d.com:7001/console

curl --verbose --head http://admin.ejb030901d.com:7001/console
*   Trying 40.76.48.36...
* TCP_NODELAY set
* Connected to admin.ejb030901d.com (40.76.48.36) port 7001 (#0)
> HEAD /console HTTP/1.1
> Host: admin.ejb030901d.com:7001
> User-Agent: curl/7.58.0
> Accept: */*
> 
< HTTP/1.1 302 Moved Temporarily
HTTP/1.1 302 Moved Temporarily
< Date: Wed, 10 Mar 2021 02:56:23 GMT
Date: Wed, 10 Mar 2021 02:56:23 GMT
< Location: http://admin.ejb030901d.com:7001/console/
Location: http://admin.ejb030901d.com:7001/console/
< Content-Type: text/html
Content-Type: text/html
< X-ORACLE-DMS-ECID: fa6b968b-aac7-4261-9bd7-0cb888a0b9fc-0000013f
X-ORACLE-DMS-ECID: fa6b968b-aac7-4261-9bd7-0cb888a0b9fc-0000013f
< X-ORACLE-DMS-RID: 0
X-ORACLE-DMS-RID: 0

< 
* Connection #0 to host admin.ejb030901d.com left intact

curl --verbose --head --insecure https://admin.ejb030901d.com:7002/console

$ curl --verbose --head --insecure https://admin.ejb030901d.com:7002/console
*   Trying 40.76.48.36...
* TCP_NODELAY set
* Connected to admin.ejb030901d.com (40.76.48.36) port 7002 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
 CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (OUT), TLS change cipher, Client hello (1):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Unknown (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS Unknown, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=MyState; L=MyTown; O=MyOrganization; OU=FOR TESTING ONLY; CN=adminVM
*  start date: Mar  7 19:37:02 2021 GMT
*  expire date: Mar  8 19:37:02 2036 GMT
*  issuer: C=US; ST=MyState; L=MyTown; O=MyOrganization; OU=FOR TESTING ONLY; CN=CertGenCA
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
* Using Stream ID: 1 (easy handle 0x5615091cb580)
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
> HEAD /console HTTP/2
> Host: admin.ejb030901d.com:7002
> User-Agent: curl/7.58.0
> Accept: */*
> 
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
< HTTP/2 302 
HTTP/2 302 
< date: Wed, 10 Mar 2021 02:56:57 GMT
date: Wed, 10 Mar 2021 02:56:57 GMT
< location: https://admin.ejb030901d.com:7002/console/
location: https://admin.ejb030901d.com:7002/console/
< content-type: text/html
content-type: text/html
< x-oracle-dms-ecid: fa6b968b-aac7-4261-9bd7-0cb888a0b9fc-00000140
x-oracle-dms-ecid: fa6b968b-aac7-4261-9bd7-0cb888a0b9fc-00000140
< x-oracle-dms-rid: 0
x-oracle-dms-rid: 0

< 
* Connection #0 to host admin.ejb030901d.com left intact

Verify certificate is as expected.

* Server certificate:
*  subject: C=US; ST=MyState; L=MyTown; O=MyOrganization; OU=FOR TESTING ONLY; CN=adminVM
*  start date: Mar  7 19:37:02 2021 GMT
*  expire date: Mar  8 19:37:02 2036 GMT
*  issuer: C=US; ST=MyState; L=MyTown; O=MyOrganization; OU=FOR TESTING ONLY; CN=CertGenCA
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.

wls-eng / arm-oraclelinux-wls

Dynamic Cluster: even with correctly formatted and filled out certificates, TLS/SSL blade prevents cluster formation #289

Basics

TLS/SSL

OHS

DNS -> Create+Review