1) Describe the test strategy & approach for this feature, and describe how the approach verifies the functions delivered by this feature.
Due to Liberty’s default SSL protocol settings, the TLSv1.3 protocol is enabled by default when using a Liberty default SSL configuration. The Liberty default setting allows for all TLS values, that is TLSv1, TLSv1.1, TLSv1.2 and on java 11 TLSv1.3. Most tests outside or the SSL test bucket using SSL are not doing much along the line of custom configuration of SSL, so they end up using the TLSv1.3 protocol.
New and modified test added to com.ibm.ws.ssl_fat bucket
Positive scenario tests
Existing tests in the Positive scenario, that tests client access to the server with each protocol type with the server’s default protocol setting of all TLS values. This test was augmented to include a client access with TLSv1.3 when running on Java 11.
New tests were added to the Positive scenario to include tests where the server is configured with an exclusive TLSv1.3 setting. Doing a normal connection and a client authenticated connection.
Negative scenario tests
New tests added to the Negative scenario tests to make sure client error when it tries to access the server configured for TLSv1.3 only and the client is using a lower protocol. Tested client authenticated SSL failures when TLSv1.3 is enabled.
Existing tests across the Liberty test bucket.
Like mentioned above any test in the bucket using Liberty’s default SSL configuration will end up using TLSv1.3, due to the JDK’s design. Some notable buckets that use extension SSL connections include:
All the collective tests, com.ibm.ws.collective*_fat buckets. All collective controller/member connections are over SSL, most tests are using default SSL setting so they end up using TLSv1.3 when running on Java 11.
The webcontainer security test, com.ibm.ws.webcontainer.security_fat. This bucket includes tests that do certificate logins, which depends on SSL client authentication feature.
The LDAP test cases, com.ibm.ws.security.wim.adapter.ldap_fat. There is extensive testing with SSL connections to LDAP servers. While the LDAP servers are not supporting TLSv1.3 yet these test show that Liberty’s default setting of a range of protocols works when accessing server that do not support TLSv1.3.
There are many other tests in Liberty using SSL connections. The more notable ones were mentioned before.
2) Collectively as a team you need to assess your confidence in the testing delivered based on the values below. This should be done as a team and not an individual to ensure more eyes are on it and that pressures to deliver quickly are absorbed by the team as a whole.
Please indicate your confidence in the testing (up to and including FAT) delivered with this feature by selecting one of these values:
0 - No automated testing delivered
1 - We have minimal automated coverage of the feature including golden paths. There is a relatively high risk that defects or issues could be found in this feature.
2 - We have delivered a reasonable automated coverage of the golden paths of this feature but are aware of gaps and extra testing that could be done here. Error/outlying scenarios are not really covered. There are likely risks that issues may exist in the golden paths
3 - We have delivered all automated testing we believe is needed for the golden paths of this feature and minimal coverage of the error/outlying scenarios. There is a risk when the feature is used outside the golden paths however we are confident on the golden path. Note: This may still be a valid end state for a feature... things like Beta features may well suffice at this level.
4 - We have delivered all automated testing we believe is needed for the golden paths of this feature and have good coverage of the error/outlying scenarios. While more testing of the error/outlying scenarios could be added we believe there is minimal risk here and the cost of providing these is considered higher than the benefit they would provide.
5 - We have delivered all automated testing we believe is needed for this feature. The testing covers all golden path cases as well as all the error/outlying scenarios that make sense. We are not aware of any gaps in the testing at this time. No manual testing is required to verify this feature.
Our collective confidence is 4. Due to the new test added and the amount of using Liberty default SSL settings and new test cases added to use TLSv1.3 coverage is good.
arkarkala
commented
5 years ago
1) Describe the test strategy & approach for this feature, and describe how the approach verifies the functions delivered by this feature.
Due to Liberty’s default SSL protocol settings, the TLSv1.3 protocol is enabled by default when using a Liberty default SSL configuration. The Liberty default setting allows for all TLS values, that is TLSv1, TLSv1.1, TLSv1.2 and on java 11 TLSv1.3. Most tests outside or the SSL test bucket using SSL are not doing much along the line of custom configuration of SSL, so they end up using the TLSv1.3 protocol.
New and modified test added to com.ibm.ws.ssl_fat bucket
Positive scenario tests
Existing tests in the Positive scenario, that tests client access to the server with each protocol type with the server’s default protocol setting of all TLS values. This test was augmented to include a client access with TLSv1.3 when running on Java 11.
New tests were added to the Positive scenario to include tests where the server is configured with an exclusive TLSv1.3 setting. Doing a normal connection and a client authenticated connection.
Negative scenario tests
New tests added to the Negative scenario tests to make sure client error when it tries to access the server configured for TLSv1.3 only and the client is using a lower protocol. Tested client authenticated SSL failures when TLSv1.3 is enabled.
Existing tests across the Liberty test bucket.
Like mentioned above any test in the bucket using Liberty’s default SSL configuration will end up using TLSv1.3, due to the JDK’s design. Some notable buckets that use extension SSL connections include:
All the collective tests, com.ibm.ws.collective*_fat buckets. All collective controller/member connections are over SSL, most tests are using default SSL setting so they end up using TLSv1.3 when running on Java 11. The webcontainer security test, com.ibm.ws.webcontainer.security_fat. This bucket includes tests that do certificate logins, which depends on SSL client authentication feature.
The LDAP test cases, com.ibm.ws.security.wim.adapter.ldap_fat. There is extensive testing with SSL connections to LDAP servers. While the LDAP servers are not supporting TLSv1.3 yet these test show that Liberty’s default setting of a range of protocols works when accessing server that do not support TLSv1.3. There are many other tests in Liberty using SSL connections. The more notable ones were mentioned before.
2) Collectively as a team you need to assess your confidence in the testing delivered based on the values below. This should be done as a team and not an individual to ensure more eyes are on it and that pressures to deliver quickly are absorbed by the team as a whole.
Please indicate your confidence in the testing (up to and including FAT) delivered with this feature by selecting one of these values:
0 - No automated testing delivered
1 - We have minimal automated coverage of the feature including golden paths. There is a relatively high risk that defects or issues could be found in this feature.
2 - We have delivered a reasonable automated coverage of the golden paths of this feature but are aware of gaps and extra testing that could be done here. Error/outlying scenarios are not really covered. There are likely risks that issues may exist in the golden paths
3 - We have delivered all automated testing we believe is needed for the golden paths of this feature and minimal coverage of the error/outlying scenarios. There is a risk when the feature is used outside the golden paths however we are confident on the golden path. Note: This may still be a valid end state for a feature... things like Beta features may well suffice at this level.
4 - We have delivered all automated testing we believe is needed for the golden paths of this feature and have good coverage of the error/outlying scenarios. While more testing of the error/outlying scenarios could be added we believe there is minimal risk here and the cost of providing these is considered higher than the benefit they would provide.
5 - We have delivered all automated testing we believe is needed for this feature. The testing covers all golden path cases as well as all the error/outlying scenarios that make sense. We are not aware of any gaps in the testing at this time. No manual testing is required to verify this feature.
Our collective confidence is 4. Due to the new test added and the amount of using Liberty default SSL settings and new test cases added to use TLSv1.3 coverage is good.