ayoho
commented
3 years ago | Approved? | Spec | Issue | Contact | Last update | Comments |
|---|---|---|---|---|---|
| š | annotations-2.0 / Common Annotations | #11849 | @tkburroughs | 2021-03-02 | |
| š | interceptors-2.0 | #11853 | @tkburroughs | 2021-03-02 | |
| š | Injection (@Resource, @EJB, etc.) |
#11446 | @tkburroughs | 2021-03-02 | |
| š | CDI (@Inject etc) |
#11658 | 2021-03-23 | ||
| š | managedBeans-2.0 | #11850 | @tkburroughs | 2021-03-02 | |
| š | ejb-4.0 : Jakarta Enterprise Beans | #11450 | @tkburroughs | 2021-10-04 | |
| š | Jakarta Transactions 2.0 | #11772 | @dazavala | 2021-03-03 | |
| š | beanValidation-3.0 | #11916 | @nmittles | 2021-03-03 | |
| š | Jakarta Authentication | #11858 | @jvanhill | 2021-03-04 | |
| š | Jakarta Authorization | #11857 | @jvanhill | 2021-03-11 | |
| š | Jakarta Security | #11864 | @jvanhill | 2021-05-06 | |
| š | Password Utilities | #14198 | @jvanhill | 2021-03-04 | |
| š | Connector-2.0 / Jakarta Connection Architecture | #11330 | @KyleAure | 2021-05-10 | |
| š | Connector Inbound Security | #14014 | @jvanhill | 2021-03-04 | |
| š | JWT | #14858 | @jvanhill | 2021-03-11 | |
| š | Spnego | #14098 | @jvanhill | 2021-05-19 | |
| š | Audit | #14711 | @jvanhill | 2021-03-11 | |
| š | OAuth and SAML | #14873 | @jvanhill | 2021-03-11 | |
| š | Persistence Service for EJB Timers and Batch | #11951 | @jgrassel | 2021-03-11 | |
| š | JPA 3.0 | #11449 | @jgrassel | 2021-04-12 | |
| š | Concurrency 2.0 | #11329 | 2021-03-11 | ||
| š | Persistent Executor | #11952 | @njr-11 | 2021-04-05 | |
| š | JDBC | #11771 | 2021-03-11 | ||
| š | Jakarta Messaging 3.0 | #11851 | @dazavala | 2021-09-16 | |
| š | Jakarta JSON Processing and Binding 2.0 | #11278 | @dazavala | 2021-03-11 | |
| š | Websocket 2.0 | #11610 | @isaacrivriv | 2021-06-29 | |
| š | WebBundle (WAB) protected feature support for Jakarta 9 | #12281 | @sebratton | 2021-04-30 | |
| š | Jakarta Debugging Support for Other Languages | #11861 | 2021-03-12 | ||
| š | RestConnector and RestHandler | #11885 and #11886 | @anjumfatima90 | 2021-11-02 | |
| š | Session Cache and Session Database | #12984 | @randyclee | 2021-06-15 | |
| š | Faces 3.0 | #11611 | @volosied | 2021-06-30 | |
| š | Expression Language 4.0 | #11720 | @volosied | 2021-10-05 | |
| š | Servlet 5.0 | #11819 | @volosied | 2021-10-25 | |
| š | OpenIdConnect and / or Social Login | #14874 | @jvanhill | 2021-05-19 | |
| š | RESTful Web Services 3.0 | #11803 | @andymc12 | 2021-09-24 | |
| š | JWS API | #11852 | @neuwerk | 2021-10-14 | |
| š | XML WS | #11862 | @neuwerk | 2021-10-28 | |
| š | Server Pages | #11717 | @volosied | 2021-10-18 | |
| š | Batch | #11863 | @cgianfra or @scottkurz | 2021-11-02 | |
| š | batchManagement-1.0 | #16646 | @cgianfra or @scottkurz | 2021-11-02 | |
| š | Activation | #11856 | @neuwerk | 2021-10-28 | |
| š | JakartaeeClient-9.0 feature | #11890 | @cbridgha | 2021-06-28 | |
| š | XML Binding API | #11859 | @neuwerk | 2021-11-11 | |
| š | #11860 | @neuwerk | 2021-10-14 | ||
| š | FacesContainer-3.0 | #12234 | @volosied | 2021-06-28 | |
| š | adminCenter-1.0 | #16640 | @jhanders34 | 2021-10-28 | |
| š | acmeCA-2.0 | #16641 | @jvanhill | 2021-05-19 | |
| š | distributedMap / webCache | #16649 | @jhanders34 | 2021-10-07 | |
| š | federatedRegistry-1.0 | #16642 | @jvanhill | 2021-06-07 | |
| š | grpc-1.0 & grpcClient-1.0 | #16647 | @pnicolucci / @volosied | 2021-11-02 | |
| š | wsAtomicTransaction-1.2 | #12726 | @jonhawkes / @jhanders34 | 2021-10-28 | |
| š | wsSecurity-1.1 / wsSecuritySaml-1.1 | #12449 | @arunavemulapalli | 2021-10-28 |
1) Describe the test strategy & approach for this feature, and describe how the approach verifies the functions delivered by this feature. The description should include the positive and negative testing done, whether all testing is automated, what manual tests exist (if any) and where the tests are stored (source control). Automated testing is expected for all features with manual testing considered an exception to the rule.
For any feature, be aware that only FAT tests (not unit or BVT) are executed in our cross platform testing. To ensure cross platform testing ensure you have sufficient FAT coverage to verify the feature.
If delivering tests outside of the standard Liberty FAT framework, do the tests push the results into cognitive testing database (if not, consult with the CSI Team who can provide advice and verify if results are being received)?
The general test strategy for the Jakarta EE 9 Feature is to repeat all tests run for Jakarta EE 8 by transforming them to use the Jakarta packages. Here are the summaries per Jakarta EE 9 feature/component area. It may be assumed that the features/components documented here had no new function introduced, other than support for the new deployment descriptor versions (web.xml, application.xml, ejb-jar.xml etc). Jakarta EE 9 features that did contain functional changes should have separate feature/epics to track the work and FAT.
š annotations-2.0 / Common Annotations : https://github.com/openliberty/open-liberty/issues/11849 (@tkburroughs)
š interceptors-2.0 : https://github.com/openliberty/open-liberty/issues/11853 (@tkburroughs)
š Injection (
@Resource,@EJB, etc.) : https://github.com/openliberty/open-liberty/issues/11446 (@tkburroughs)š CDI (
@Injectetc) : https://github.com/OpenLiberty/open-liberty/issues/11658 (@tevans78)š managedBeans-2.0 : https://github.com/openliberty/open-liberty/issues/11850 (@tkburroughs)
š ejb-4.0 : Jakarta Enterprise Beans : https://github.com/OpenLiberty/open-liberty/issues/11450 (@tkburroughs)
š Jakarta Transactions 2.0: #11772 (@dazavala)
transaction-2.0,jta-2.0. All tests repeat for EE9 except one test involving MicroProfile.Confidence: 4. The Transactions FAT inventory repeats for EE9 all use-cases that verify Jakarta Transaction 1.3 and proprietary SPI requirements. FAT suites for non-Transactions EE9 features that require Jakarta Transactions 2.0 support pass at 100%.
š beanValidation-3.0 : https://github.com/OpenLiberty/open-liberty/issues/11916 (@nmittles)
š Jakarta Authentication: #11858 (@jvanhill)
š Jakarta Authorization: #11857 (@jvanhill)
š Jakarta Security: #11864 (@jvanhill)
š Password Utilities: #14198 (@jvanhill)
š Connector-2.0 / Jakarta Connection Architecture : #11330 (@KyleAure)
š Connector Inbound Security: #14014 (@jvanhill)
š JWT: #14858 (@jvanhill)
š Spnego: #14098 (@jvanhill)
š Audit: #14711 (@jvanhill)
š OAuth and SAML: #14873 (@jvanhill)
š Persistence Service for EJB Timers and Batch: #11951 (@jgrassel)
š JPA 3.0: #11449 (@jgrassel)
š Concurrency 2.0: #11329
š Persistent Executor: #11952
š JDBC: #11771
š Jakarta Messaging 3.0: #11851 (@dazavala)
messaging-3.0,messagingClient-3.0,messagingServer-3.0,messagingSecurity-3.0. All tests repeat for EE9.š Jakarta JSON Processing and Binding 2.0: #11278 (@dazavala)
jsonp-2.0,jsonb-2.0,jsonpContainer-2.0,jsonpContainer-2.0. All tests repeat for EE9 except those involving user features that export the johnzon jsonp provider.š Websocket 2.0: #11610
š WebBundle (WAB) protected feature support for Jakarta 9 #12281. Supports WAB coexistence with servlet-5.0
com.ibm.ws.app.manager.wab.installer_fat was ported from closed Liberty for basic golden path coverage and limited negative testing. Tests interaction of web container with WAB with and without jakartaEE9. Tests were added to exercise servlet filters, servlet context listeners, web.xml descriptor and static web content. There are additional WAB tests in closed liberty which could not be ported to the open side since they are based on OSGi applications.
com.ibm.ws.rest.handler.config.fat further exercises the webBundle feature with servlet-5.
Confidence: 2. This is deemed to be sufficient given the limited points of interaction between the WAB manager and servlet 5 packages.
š Jakarta Debugging Support for Other Languages #11861
š RestConnector and RestHandler: #11885 and #11886: RestHandler is a protected feature in RestConnector.
š Session Cache and Session Database: #12984
sessionCache-1.0
Tests in the following buckets were not repeated due to absence of jakartaee enabled microprofile features mpMetrics-2.0 and mpReactiveStreams-1.0
Confidence: 3. All the tests were repeated except those requiring microProfile.
sessionDatabase-1.0
Confidence: 3. All the tests requiring sessionDatabse were set to repeat with EE9.
š Faces 3.0: #11611
š Expression Language 4.0: #11720
š OpenIdConnect and SocialLogin: #14874 (@jvanhill)
š acmeCA-2.0: #16641 (@jvanhill)
š Jakarta App Client Support: #11890 (@cbridgha)
appClientSupport-2.0. Many tests repeat for EE9 where applicable. - No new tests added. - Initially needed to add additional "client" support to transform test apps. Buckets:š xmlWS-3.0 (JWS + XML WS): #11862 (@neuwerk)
š xmlBinding-3.0 (XML binding): #11859 (@neuwerk)
š mail-2.0: #11860 (@neuwerk)
š federatedRegistry-1.0: #16642 (@jvanhill)
š facesContainer-3.0: #12234 (@volosied)
š pages-3.0: #11717 (@volosied)
š servlet-5.0: #11819 (@volosied)
š adminCenter-1.0: #16640 (@amyreit)
š grpc-1.0 & grpcClient-1.0: #16647 (@volosied)
See the follow up issue #18137. However, Bill Lucy, Jared Anderson, and Paul Nicolucci agree that we can still mark grpc-1.0 and grpcClient-1.0 as FAT complete since the features themselves work properly when used with servlet-5.0.
š wsAtomicTransaction-1.2: #12726 (@jhanders, @neuwerk, @jonhawkes )
š wsSecurity-1.1 and wsSecuritySaml-1.1: #12449 (@arunavemulapalli )
2) Collectively as a team you need to assess your confidence in the testing delivered based on the values below. This should be done as a team and not an individual to ensure more eyes are on it and that pressures to deliver quickly are absorbed by the team as a whole.
Please indicate your confidence in the testing (up to and including FAT) delivered with this feature by selecting one of these values:
0 - No automated testing delivered
1 - We have minimal automated coverage of the feature including golden paths. There is a relatively high risk that defects or issues could be found in this feature.
2 - We have delivered a reasonable automated coverage of the golden paths of this feature but are aware of gaps and extra testing that could be done here. Error/outlying scenarios are not really covered. There are likely risks that issues may exist in the golden paths
3 - We have delivered all automated testing we believe is needed for the golden paths of this feature and minimal coverage of the error/outlying scenarios. There is a risk when the feature is used outside the golden paths however we are confident on the golden path. Note: This may still be a valid end state for a feature... things like Beta features may well suffice at this level.
4 - We have delivered all automated testing we believe is needed for the golden paths of this feature and have good coverage of the error/outlying scenarios. While more testing of the error/outlying scenarios could be added we believe there is minimal risk here and the cost of providing these is considered higher than the benefit they would provide.
5 - We have delivered all automated testing we believe is needed for this feature. The testing covers all golden path cases as well as all the error/outlying scenarios that make sense. We are not aware of any gaps in the testing at this time. No manual testing is required to verify this feature.
Based on your answer above, for any answer other than a 4 or 5 please provide details of what drove your answer. Please be aware, it may be perfectly reasonable in some scenarios to deliver with any value above. We may accept no automated testing is needed for some features, we may be happy with low levels of testing on samples for instance so please don't feel the need to drive to a 5. We need your honest assessment as a team and the reasoning for why you believe shipping at that level is valid. What are the gaps, what is the risk etc. Please also provide links to the follow on work that is needed to close the gaps (should you deem it needed)