search

OpenLiberty / open-liberty

Open Liberty is a highly composable, fast to start, dynamic application server runtime environment
https://openliberty.io
Eclipse Public License 2.0
1.14k stars 587 forks source link

Feature Test Summary: MicroProfile Reactive Streams Operators 1.0 #4087

Closed Emily-Jiang closed 5 years ago

Emily-Jiang commented 6 years ago

**1) Describe the test strategy & approach for this feature, and describe how the approach verifies the functions delivered by this feature.

All testing is automated and fully integrated with the Simplicity Test Infrastructure for reporting anything less than 100% pass:

Testing occurs in 4 layers of automated test suites:

Unit Testing:

Unit tests are stored in com.ibm.ws.microprofile.reactive.streams.operators/test/src/com/ibm/ws/microprofile/reactive/streams/test and are run automatically as part of building the implementation bundle or by running

./gradlew com.ibm.ws.microprofile.reactive.streams.operators:test

There are currently 8 unit test classes with 25 tests that have been peer reviewed at https://github.com/OpenLiberty/open-liberty/pull/6617#pullrequestreview-206221908 as a valid UT suite at. The purpose of this suite is to check a basic implementation and integration are operational.

The Reactive Streams .Org TCK

This is a highly mature, comprehensive and stable test suite which lives at: https://github.com/reactive-streams/reactive-streams-jvm/tree/master/tck There are 251 automated tests that provide a thorough verification that the implementation is compliant with the Reactive Streams TCK.

The Reactive Streams .Org TCK is run as part of executing:

./gradlew com.ibm.ws.microprofile.reactive.streams.operators_fat_tck:buildandrun

This test suite is run automatically as part of the main build verification process.

The MicroProfile Reactive Streams Operators TCK

This is a cross vendor TCK that has multiple authors and is reviewed as being a comprehensive verification suite for the specification. It is stored at: [https://github.com/eclipse/microprofile-reactive-streams-operators/tree/master/tck] (https://github.com/eclipse/microprofile-reactive-streams-operators/tree/master/tck) It contains 339 testcases (not including the embedded Reactive Streams .Org TCK)

The MicroProfile Reactive Streams Operators TCK is run by executing:

./gradlew com.ibm.ws.microprofile.reactive.streams.operators_fat_tck:buildandrun

The WAS mpReactiveStreams Feature FAT bucket

The MicroProfile Reactive Streams Operators IBM implementation in WAS has an IBM specific FAT test bucket. This bucket aims to test the integration of the base implementation in the Liberty environment together with some basic testing of the function in the context of a Web Application (Servlet/CDI etc). There are over 500 tests in the automated TCK test suites, so this test bucket tries to concentrate on inter-component and contextual tests.

ReactiveJaxRSTest

This test reactive stream usage in the context of a JAX-RS application with Publisher to Subscriber linking and the use of the default reactive streams engine.

ContextPropogationTest

This test tests that the callback elements (Publishers/Processors/Subscribers and operators' callbacks and lambdas) are all run with contexts that are as expected according to the MicroProfile Concurrently Specification.

Tests using the stream completion CompletableFuture

PostStreamCompletionCapturedThreadContextCleared PostStreamCompletionCapturedThreadContextProgated PostStreamCompletionCapturedThreadContextUnchanged

Tests using a Default Engine for CDI Propogation

Contexts tested are:

We test that the correct (different/same) bean is injectable at different stages of the pipeline.

ConcurrentWorkTest

We test that it is possible to run stream methods with appropiate concurrency

StreamContextViaThreadContextProviderSPI

This explores using the MicroProfile Concurrency SPI to provide some interesting custom 'stream' contexts down the stream alongside application data (for example for tracing or workflow identifier).

Error Messages

Explore rxjava, smallrye and IBM messages are all OK.

Running the FAT bucket

The FAT bucket runs by default with each build and is also integrated into the standard Simplicity framework and can be run with

./gradlew com.ibm.ws.microprofile.reactive.streams.operators_fat:buildandrun

SVT Testing

The SVT Test Team (Brian P. Hanczaryk et al @hanczaryk ) are already testing this feature in their stress and load tests, for example 10 million server requests @ 4K per second to raise this defect:

https://github.ibm.com/was-liberty/WS-CD-Open/issues/15521

SVT Objective: Test a Liberty Collective topology leveraging mpReactiveStreams-1.0 with the ProShop application under load for 72 consecutive hours

Dependencies

VM needs VM #1-2 (controller/liberty server) : (2 CPU / 4 GB RAM) VM #3 (SQLServer 2016) : (4 CPU / 16 GB RAM) VM #4-5 (RPT/jmeter) : (2 CPU / 4 GB RAM)

Scenario test files

ProShop app updated during Test Case SVT: APSC_DeveloperExperience_mpReactiveStreams1.0_ProShop_Tooling (#15591)

WebSphere documentation The documentation listed should be verified throughout the testcase execution with a final verification in section 3.

WebSphere topology

VM #1-2 (controller/liberty server) : (2 CPU / 4 GB RAM) VM #3 (SQLServer 2016) : (4 CPU / 16 GB RAM) VM #4-5 (RPT/jibe) : (2 CPU / 4 GB RAM)

Installation/Configuration Instructions

Machine #1-2

Install lWAS on machine Create controller a) ./server create myController b) ./collective create myController --keystorePassowrd=<> c) copy command window output into server.xml as instructed d) ./server start myController Create liberty servers onVMs #2 a) ./server create myMemberX Join liberty servers to the collective a) ./collective join myMemberX --host=<> --port=<> --user=<> --password=<> --keystorePassword=<> Edit values for each myMemberX's server.xml with the result from collective join , unique port info, and cluster info.

Copy ProShop.war into app directory for each member. Restore/Create ProShop SQL Server 2016 for use with ProShop app stress Ensure that ProShop is functional. Execution Instructions

Setup RPT/JMeter/Jibe on VM #4 & #5 if not already done. Start RPT and capture test recording for ProShop at http://:80/ProShop/ (i.e. for mpReactiveStreams-1.0 test, hit http://:80/ProShop/ReactiveStreamsTestServlet) Create a performance test schedule to drive app load and start stress run Run stress against ProShop.war to push a baseline cpu load. While the stress run takes place : Check logs for exceptions/failures Check ffdc logs for serviceability defects

After 72 hours run of app load and admin load, stop RPT & Jibe stress.

Final Verification Instructions

After 72 hours of RPT run has completed.

Stop the RPT run Check logs for exceptions/failures Check ffdc logs for serviceability defects Zip logs and upload to GSA space

See https://github.ibm.com/was-liberty/WS-CD-Open/issues/15599 for more information.

Test Repository

All of Development's tests are stored in github in the paths above suffixed onto Open Liberty Github

The SVT ProShop test is stored at: https://github.ibm.com/hanczar-maven/ProShop

Confidence Level

Please indicate your confidence in the testing (up to and including FAT) delivered with this feature by selecting one of these values:

The current assessment from the squad is 4. (The testing that does not run as part of the FAT suite is the long running (millions of hits) SVT style testing.)

0 - No automated testing delivered

1 - We have minimal automated coverage of the feature including golden paths. There is a relatively high risk that defects or issues could be found in this feature.

2 - We have delivered a reasonable automated coverage of the golden paths of this feature but are aware of gaps and extra testing that could be done here. Error/outlying scenarios are not really covered. There are likely risks that issues may exist in the golden paths

3 - We have delivered all automated testing we believe is needed for the golden paths of this feature and minimal coverage of the error/outlying scenarios. There is a risk when the feature is used outside the golden paths however we are confident on the golden path. Note: This may still be a valid end state for a feature... things like Beta features may well suffice at this level.

4 - We have delivered all automated testing we believe is needed for the golden paths of this feature and have good coverage of the error/outlying scenarios. While more testing of the error/outlying scenarios could be added we believe there is minimal risk here and the cost of providing these is considered higher than the benefit they would provide.

5 - We have delivered all automated testing we believe is needed for this feature. The testing covers all golden path cases as well as all the error/outlying scenarios that make sense. We are not aware of any gaps in the testing at this time. No manual testing is required to verify this feature.

hutchig commented 5 years ago

I will follow this process as it was at the date of this comment: https://github.ibm.com/was-liberty/WS-CD-Open/wiki/Feature-Review-(Feature-Test-Summary-Process)

hutchig commented 5 years ago

Any build that contains https://github.com/OpenLiberty/open-liberty/pull/7191 #7191 will contain 100% of the features and tests we plan to deliver for this feature. That pull request is now part of integration branch.

hutchig commented 5 years ago

All FATs are delivered and passing.