Plan to execute pre-checks and report outcome

fabiobrz commented 4 years ago

Pre-checks must be done ASAP to comply with TP

Current TD status with respect to requirements in TP is missing some tests that have yet to be developed, here's a list of the missing cases:

Integration testing

MP specs

MP Config: MP OpenAPI annotations sources and MP OpenAPI extended properties effect on multiple deployment environment (e.g. WARNs)

Customer/advanced/negative scenarios

one test on JWT related Security annotations, see https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/quarkus-dev/QZgqrblsYEc/sgDcnHYlBAAJ (should this one be added to TP - Integration testing?)
mixed approach to OpenAPI document model generation (e.g.: static file + programming model + annotations)
Negative scenario about headers, see https://github.com/fabiobrz/eap-microprofile-test-suite/issues/1#issuecomment-564622031
multiple and subsequent deployments/undeployments
WARN for missing HTTP listener

Security testing

The endpoint should be exposed via HTTP per specs but Wildfly will allow access via HTTPS transport in case such a listener is configured.
Tests to verify that the /openapi endpoint will be available in such case and to evaluate WARNs.

Also misc tests

Performance: time to boot/deploy, memory/disk pressure with hundreds of annotated endpoints

OBSOLETE (see new checklist https://github.com/fabiobrz/eap-microprofile-test-suite/issues/11#issuecomment-570642787): Plan

[x] define how to perform manual checks on missing items (MP config first)
[x] start with manual checks on missing items
OBSOLETE (see new checklist https://github.com/fabiobrz/eap-microprofile-test-suite/issues/11#issuecomment-570642787): check WARNINGs not logged by SmallRye? File a JIRA for not logging unwanted/misplaced properties like in JWT sample?
OBSOLETE (see new checklist https://github.com/fabiobrz/eap-microprofile-test-suite/issues/11#issuecomment-570642787): execute the TS for the MP OpenAPI spec
[x] execute Wildfly TS MP OpenAPI integration tests
[x] execute manual experiments (try generate and use clients too, i.e. swagger editor)
[x] provide current status to DEV - i.e.
TP/TD ratio
enlist missing scenarios and that those are covered manually
status/outcome of tests (both manual and automated)
[x] update document at https://docs.google.com/document/d/1DRo5RIT-E7g-GuRztB_7r5hTDc89e0-sEdTh8kW28UU/edit
[x] update JIRA

Roadmap/ETA

This is top priority so all automated and manual pre-checks must be executed today/tomorrow morning
Report must be given within tomorrow EOB

fabiobrz commented 4 years ago

define how to perform manual checks on missing items (MP config first)

MP Config Integration

Manual testing will use an ad-hoc basic application for the following scenario: Local Services Router application will have to override OpenAPI doc generated from openapi.yaml file provided by Service Provider through the use of the mp.openapi.servers property, so that the server will be referencing the local domain. Tests will assess that the generated OpenAPI doc is actually returning modified elements.

Incorporated into TS, see https://github.com/jboss-eap-qe/eap-microprofile-test-suite/pull/117

Customer/advanced/negative scenarios

one test on JWT related Security annotations, see https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/quarkus-dev/QZgqrblsYEc/sgDcnHYlBAAJ (should this one be added to TP - Integration testing?)

Done, OK - Expected security info is properly documented in resource class section

fabiobrz commented 4 years ago

Security testing

The endpoint should be exposed via HTTP per specs but Wildfly will allow access via HTTPS transport in case such a listener is configured.

Tests to verify that the /openapi endpoint will be available in such case and to evaluate WARNs

Done, incorporated into TS.

fabiobrz commented 4 years ago

Define how to perform manual checks on missing items (MP config first)

Done: everything is goingh to be incorporated as just a small amount of tests is missing ATM

Start with manual checks on missing items

Done: worked for config and JWT/security annotations, from now on, no manual except for anything that could not hit the deadline

Execute Wildfly TS MP OpenAPI integration tests

Done - TBD: attach build output

Provide current status to DEV
Done, see https://issues.redhat.com/browse/EAP7-1319?focusedCommentId=13938910&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-13938910

Update document at https://docs.google.com/document/d/1DRo5RIT-E7g-GuRztB_7r5hTDc89e0-sEdTh8kW28UU/edit

Done

Update JIRA

Done, see https://issues.redhat.com/browse/EAP7-1319?focusedCommentId=13938910&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-13938910

fabiobrz commented 4 years ago

New checklist for pre-check

[x] 1. More tests not in TP
negative scenario - multiple yamls (?) => test needed, see https://download.eclipse.org/microprofile/microprofile-open-api-1.1.2/microprofile-openapi-spec.html#_location_and_formats as the spec does not mandate vendor implementation UPDATE TBD ->Done
server/pathItem => test needed, see https://download.eclipse.org/microprofile/microprofile-open-api-1.1.2/microprofile-openapi-spec.html#_context_root_behavior TBD although Wildfly TS already tests for auto generated servers, an operation could override generated values. What are we expecting for?
time to serve doc when there are a lot of endpoints as DEV report that generation is on-demand, no persistence at all UPDATE TBD ->Done
test broken static file Done
test EAR Done
[x] 2. Re-run initial checks and all of the TSs against the feature branch Was "execute the TS for the MP OpenAPI spec", TBD Widfly TS + EAP QE TS on feature branch.
[x] 3. Perform some log observation TBD (currently no messages at all coming from the io.smallrye.open-api and DEV says it's just like that, it isn't logging at all...) Specifically check WARNINGs not logged by SmallRye. File a JIRA for not logging unwanted/misplaced properties like in JWT sample?
[x] 4. Finish looking at the Wildfly PR changes TBD, about 20/58 at the moment
[x] 5. Execute manual experiments (try generate and use clients too, i.e. swagger editor) Done, see https://github.com/fabiobrz/eap-microprofile-test-suite/issues/1#issuecomment-570638749

fabiobrz commented 4 years ago

1. More tests not in TP

negative scenario - multiple yamls (?) => test needed, see https://download.eclipse.org/microprofile/microprofile-open-api-1.1.2/microprofile-openapi-spec.html#_location_and_formats Done - OK: only one file is always properly selected.

fabiobrz commented 4 years ago

1. More tests not in TP

test broken static file Done => file a non blocking JIRA because spec don't mandate vendors for model validation
test EAR Done => see whether MP Config issue with EAR shows up when deploying manually/file a JIRA (MP Config)?

fabiobrz commented 4 years ago

3. Perform some log observation Done messages from io.smallrye.openapi are logged but only "DEBUG" messages are logged, there are no WARNs - even when model is not valid - and definitely no message IDs. TBD: file JIRA for message IDs and missing WARNs?

2020-01-05 16:58:49,225 DEBUG [io.smallrye.openapi.runtime.util.SchemaFactory] (MSC service thread 1-4) Processing a single @Schema annotation.
2020-01-05 16:58:49,225 DEBUG [io.smallrye.openapi.runtime.scanner.OpenApiAnnotationScanner] (MSC service thread 1-4) Processing a single @Response annotation.
2020-01-05 16:58:49,225 DEBUG [io.smallrye.openapi.runtime.scanner.OpenApiAnnotationScanner] (MSC service thread 1-4) Processing jax-rs method: {0}
2020-01-05 16:58:49,225 DEBUG [io.smallrye.openapi.runtime.scanner.OpenApiDataObjectScanner] (MSC service thread 1-4) Starting processing with root: java.lang.String
2020-01-05 16:58:49,225 DEBUG [io.smallrye.openapi.runtime.scanner.OpenApiAnnotationScanner] (MSC service thread 1-4) Processing a single @RequestBody annotation.
2020-01-05 16:58:49,225 DEBUG [io.smallrye.openapi.runtime.scanner.OpenApiAnnotationScanner] (MSC service thread 1-4) Processing a single @Response annotation.
2020-01-05 16:58:49,225 DEBUG [io.smallrye.openapi.runtime.scanner.OpenApiAnnotationScanner] (MSC service thread 1-4) Processing a single @Content annotation.
2020-01-05 16:58:49,225 DEBUG [io.smallrye.openapi.runtime.scanner.OpenApiAnnotationScanner] (MSC service thread 1-4) Processing a single @Content annotation as a MediaType.
2020-01-05 16:58:49,225 DEBUG [io.smallrye.openapi.runtime.util.SchemaFactory] (MSC service thread 1-4) Processing a single @Schema annotation.
2020-01-05 16:58:49,225 DEBUG [io.smallrye.openapi.runtime.scanner.OpenApiAnnotationScanner] (MSC service thread 1-4) Processing a single @Response annotation.

UPDATE: Done - JIRA https://issues.redhat.com/browse/WFWIP-291 filed.

fabiobrz commented 4 years ago

1. More tests not in TP

test broken static file file a non blocking JIRA because spec don't mandate vendors for model validation UPDATE: TBD -> Done - JIRA https://issues.redhat.com/browse/WFWIP-292 filed

fabiobrz commented 4 years ago

1. More tests not in TP

time to serve doc when there are a lot of endpoints Done - OK but discuss with team (DEV) for a 50 MB peak because of a 3MB yaml.

fabiobrz commented 4 years ago

1. More tests not in TP

server/pathItem test needed, see https://download.eclipse.org/microprofile/microprofile-open-api-1.1.2/microprofile-openapi-spec.html#_context_root_behavior

although Wildfly TS already tests for auto generated servers, an operation could override generated values. We're expecting it works as per spec - i.e. overriding the higher level information

UPDATE TBD ->Done

fabiobrz commented 4 years ago

Pre-check protocol

Wildfly feature branch: https://github.com/pferraro/wildfly/tree/WFLY-12313
commit: 803a542bba53b30bacf4c0003d46838191c6afd3
TS version: https://github.com/fabiobrz/eap-microprofile-test-suite/tree/mp-openapi-non-tp-misc-advanced

Checklist

[x] Re-run initial checks and all of the TSs against the feature branch
Widfly TS integration tests
Wildfly MicroProfile tck
EAP QE TS
[x] Perform some log observation Specifically to confirm findings reported in https://issues.redhat.com/projects/WFWIP/issues/WFWIP-291
[x] Execute manual experiments (try generate and use clients too, i.e. swagger editor) See https://github.com/fabiobrz/eap-microprofile-test-suite/issues/1#issuecomment-570638749
[x] Execute memory test for multiple redeploy See https://issues.redhat.com/browse/WFLY-10991?focusedCommentId=13630480&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-13630480
[x] Finish looking at the Wildfly PR changes About 20/58 at the moment

fabiobrz commented 4 years ago

Re-run initial checks and all of the TSs against the feature branch

Widfly TS integration tests

$ mvn clean test -Dmaven.repo.local=local-repo -Dts.integration -pl testsuite/integration/microprofile
...
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.wildfly.test.integration.microprofile.faulttolerance.async.requestcontext.AsynchronousRequestContextTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.874 s - in org.wildfly.test.integration.microprofile.faulttolerance.async.requestcontext.AsynchronousRequestContextTest
[INFO] Running org.wildfly.test.integration.microprofile.faulttolerance.sync.SyncCircuitBreakerDisabledTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.889 s - in org.wildfly.test.integration.microprofile.faulttolerance.sync.SyncCircuitBreakerDisabledTest
[INFO] Running org.wildfly.test.integration.microprofile.openapi.OpenAPIMultiModuleDeploymentTestCase
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.376 s - in org.wildfly.test.integration.microprofile.openapi.OpenAPIMultiModuleDeploymentTestCase
[INFO] Running org.wildfly.test.integration.microprofile.openapi.OpenAPIDisabledTestCase
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.518 s - in org.wildfly.test.integration.microprofile.openapi.OpenAPIDisabledTestCase
[INFO] Running org.wildfly.test.integration.microprofile.openapi.OpenAPIFormatTestCase
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.651 s - in org.wildfly.test.integration.microprofile.openapi.OpenAPIFormatTestCase
[INFO] Running org.wildfly.test.integration.microprofile.openapi.OpenAPIAbsoluteServersTestCase
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.463 s - in org.wildfly.test.integration.microprofile.openapi.OpenAPIAbsoluteServersTestCase
[INFO] Running org.wildfly.test.integration.microprofile.openapi.OpenAPIAltPathTestCase
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.351 s - in org.wildfly.test.integration.microprofile.openapi.OpenAPIAltPathTestCase

Result: GREEN

fabiobrz commented 4 years ago

Re-run initial checks and all of the TSs against the feature branch

Wildfly MicroProfile tck

$ mvn clean test -Dts.layers -pl testsuite/integration/microprofile-tck/openapi -Dmaven.repo.local=local-repo
...
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running TestSuite
[INFO] Tests run: 230, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 40.5 s - in TestSuite
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 230, Failures: 0, Errors: 0, Skipped: 0

Result: GREEN

fabiobrz commented 4 years ago

Re-run initial checks and all of the TSs against the feature branch

EAP QE TS

$ mvn clean test -pl microprofile-open-api -Djboss.home=/home/fburzigo/projects/git/pferraro-precheck/wildfly/dist/target/wildfly-19.0.0.Beta1-SNAPSHOT -Pmp-open-api
...
[INFO] 
[INFO] Results:
[INFO] 
[WARNING] Tests run: 54, Failures: 0, Errors: 0, Skipped: 1

Result: GREEN

fabiobrz commented 4 years ago

Re-run initial checks and all of the TSs against the feature branch

Overall result is GREEN

fabiobrz commented 4 years ago

Perform some log observation

What reported by https://issues.redhat.com/projects/WFWIP/issues/WFWIP-291 is confirmed, WARN and INFO messages logged by io.smallrye.* do not have message IDs:

During a TS run:

$ tail -f server.log | grep io.smallrye.
...
2020-01-08 17:14:05,890 INFO  [io.smallrye.metrics] (MSC service thread 1-1) MicroProfile: Metrics activated (SmallRye Metrics version: 2.3.2)
2020-01-08 17:14:05,938 DEBUG [io.smallrye.openapi.runtime.scanner.OpenApiAnnotationScanner] (MSC service thread 1-6) Scanning deployment for OpenAPI and JAX-RS Annotations.
2020-01-08 17:14:05,938 DEBUG [io.smallrye.openapi.runtime.scanner.OpenApiAnnotationScanner] (MSC service thread 1-6) Processing a JAX-RS resource class: DistrictsResource
2020-01-08 17:14:05,939 DEBUG [io.smallrye.openapi.runtime.scanner.OpenApiAnnotationScanner] (MSC service thread 1-6) Processing jax-rs method: {0}
2020-01-08 17:14:05,939 DEBUG [io.smallrye.openapi.runtime.scanner.OpenApiAnnotationScanner] (MSC service thread 1-6) Processing a single @Response annotation.
2020-01-08 17:14:05,939 DEBUG [io.smallrye.openapi.runtime.scanner.OpenApiAnnotationScanner] (MSC service thread 1-6) Processing a single @Content annotation.
2020-01-08 17:14:05,939 DEBUG [io.smallrye.openapi.runtime.scanner.OpenApiAnnotationScanner] (MSC service thread 1-6) Processing a single @Content annotation as a MediaType.

Sources:

~/projects/git/smallrye/smallrye-open-api (master) $ find . -name '*.*' -type f -print0 | xargs -0 grep --color -n -e 'LOG.warn'
./implementation/src/main/java/io/smallrye/openapi/runtime/scanner/ParameterProcessor.java:551:                        LOG.warnf("Matrix parameter references missing path segment: %s", segmentName);
./implementation/src/main/java/io/smallrye/openapi/runtime/scanner/ParameterProcessor.java:1147:            LOG.warnf("Value '%s' is not a valid %s default", stringValue, primitive.name().toLowerCase());

This is major bug tracked via WFWIP (and possibly via a future WFLY "issue"). There is agreement for re-opening it.

So it isn't a blocker

fabiobrz commented 4 years ago

Execute manual experiments (try generate and use clients too, i.e. swagger editor) Done, staus is confirmed to be as described by https://github.com/fabiobrz/eap-microprofile-test-suite/issues/1#issuecomment-570638749.

Result is GREEN

fabiobrz commented 4 years ago

Finish looking at the Wildfly PR changes

Done. "High-level" check, I noticed one property name (mp.openapi.extensions.servers.relative) which is wrongly detailed in the AD with respect to sources (mp.openapi.extensions.relative). Added a comment to AD for this, see https://github.com/wildfly/wildfly-proposals/pull/251/files#r364349758.

Result is GREEN

fabiobrz commented 4 years ago

Execute memory test for multiple redeploy

Test failed, branch was rebased this morning, so the protocol is to be executed again.

Overall pre-check protocol result for commit 803a542bba53b30bacf4c0003d46838191c6afd3 is RED

fabiobrz commented 4 years ago

Another failure was noticed when executing the protocol again for commit b573c3a4c1a94146197c01b3216bc95b95202230 (rebased this morning to incorporate fix for Jaeger memory leak issue), as TCK run is failing Galleon provisioning. DEV was notified via Zulip and updated dependencies for it.

Overall pre-check protocol result for commit b573c3a4c1a94146197c01b3216bc95b95202230 is RED

fabiobrz commented 4 years ago

Pre-check protocol

Wildfly feature branch: https://github.com/pferraro/wildfly/tree/WFLY-12313

commit: 7c53fecadc5df6b8e5f0c2795bc8b9daf58055bd

EAP QE TS for MicroProfile version: https://github.com/fabiobrz/eap-microprofile-test-suite/tree/mp-openapi-non-tp-misc-advanced

fabiobrz commented 4 years ago

Re-run initial checks and all of the TSs against the feature branch

Widfly TS integration tests

$ mvn clean test -Dts.integration -pl testsuite/integration/microprofile -Dmaven.repo.local=/home/fburzigo/projects/git/pferraro-precheck/wildfly/local-repo
...
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.wildfly.test.integration.microprofile.faulttolerance.async.requestcontext.AsynchronousRequestContextTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.757 s - in org.wildfly.test.integration.microprofile.faulttolerance.async.requestcontext.AsynchronousRequestContextTest
[INFO] Running org.wildfly.test.integration.microprofile.faulttolerance.sync.SyncCircuitBreakerDisabledTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.871 s - in org.wildfly.test.integration.microprofile.faulttolerance.sync.SyncCircuitBreakerDisabledTest
[INFO] Running org.wildfly.test.integration.microprofile.openapi.OpenAPIMultiModuleDeploymentTestCase
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.737 s - in org.wildfly.test.integration.microprofile.openapi.OpenAPIMultiModuleDeploymentTestCase
[INFO] Running org.wildfly.test.integration.microprofile.openapi.OpenAPIDisabledTestCase
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.503 s - in org.wildfly.test.integration.microprofile.openapi.OpenAPIDisabledTestCase
[INFO] Running org.wildfly.test.integration.microprofile.openapi.OpenAPIFormatTestCase
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.56 s - in org.wildfly.test.integration.microprofile.openapi.OpenAPIFormatTestCase
[INFO] Running org.wildfly.test.integration.microprofile.openapi.OpenAPIAbsoluteServersTestCase
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.402 s - in org.wildfly.test.integration.microprofile.openapi.OpenAPIAbsoluteServersTestCase
[INFO] Running org.wildfly.test.integration.microprofile.openapi.OpenAPIAltPathTestCase
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.426 s - in org.wildfly.test.integration.microprofile.openapi.OpenAPIAltPathTestCase
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  25.438 s
[INFO] Finished at: 2020-01-09T08:33:14+01:00
[INFO] ------------------------------------------------------------------------

Result is GREEN

fabiobrz commented 4 years ago

Re-run initial checks and all of the TSs against the feature branch

Wildfly MicroProfile tck

$ mvn clean test -Dts.layers -pl testsuite/integration/microprofile-tck/openapi -Dmaven.repo.local=/home/fburzigo/projects/git/pferraro-precheck/wildfly/local-repo
...
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running TestSuite
[INFO] Tests run: 230, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 43.8 s - in TestSuite
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 230, Failures: 0, Errors: 0, Skipped: 0
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  56.884 s
[INFO] Finished at: 2020-01-09T08:34:27+01:00
[INFO] ------------------------------------------------------------------------

Result is GREEN

fabiobrz commented 4 years ago

Execute EAP QE TS for MicroProfile

$ mvn clean verify -pl microprofile-open-api -Djboss.home=/home/fburzigo/projects/git/pferraro-precheck/wildfly/dist/target/wildfly-19.0.0.Beta1-SNAPSHOT -Pmp-open-api
...

[INFO] 
[INFO] Results:
[INFO] 
[WARNING] Tests run: 54, Failures: 0, Errors: 0, Skipped: 1
[INFO] 
[INFO] 
[INFO] --- maven-jar-plugin:3.1.0:jar (default-jar) @ microprofile-open-api ---
[INFO] Building jar: /home/fburzigo/projects/git/jboss-eap-qe/eap-microprofile-test-suite/microprofile-open-api/target/microprofile-open-api-1.0.0.Final-SNAPSHOT.jar
[INFO] 
[INFO] --- maven-source-plugin:3.0.1:jar-no-fork (attach-sources) @ microprofile-open-api ---
[INFO] Building jar: /home/fburzigo/projects/git/jboss-eap-qe/eap-microprofile-test-suite/microprofile-open-api/target/microprofile-open-api-1.0.0.Final-SNAPSHOT-sources.jar
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  48.731 s
[INFO] Finished at: 2020-01-09T08:58:18+01:00
[INFO] ------------------------------------------------------------------------

Result is GREEN

fabiobrz commented 4 years ago

Re-run initial checks and all of the TSs against the feature branch

Overall result is GREEN

fabiobrz commented 4 years ago

Perform some log observation

What reported by https://issues.redhat.com/projects/WFWIP/issues/WFWIP-291 is confirmed, WARN and INFO messages logged by io.smallrye.* do not have message IDs:

< < DURING EAP QE TS for MicroProfile run > >

$ tail -f server.log | grep io.smallrye.
...
2020-01-09 08:58:12,220 DEBUG [io.smallrye.openapi.runtime.scanner.OpenApiDataObjectScanner] (MSC service thread 1-5) Starting processing with root: org.jboss.eap.qe.microprofile.openapi.apps.routing.router.model.DistrictObject
2020-01-09 08:58:12,220 DEBUG [io.smallrye.openapi.runtime.scanner.OpenApiDataObjectScanner] (MSC service thread 1-5) Getting all fields for: org.jboss.eap.qe.microprofile.openapi.apps.routing.router.model.DistrictObject in class: org.jboss.eap.qe.microprofile.openapi.apps.routing.router.model.DistrictObject
2020-01-09 08:58:12,323 INFO  [io.smallrye.metrics] (MSC service thread 1-6) MicroProfile: Metrics activated (SmallRye Metrics version: 2.3.2)
2020-01-09 08:58:12,490 INFO  [io.smallrye.metrics] (MSC service thread 1-7) MicroProfile: Metrics activated (SmallRye Metrics version: 2.3.2)
2020-01-09 08:58:15,360 INFO  [io.smallrye.metrics] (MSC service thread 1-6) MicroProfile: Metrics activated (SmallRye Metrics version: 2.3.2)
2020-01-09 08:58:15,422 DEBUG [io.smallrye.openapi.runtime.scanner.OpenApiAnnotationScanner] (MSC service thread 1-2) Scanning deployment for OpenAPI and JAX-RS Annotations.
2020-01-09 08:58:15,422 DEBUG [io.smallrye.openapi.runtime.scanner.OpenApiAnnotationScanner] (MSC service thread 1-2) Processing a JAX-RS resource class: Contact
2020-01-09 08:58:15,422 DEBUG [io.smallrye.openapi.runtime.scanner.OpenApiAnnotationScanner] (MSC service thread 1-2) Processing jax-rs method: {0}
2020-01-09 08:58:15,423 DEBUG [io.smallrye.openapi.runtime.scanner.OpenApiDataObjectScanner] (MSC service thread 1-2) Starting processing with root: java.lang.String
2020-01-09 08:58:15,423 DEBUG [io.smallrye.openapi.runtime.scanner.OpenApiDataObjectScanner] (MSC service thread 1-2) Starting processing with root: java.lang.String
2020-01-09 08:58:16,695 INFO  [io.smallrye.metrics] (MSC service thread 1-3) MicroProfile: Metrics activated (SmallRye Metrics version: 2.3.2)
2020-01-09 08:58:16,750 DEBUG [io.smallrye.openapi.runtime.scanner.OpenApiAnnotationScanner] (MSC service thread 1-1) Scanning deployment for OpenAPI and JAX-RS Annotations.
2020-01-09 08:58:16,751 DEBUG [io.smallrye.openapi.runtime.scanner.OpenApiAnnotationScanner] (MSC service thread 1-1) Processing a JAX-RS resource class: DistrictsResource
2020-01-09 08:58:16,751 DEBUG [io.smallrye.openapi.runtime.scanner.OpenApiAnnotationScanner] (MSC service thread 1-1) Processing jax-rs method: {0}
2020-01-09 08:58:16,751 DEBUG [io.smallrye.openapi.runtime.scanner.OpenApiAnnotationScanner] (MSC service thread 1-1) Processing a single @Response annotation.
2020-01-09 08:58:16,751 DEBUG [io.smallrye.openapi.runtime.scanner.OpenApiAnnotationScanner] (MSC service thread 1-1) Processing a single @Content annotation.
2020-01-09 08:58:16,751 DEBUG [io.smallrye.openapi.runtime.scanner.OpenApiAnnotationScanner] (MSC service thread 1-1) Processing a single @Content annotation as a MediaType.
2020-01-09 08:58:16,751 DEBUG [io.smallrye.openapi.runtime.util.SchemaFactory] (MSC service thread 1-1) Processing a single @Schema annotation.
...

Sources:

~/projects/git/smallrye/smallrye-open-api (master) $ find . -name '*.*' -type f -print0 | xargs -0 grep --color -n -e 'LOG.warn'
./implementation/src/main/java/io/smallrye/openapi/runtime/scanner/ParameterProcessor.java:551:                        LOG.warnf("Matrix parameter references missing path segment: %s", segmentName);
./implementation/src/main/java/io/smallrye/openapi/runtime/scanner/ParameterProcessor.java:1147:            LOG.warnf("Value '%s' is not a valid %s default", stringValue, primitive.name().toLowerCase());

This is major bug tracked via WFWIP - see [1] - (and possibly via a future WFLY "issue"). There is agreement for re-opening it.

So this isn't a blocker

[1] https://issues.redhat.com/projects/WFWIP/issues/WFWIP-291

fabiobrz commented 4 years ago

Execute manual experiments (try generate and use clients too, i.e. swagger editor) Done, staus is confirmed to be as described by https://github.com/fabiobrz/eap-microprofile-test-suite/issues/1#issuecomment-570638749.

Result is GREEN

fabiobrz commented 4 years ago

Execute memory test for multiple redeploy

~$ git clone git@gitlab.mw.lab.eng.bos.redhat.com:jbossqe-eap/tests-management.git
~$ cd tests-management
~$ mvn clean integration-test -Dit.test="UndeployMemoryLeakIT" -Djboss.home=/home/fburzigo/projects/git/pferraro-precheck/wildfly/dist/target/wildfly-19.0.0.Beta1-SNAPSHOT
...
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 43.623 s - in org.jboss.qa.management.web.UndeployMemoryLeakIT
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
[INFO] 
[INFO] 
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.jboss.qa.management.web.UndeployMemoryLeakIT
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
[INFO] 
[INFO] 
[INFO] --- maven-failsafe-plugin:2.22.1:integration-test (manual-testng) @ jbossqe-eap-tests-management-integration-tests ---
[INFO] 
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
[INFO] 
[INFO] 
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.jboss.qa.management.web.UndeployMemoryLeakIT
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.229 s - in org.jboss.qa.management.web.UndeployMemoryLeakIT
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
[INFO] 
[INFO] 
[INFO] --- maven-failsafe-plugin:2.22.1:integration-test (managed-testng) @ jbossqe-eap-tests-management-integration-tests ---
[INFO] 
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
[INFO] 
[INFO] 
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.jboss.qa.management.web.UndeployMemoryLeakIT
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.273 s - in org.jboss.qa.management.web.UndeployMemoryLeakIT
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for jbossqe-eap-tests-management-parent 4.0.0-SNAPSHOT:
[INFO] 
[INFO] jbossqe-eap-tests-management-parent ................ SUCCESS [  0.115 s]
[INFO] jbossqe-eap-tests-management-common ................ SUCCESS [  4.883 s]
[INFO] jbossqe-eap-tests-management-integration-tests ..... SUCCESS [ 59.875 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  01:05 min
[INFO] Finished at: 2020-01-09T09:54:43+01:00
[INFO] ------------------------------------------------------------------------

Result is GREEN

fabiobrz commented 4 years ago

RESULTS of pre-check protocol

Wildfly feature branch: https://github.com/pferraro/wildfly/tree/WFLY-12313

commit: 7c53fecadc5df6b8e5f0c2795bc8b9daf58055bd

EAP QE TS for MicroProfile version: https://github.com/fabiobrz/eap-microprofile-test-suite/tree/mp-openapi-non-tp-misc-advanced

Re-run initial checks and all of the TSs against the feature branch: GREEN
Perform some log observation: no blockers https://issues.redhat.com/projects/WFWIP/issues/WFWIP-291 will be used to track INFO/WARN messages with missing message ID logged by io.smallrye.* until a RFE or WFLY Jira will be filed
Execute manual experiments: GREEN
Execute memory test for multiple redeploy: GREEN
Finish looking at the Wildfly PR changes Done. "High-level" check, I noticed one property name (mp.openapi.extensions.servers.relative) which is wrongly detailed in the AD with respect to sources (mp.openapi.extensions.relative). Added a comment to AD for this, see https://github.com/wildfly/wildfly-proposals/pull/251/files#r364349758. DEV fixed, no blockers

fabiobrz commented 4 years ago

Closing as done, see https://github.com/wildfly/wildfly/pull/12852#issuecomment-572489497

fabiobrz / eap-microprofile-test-suite