Prepare new Faces TCK - Githubissues

arjantijms commented 2 years ago

Prepare a new Faces TCK by

[x] Copying https://github.com/eclipse-ee4j/mojarra/tree/master/test2 to /tck in this project.
[x] Changing package names to ee.jakarta.tck
Adding comments to test in https://github.com/jakartaee/faces/tree/master/tck (see https://github.com/eclipse-ee4j/mojarra/issues/5046)
- [x] Faces22
- [x] Faces23 - Bauke/BalusC
- [x] Faces40
[ ] Move additional tests from https://github.com/eclipse-ee4j/mojarra/tree/3.0/test
[ ] ~Where needed, add assertion IDs~ (on hold, nobody understands this) (see https://github.com/jakartaee/faces/issues/1600#issuecomment-1029324422)
[ ] Incorporating the existing TCK from https://github.com/eclipse-ee4j/jakartaee-tck and disabling whatever doesn't work anymore (?) (For example, see https://github.com/arjantijms/glassfish/blob/master/appserver/tests/tck/faces/pom.xml)
[x] Creating single zip distribution of the entire TCK
[ ] Create / update guide / docs on how to run the TCK
[ ] Adding example to run TCK as dependency from Maven into your own pom. (see https://github.com/piranhacloud/piranha/tree/current/external/tck/jwt)
- [ ] Arjan

cc @BalusC @scottmarlow @mriem

BalusC commented 2 years ago

Gotcha, on it now. Progress so far: https://github.com/eclipse-ee4j/jakartaee-tck/compare/master...BalusC:migrate_jsp_to_xhtml_in_faces_tck?expand=1

Now only yet those <managed-bean> entries. They were not trivial to find&replace. That'll probably have to be done by hand. Will continue tomorrow or later.

alwin-joseph commented 2 years ago

Gotcha, on it now. Progress so far: https://github.com/eclipse-ee4j/jakartaee-tck/compare/master...BalusC:migrate_jsp_to_xhtml_in_faces_tck?expand=1

Now only yet those <managed-bean> entries. They were not trivial to find&replace. That'll probably have to be done by hand. Will continue tomorrow or later.

Thanks. Can the managed-bean changes be shared ? Please let me know when I can run the tests with Glassfish to get latest status of tests, post your changes.

BalusC commented 2 years ago

I've migrated well over half of of those entries in https://github.com/eclipse-ee4j/jakartaee-tck/commit/22abd19f1a6127de05de3d177b45b88984b3ccff until I stumbled upon a ManagedPropertyBean which I don't know what to do with. They can either be removed or be reworked to test the new @ManagedProperty instead, @alwin-joseph feel free to continue from here. I guess removing is easier for now as new tests can always be added later.

alwin-joseph commented 2 years ago

While we continue to work on this can I raise an issue happening in the server logs before the faces tck(from jakartaee-tck) test execution. We are facing the below error while when the archives are being deployed before the faces tests(from jakartaee-tck) are run against latest glassfish nightly build. I presume this is causing most tests to fail at the moment.

Does anyone have hints on why this is happening or possible solution.

  Critical error during deployment:
java.lang.IllegalStateException: CDI is not available
        at com.sun.faces.util.Util.getCdiBeanManager(Util.java:1493)
        at com.sun.faces.el.ELUtils.addCDIELResolver(ELUtils.java:192)
        at com.sun.faces.el.ELUtils.buildFacesResolver(ELUtils.java:164)
        at com.sun.faces.application.ApplicationAssociate.initializeELResolverChains(ApplicationAssociate.java:416)
        at com.sun.faces.application.applicationimpl.ExpressionLanguage.performOneTimeELInitialization(ExpressionLanguage.java:144)
        at com.sun.faces.application.applicationimpl.ExpressionLanguage.getELResolver(ExpressionLanguage.java:89)
        at com.sun.faces.application.ApplicationImpl.getELResolver(ApplicationImpl.java:200)
        at com.sun.faces.el.ELContextImpl.<init>(ELContextImpl.java:60)
        at com.sun.faces.config.ConfigureListener.contextInitialized(ConfigureListener.java:202)
        at org.apache.catalina.core.StandardContext.contextListenerStart(StandardContext.java:5023)
        at com.sun.enterprise.web.WebModule.contextListenerStart(WebModule.java:563)
        at org.apache.catalina.core.StandardContext.start(StandardContext.java:5592)
        at com.sun.enterprise.web.WebModule.start(WebModule.java:511)
        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:877)
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:860)
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:644)
        at com.sun.enterprise.web.WebContainer.loadWebModule(WebContainer.java:2021)
        at com.sun.enterprise.web.WebContainer.loadWebModule(WebContainer.java:1667)
        at com.sun.enterprise.web.WebApplication.start(WebApplication.java:82)
        at org.glassfish.internal.data.EngineRef.start(EngineRef.java:98)
        at org.glassfish.internal.data.ModuleInfo.start(ModuleInfo.java:267)
        at org.glassfish.internal.data.ApplicationInfo.start(ApplicationInfo.java:328)
        at com.sun.enterprise.v3.server.ApplicationLifecycle.deploy(ApplicationLifecycle.java:480)
        at com.sun.enterprise.v3.server.ApplicationLifecycle.deploy(ApplicationLifecycle.java:199)
        at org.glassfish.deployment.admin.DeployCommand.execute(DeployCommand.java:467)
        at com.sun.enterprise.v3.admin.CommandRunnerImpl$2$1.run(CommandRunnerImpl.java:516)
        at com.sun.enterprise.v3.admin.CommandRunnerImpl$2$1.run(CommandRunnerImpl.java:512)
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at java.base/javax.security.auth.Subject.doAs(Subject.java:361)
        at com.sun.enterprise.v3.admin.CommandRunnerImpl$2.execute(CommandRunnerImpl.java:511)
        at com.sun.enterprise.v3.admin.CommandRunnerImpl$3.run(CommandRunnerImpl.java:542)
        at com.sun.enterprise.v3.admin.CommandRunnerImpl$3.run(CommandRunnerImpl.java:534)
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at java.base/javax.security.auth.Subject.doAs(Subject.java:361)
        at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:533)
        at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:1441)
        at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1823)
        at org.glassfish.deployment.autodeploy.AutoOperation.run(AutoOperation.java:140)
        at org.glassfish.deployment.autodeploy.AutoDeployer.deploy(AutoDeployer.java:573)
        at org.glassfish.deployment.autodeploy.AutoDeployer.deployAll(AutoDeployer.java:460)
        at org.glassfish.deployment.autodeploy.AutoDeployer.run(AutoDeployer.java:388)
        at org.glassfish.deployment.autodeploy.AutoDeployer.run(AutoDeployer.java:379)
        at org.glassfish.deployment.autodeploy.AutoDeployService$1.run(AutoDeployService.java:209)
        at java.base/java.util.TimerThread.mainLoop(Timer.java:556)
        at java.base/java.util.TimerThread.run(Timer.java:506)

BalusC commented 2 years ago

Start by creating an empty WEB-INF/beans.xml file in WAR.

BalusC commented 2 years ago

@alwin-joseph can you let me know where you're at now? Have you been able to continue where I left? And if so where can I find the progress?

alwin-joseph commented 2 years ago

@alwin-joseph can you let me know where you're at now? Have you been able to continue where I left? And if so where can I find the progress?

I created a PR on top of your changes https://github.com/BalusC/jakartaee-tck/pull/1. Details in the PR.

alwin-joseph commented 2 years ago

@alwin-joseph can you let me know where you're at now? Have you been able to continue where I left? And if so where can I find the progress?

I created a PR on top of your changes BalusC/jakartaee-tck#1. Details in the PR.

Another one https://github.com/BalusC/jakartaee-tck/pull/3 .

Pending deployement errors are of type Normal scoped managed bean implementation class has a public field. Any known solution for this ?

arjantijms commented 2 years ago

Any known solution for this ?

Without looking at the exact code, the solution is typically to make the field private. Normal scoped beans are proxied beans, so the public field on the proxy would not correspond to the public field on the actual bean. Hence CDI forbids it.

alwin-joseph commented 2 years ago

Most(maybe all) of the test failures in jakartaee-tck faces tests when run against Glassfish 7 nightly bundle are due to below:

com.sun.faces.application.view.ViewHandlingStrategyNotFoundException
        at java.base/java.util.Optional.orElseThrow(Optional.java:408)
        at com.sun.faces.application.view.ViewHandlingStrategyManager.getStrategy(ViewHandlingStrategyManager.java:57)
        at com.sun.faces.application.view.ViewDeclarationLanguageFactoryImpl.getViewDeclarationLanguage(ViewDeclarationLanguageFactoryImpl.java:45)
        at com.sun.faces.application.view.MultiViewHandler.createView(MultiViewHandler.java:145)
        at jakarta.faces.application.ViewHandlerWrapper.createView(ViewHandlerWrapper.java:110)
        at com.sun.ts.tests.jsf.common.servlets.HttpTCKServlet.createViewRoot(HttpTCKServlet.java:189)
        at com.sun.ts.tests.jsf.common.servlets.HttpTCKServlet.createViewRoot(HttpTCKServlet.java:195)
        at com.sun.ts.tests.jsf.common.servlets.HttpTCKServlet.initFaces(HttpTCKServlet.java:221)
        at com.sun.ts.tests.jsf.common.servlets.HttpTCKServlet.invokeTest(HttpTCKServlet.java:144)
        at com.sun.ts.tests.jsf.common.servlets.HttpTCKServlet.doGet(HttpTCKServlet.java:104)

I see a similar issue https://github.com/omnifaces/omnifaces/issues/649 was raised and fixed. Should mojarra also need this to be addressed or does the TCK code need any change at https://github.com/eclipse-ee4j/jakartaee-tck/blob/f08f807fcb9820320eda9b9b4bd95a03118dfd1c/src/com/sun/ts/tests/jsf/common/servlets/HttpTCKServlet.java#L189 where viewId is null ?

alwin-joseph commented 2 years ago

I have created a PR https://github.com/jakartaee/faces/pull/1644 that will migrate the userguide and other documentation from old style TCK with major updates required for the new TCK. Please feel free to review.

BalusC commented 2 years ago

@alwin-joseph

Most(maybe all) of the test failures in jakartaee-tck faces tests when run against Glassfish 7 nightly bundle are due to below

I guess you just need to replace line 221 of jsf/common/servlets/HttpTCKServlet

facesContext.setViewRoot(createViewRoot());

to

facesContext.setViewRoot(new UIViewRoot());

Please try and let me know.

Or perhaps more globally, depending on the rest of the tests, rewrite this method

  protected UIViewRoot createViewRoot() {

    return createViewRoot(null);

  }

to

  protected UIViewRoot createViewRoot() {

    return new UIViewRoot();

  }

alwin-joseph commented 2 years ago

Most(maybe all) of the test failures in jakartaee-tck faces tests when run against Glassfish 7 nightly bundle are due to below

Or perhaps more globally, depending on the rest of the tests, rewrite this method
  protected UIViewRoot createViewRoot() {

    return createViewRoot(null);

  }
to
  protected UIViewRoot createViewRoot() {

    return new UIViewRoot();

  }

Thanks, this helped !

Test run status with this change : Number of tests completed: 5430 (4429 passed, 1001 failed, 0 with errors)

Some of the remaining test failures are due to exceptions of the below type (from server logs):

1.

java.lang.NullPointerException
    at jakarta.faces.component.UIComponentBase.getRenderer(UIComponentBase.java:1047)
    at jakarta.faces.component.UIComponentBase.encodeBegin(UIComponentBase.java:529)
    at com.sun.ts.tests.jsf.api.jakarta_faces.component.common.BaseUIComponentTestServlet.uiComponentEncodeBeginTest(BaseUIComponentTestServlet.java:3722)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at com.sun.ts.tests.jsf.common.servlets.HttpTCKServlet.invokeTest(HttpTCKServlet.java:163)
    at com.sun.ts.tests.jsf.common.servlets.HttpTCKServlet.doGet(HttpTCKServlet.java:104)

2.

com.sun.faces.application.view.ViewHandlingStrategyNotFoundException
        at java.base/java.util.Optional.orElseThrow(Optional.java:408)
        at com.sun.faces.application.view.ViewHandlingStrategyManager.getStrategy(ViewHandlingStrategyManager.java:57)
        at com.sun.faces.application.view.ViewDeclarationLanguageFactoryImpl.getViewDeclarationLanguage(ViewDeclarationLanguageFactoryImpl.java:45)
        at com.sun.faces.application.view.MultiViewHandler.createView(MultiViewHandler.java:145)
        at jakarta.faces.application.ViewHandlerWrapper.createView(ViewHandlerWrapper.java:110)
        at jakarta.faces.application.ViewHandlerWrapper.createView(ViewHandlerWrapper.java:110)
        at com.sun.ts.tests.jsf.api.jakarta_faces.application.viewhandlerwrapper.TestServlet.viewHandlerCreateViewTest(TestServlet.java:105)

3.

StandardWrapperValve[TestServlet]: Servlet.service() for servlet TestServlet threw exception
      java.lang.NullPointerException
              at jakarta.faces.component.UIComponentBase.getRenderer(UIComponentBase.java:1047)
              at jakarta.faces.component.UIComponentBase.decode(UIComponentBase.java:498)
              at jakarta.faces.component.UIComponentBase.processDecodes(UIComponentBase.java:872)
              at jakarta.faces.component.UIComponentBase.processDecodes(UIComponentBase.java:867)
              at jakarta.faces.component.UIViewRoot.processDecodes(UIViewRoot.java:997)
              at com.sun.ts.tests.jsf.api.jakarta_faces.component.behavior.ajax.TestServlet.ajaxBehaviorBroadcastTest(TestServlet.java:87)

java.lang.InstantiationException: org.jboss.weld.module.jsf.ConversationAwareViewHandler
  at java.base/java.lang.Class.newInstance(Class.java:571)
  at com.sun.ts.tests.jsf.common.util.JSFTestUtil.checkForNPE(JSFTestUtil.java:440)
  at com.sun.ts.tests.jsf.api.jakarta_faces.application.viewhandler.TestServlet.viewHandlerCalculateLocaleNPETest(TestServlet.java:264)
...
Caused by: java.lang.NoSuchMethodException: org.jboss.weld.module.jsf.ConversationAwareViewHandler.<init>()

StandardWrapperValve[TestServlet]: Servlet.service() for servlet TestServlet threw exception
java.lang.NullPointerException: Argument Error: Parameter renderKitId is null
  at com.sun.faces.renderkit.RenderKitFactoryImpl.getRenderKit(RenderKitFactoryImpl.java:78)
  at com.sun.ts.tests.jsf.api.jakarta_faces.component.common.BaseUIComponentTestServlet.uiComponentGetRendersChildrenTest(BaseUIComponentTestServlet.java:1997)
  at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.base/java.lang.reflect.Method.invoke(Method.java:566)
  at com.sun.ts.tests.jsf.common.servlets.HttpTCKServlet.invokeTest(HttpTCKServlet.java:163)

alwin-joseph commented 2 years ago

Prepare a new Faces TCK by
6. [ ]  Incorporating the existing TCK from https://github.com/eclipse-ee4j/jakartaee-tck and disabling whatever doesn't work anymore (?)
   (For example, see https://github.com/arjantijms/glassfish/blob/master/appserver/tests/tck/faces/pom.xml)
Hi @arjantijms Can you specify how to run the old-style faces tck from https://github.com/eclipse-ee4j/glassfish/blob/master/appserver/tests/tck/faces/pom.xml. What would be the command/parameters to download and execute the tck against glassfish. I tried mvn verify -Dant.version=1.10.11 after downloading tck using tck-download/jakarta-faces-tck but it does not start tests for me.

IMO we should include the old tck source(maybe the bundle as such) within the new tck bundle.

arjantijms commented 2 years ago

@alwin-joseph just "mvn clean install" should work. If you copied that pom directly, did you uncomment the part where the faces TCK is unpacked? The part here: https://github.com/arjantijms/glassfish/blob/master/appserver/tests/tck/faces/pom.xml#L114

arjantijms commented 2 years ago

@alwin-joseph @BalusC what is the current status?

This one shows 1001 failures for Faces still. Can we look into what's causing those? https://github.com/eclipse-ee4j/jakartaee-tck/wiki/Jakarta-EE-10.0-TCK-results

alwin-joseph commented 2 years ago

@alwin-joseph @BalusC what is the current status?

This one shows 1001 failures for Faces still. Can we look into what's causing those? https://github.com/eclipse-ee4j/jakartaee-tck/wiki/Jakarta-EE-10.0-TCK-results

These failures are tracked at https://github.com/eclipse-ee4j/jakartaee-tck/issues/913. Pasted the exceptions from server logs and uploaded the server logs too. @BalusC @kito99 If you are aware of the fix for any of the exceptions at https://github.com/eclipse-ee4j/jakartaee-tck/issues/913#issuecomment-1082319762 , please suggest.

alwin-joseph commented 2 years ago

6. [ ]  Incorporating the existing TCK from https://github.com/eclipse-ee4j/jakartaee-tck and disabling whatever doesn't work anymore (?)
   (For example, see https://github.com/arjantijms/glassfish/blob/master/appserver/tests/tck/faces/pom.xml)
Would it be better if we move the faces tests from https://github.com/eclipse-ee4j/jakartaee-tck by retaining the current setup(without converting to junit or maven). We could create a new folder as old-tck and have all the files(including the lib/jars) required to build and run the old style tck.

Otherwise the new TCK will always be depended on the jakartaee-tck project to produce the old tck bundle and the final TCK will have sources distributed in multiple repositories. It was suggested that this could be difficult especially when we have TCK challenges for tests in old style tck.

arjantijms commented 2 years ago

We could create a new folder as old-tck and have all the files(including the lib/jars) required to build and run the old style tck.

That sounds like a pretty good idea. I proposed the other approach mostly because I thought it would be easier/faster to do, but having the actual files here and not have that dependency would indeed be better.

alwin-joseph commented 2 years ago

We could create a new folder as old-tck and have all the files(including the lib/jars) required to build and run the old style tck.

That sounds like a pretty good idea. I proposed the other approach mostly because I thought it would be easier/faster to do, but having the actual files here and not have that dependency would indeed be better.

Created PR https://github.com/jakartaee/faces/pull/1649 for migrating the faces test sources from jakartaee-tck.

arjantijms commented 2 years ago

@kito99 sorry to bother again, just wondering if you already have been able to find some time with helping to fix the outstanding TCK issues. We're done to 1001 failures, and every fix helps.

arjantijms commented 2 years ago

@alwin-joseph I just merged #1649

@BalusC can you look to see if you can get the old TCK building / running?

arjantijms commented 2 years ago

@tandraschko do you have time to try to run the TCK (old and new) against MyFaces?

alwin-joseph commented 2 years ago

It would be great if we can generate the new TCK bundle via jenkins job(in faces Jenkins CI) and make it available at a download location so we can review the structure/contents and make amends if required.

arjantijms commented 2 years ago

@alwin-joseph Indeed, that should be doable for sure.

@BalusC @tandraschko @kito99 @mnriem Did any of you succeeded in building the old TCK as transferred to this project? Not talking about fixing or running anything yet, just trying to build it. Nothing more. Hopefully just a single person has this 10 to 20 minutes to spare.

arjantijms commented 2 years ago

It's quite unclear how to build the old TCK, but this one gives an example for expresion language:

https://github.com/eclipse-ee4j/jakartaee-tck/pull/933#issuecomment-1099537995

I'll copy it here:

export ANT_HOME=/opt/apache-ant-1.9.12
export M2_HOME=/opt/apache-maven-3.6.3
export GF_BUNDLE_URL=https://download.eclipse.org/ee4j/glassfish/glassfish-7.0.0-SNAPSHOT-nightly.zip
export JDK11_HOME=/opt/java/temurin-jdk-11-latest

docker/build_standalone-tcks.sh el

Example of installing the TCK after building it:

https://github.com/eclipse-ee4j/glassfish/blob/master/appserver/tests/tck/tck-download/jakarta-faces-tck/pom.xml#L69

Example of running the TCK:

https://github.com/eclipse-ee4j/glassfish/blob/master/appserver/tests/tck/pages/pom.xml

cc @dmatej

alwin-joseph commented 2 years ago

It's quite unclear how to build the old TCK, but this one gives an example for expresion language:

eclipse-ee4j/jakartaee-tck#933 (comment)

I'll copy it here:
export ANT_HOME=/opt/apache-ant-1.9.12
export M2_HOME=/opt/apache-maven-3.6.3
export GF_BUNDLE_URL=https://download.eclipse.org/ee4j/glassfish/glassfish-7.0.0-SNAPSHOT-nightly.zip
export JDK11_HOME=/opt/java/temurin-jdk-11-latest

docker/build_standalone-tcks.sh el
Example of installing the TCK after building it:

https://github.com/eclipse-ee4j/glassfish/blob/master/appserver/tests/tck/tck-download/jakarta-faces-tck/pom.xml#L69

Example of running the TCK:

https://github.com/eclipse-ee4j/glassfish/blob/master/appserver/tests/tck/pages/pom.xml

cc @dmatej

mvn clean package using tck/pom.xml should build the old tck & new tests and create the final bundle at tck/target. The old tck build script is integrated to the tck pom at https://github.com/jakartaee/faces/blob/master/tck/pom.xml#L378.

mvn clean verify using tck/faces-tck/pom.xml will build and run the tests in old tck alone.

arjantijms commented 2 years ago

mvn clean verify using tck/faces-tck/pom.xml will build and run the tests in old tck alone.

Sounds great Alwin, thanks!

Now maybe the only thing needed there is the ability to deploy and run a single test. I added that for Pages here:

https://github.com/eclipse-ee4j/glassfish/blob/master/appserver/tests/tck/pages/pom.xml#L261

and here:

https://github.com/eclipse-ee4j/glassfish/blob/master/appserver/tests/tck/pages/pom.xml#L295

@BalusC with that new info from Alwin, can you continue on?

scottmarlow commented 2 years ago

Will https://ci.eclipse.org/faces be used to build the new Faces TCK? Have any implementations started to create any public CI jobs for running the Faces TCK?

arjantijms commented 2 years ago

@scottmarlow not yet, I'm mostly waiting for some more of the old tests to be fixed. But I understand working with the (old) TCK is not so easy for everyone, even if those people could help with the Faces work itself.

volosied commented 2 years ago

Hi, I'd like to help out with Faces and the TCK here. I'm trying to follow this conversation here. Are we trying to fix the old tck and then move these tests over to the new Faces TCK?

Can anyone give me any tasks? (@arjantijms) Thanks!

Update: I'm trying to find the current failures via

git clone https://github.com/jakartaee/faces.git
cd faces/tck/faces-tck ; mvn clean verify

arjantijms commented 2 years ago

Hi, I'd like to help out with Faces and the TCK here.

Thanks a bunch, much appreciated!

Can anyone give me any tasks? (@arjantijms) Thanks!

The first task is to simply build and run the old TCK. Almost nobody (AFAIK) has been able to do that. It's perhaps a trivial task, but someone needs to do it.

There were 1001 failures when last time executed when still in the platform TCK, though typically it aren't 1001 unique failures. So the next obvious task would be to start reducing those failures.

A second (simultaneous) task would be to make it possible to run a single test, like e.g. can be done here: https://github.com/eclipse-ee4j/glassfish/blob/master/appserver/tests/tck/pages/pom.xml

alwin-joseph commented 2 years ago

@arjantijms I have created a CI job that builds the faces tck at https://ci.eclipse.org/jakartaee-tck/job/10/job/jakarta-faces-tck-build It requires PR https://github.com/jakartaee/faces/pull/1651 that includes some improvements for the tck build process.

TCK bundle is available at https://download.eclipse.org/ee4j/jakartaee-tck/jakartaee10/staged/eftl/jakarta-faces-tck-4.0.0-SNAPSHOT.zip

volosied commented 2 years ago

PR #1653 should address 1, 3, and 5 in https://github.com/jakartaee/faces/issues/1600#issuecomment-1073306615

alwin-joseph commented 2 years ago

Below are the mvn targets that can be used for new tck build/run/generation. Can someone help validate and help improve these.

1. To build and run the new tck :

cd tck/
mvn clean verify

This will first start building the old tck hence takes longer time to start running the new tests.

2. To build and run the old tck alone :

cd tck/faces-tck
mvn clean verify

3. To generate the new TCK bundle :

cd tck/
mvn clean package

The job https://ci.eclipse.org/jakartaee-tck/job/10/job/jakarta-faces-tck-build/ does the same, copies the bundle at https://download.eclipse.org/ee4j/jakartaee-tck/jakartaee10/staged/eftl/jakarta-faces-tck-4.0.0-SNAPSHOT.zip.

4. TCK run using the bundle:

unzip jakarta-faces-tck-4.0.0-SNAPSHOT.zip
cd jakarta-faces-tck-4.0.0-SNAPSHOT

a. Run old tck tests (The https://github.com/jakartaee/faces/blob/master/tck/faces-tck/pom-bundle.xml is renamed as pom.xml while building the tck and copied to the bundle):
```
cd faces-tck
mvn clean verify
```
b. Run new tests/all tests :
```
cd jakarta-faces-tck-4.0.0-SNAPSHOT
mvn clean verify
```
Currently this is broken due to 2 reasons :
- The tck/pom.xml has parent pom details.
- The tck/pom.xml has build-old-tck, build-userguide and packaging phase using maven-assembly-plugin for tck build. These 3 executions should not be executed or disabled in the tck bundle pom.xml. Is there a better solution than to have multiple pom files for repository and bundle build/run?

alwin-joseph commented 2 years ago

@volosied @arjantijms The latest test result of old tck run from repository :

     [exec] [javatest.batch] Completed running 5425 tests.
     [exec] [javatest.batch] Number of Tests Passed      = 5395
     [exec] [javatest.batch] Number of Tests Failed      = 30
     [exec] [javatest.batch] Number of Tests with Errors = 0

https://ci.eclipse.org/jakartaee-tck/job/10/job/jakarta-faces-tck-build-run/14/consoleText

volosied commented 2 years ago

Remaining failures:

     [exec] [javatest.batch] FAILED........com/sun/ts/tests/jsf/api/jakarta_faces/application/resource/URLClient.java#resourceGetRequestPathLibTest
     [exec] [javatest.batch] FAILED........com/sun/ts/tests/jsf/api/jakarta_faces/application/resource/URLClient.java#resourceGetRequestPathTest
     [exec] [javatest.batch] FAILED........com/sun/ts/tests/jsf/api/jakarta_faces/application/resource/URLClient.java#resourceToStringTest
     [exec] [javatest.batch] FAILED........com/sun/ts/tests/jsf/api/jakarta_faces/application/resourcewrapper/URLClient.java#resourceWrapperGetRequestPathLibTest
     [exec] [javatest.batch] FAILED........com/sun/ts/tests/jsf/api/jakarta_faces/application/resourcewrapper/URLClient.java#resourceWrapperGetRequestPathTest
     [exec] [javatest.batch] FAILED........com/sun/ts/tests/jsf/spec/ajax/jsresource/URLClient.java#ajaxPDLResourceTest
     [exec] [javatest.batch] FAILED........com/sun/ts/tests/jsf/spec/composite/actionsource/URLClient.java#compositeActionSourceTagTest
     [exec] [javatest.batch] FAILED........com/sun/ts/tests/jsf/spec/composite/attribute/URLClient.java#compositeAttributeTagTest
     [exec] [javatest.batch] FAILED........com/sun/ts/tests/jsf/spec/composite/editablevalueholder/URLClient.java#compositeEditableValueHolderTest
     [exec] [javatest.batch] FAILED........com/sun/ts/tests/jsf/spec/composite/facet/URLClient.java#compositeFacetsTagTest
     [exec] [javatest.batch] FAILED........com/sun/ts/tests/jsf/spec/composite/packaging/classpath/URLClient.java#compositeClasspathPKGTest
     [exec] [javatest.batch] FAILED........com/sun/ts/tests/jsf/spec/composite/packaging/webapproot/URLClient.java#compositeWebAppRootPKGTest
     [exec] [javatest.batch] FAILED........com/sun/ts/tests/jsf/spec/composite/valueholder/URLClient.java#compositeValueHolderTest
     [exec] [javatest.batch] FAILED........com/sun/ts/tests/jsf/spec/el/elresolvers/URLClient.java#facesImplicitObjectResolverFeatureDescriptorTest
     [exec] [javatest.batch] FAILED........com/sun/ts/tests/jsf/spec/el/elresolvers/URLClient.java#facesImplicitObjectResolverGetTypeTest
     [exec] [javatest.batch] FAILED........com/sun/ts/tests/jsf/spec/el/elresolvers/URLClient.java#facesImplicitObjectResolverIsReadOnlyTest
     [exec] [javatest.batch] FAILED........com/sun/ts/tests/jsf/spec/el/elresolvers/URLClient.java#facesImplicitObjectResolverSetValueTest
     [exec] [javatest.batch] FAILED........com/sun/ts/tests/jsf/spec/el/elresolvers/URLClient.java#facesManagedBeanResolverGetValueTest
     [exec] [javatest.batch] FAILED........com/sun/ts/tests/jsf/spec/el/elresolvers/URLClient.java#facesResourceBundleResolverFeatureDescriptorTest
     [exec] [javatest.batch] FAILED........com/sun/ts/tests/jsf/spec/el/elresolvers/URLClient.java#facesScopedAttributeResolverFeatureDescriptorTest
     [exec] [javatest.batch] FAILED........com/sun/ts/tests/jsf/spec/el/elresolvers/URLClient.java#managedBeanELResolverFeatureDescriptorTest
     [exec] [javatest.batch] FAILED........com/sun/ts/tests/jsf/spec/flows/basicmethodcall/URLClient.java#facesFlowBasicMethodCallTest
     [exec] [javatest.batch] FAILED........com/sun/ts/tests/jsf/spec/flows/basicmultipage/URLClient.java#facesFlowScopeTest
     [exec] [javatest.batch] FAILED........com/sun/ts/tests/jsf/spec/flows/multipagewebinf/URLClient.java#facesFlowWebInfScopeTest
     [exec] [javatest.batch] FAILED........com/sun/ts/tests/jsf/spec/render/commandbutton/URLClient.java#cbuttonRenderPassthroughTest
     [exec] [javatest.batch] FAILED........com/sun/ts/tests/jsf/spec/render/messages/URLClient.java#messagesRenderEncodeTest
     [exec] [javatest.batch] FAILED........com/sun/ts/tests/jsf/spec/render/onelistbox/URLClient.java#oneListboxRenderDecodeTest
     [exec] [javatest.batch] FAILED........com/sun/ts/tests/jsf/spec/render/outputlink/URLClient.java#outputLinkRenderPassthroughTest
     [exec] [javatest.batch] FAILED........com/sun/ts/tests/jsf/spec/resource/packaging/classpath/URLClient.java#jsfJsDoesExistTest
     [exec] [javatest.batch] FAILED........com/sun/ts/tests/jsf/spec/webapp/tldsig/URLClient.java#jsfTldSignatureTest

I will look at updating the application/resource & application/resourcewrapper test next via #1663. Tests will need to be updated to expect the correct URLs.

I think the elresolvers tests should also be looked at next.

arjantijms commented 2 years ago

@volosied @BalusC @tandraschko @alwin-joseph

As our time is essentially up, and Faces needs to go in ballot tomorrow, I think we'd best start to pack-up what we have now, excluding the small list of tests above. Perhaps we can do a maintenance update including the final tests shortly after EE 10?

alwin-joseph commented 2 years ago

As our time is essentially up, and Faces needs to go in ballot tomorrow, I think we'd best start to pack-up what we have now, excluding the small list of tests above. Perhaps we can do a maintenance update including the final tests shortly after EE 10?

Created https://github.com/jakartaee/faces/pull/1666 to exclude the remaining 25 tests that fails.

IMO the TCK userguide need to be reviewed/updated and final TCK packaging and structure need to be revisited as I had faced below issue earlier.

* The tck/pom.xml has build-old-tck, build-userguide and packaging phase using maven-assembly-plugin for tck build. These 3 executions should not be executed or disabled in the tck bundle pom.xml.

Will the TCK build and run jobs be created in faces Jenkins?

arjantijms commented 2 years ago

Will the TCK build and run jobs be created in faces Jenkins?

Yes, I've done this for Security al ready, and the plan is to do the same for Faces and Authentication.

For Security, see this job:

https://ci.eclipse.org/es/job/2_security-run-tck-against-staged-build/

#!/usr/bin/env groovy

node {

    def API_JAR_NAME = "jakarta.security.enterprise-api.jar"
    def DOWNLOAD_API_JAR_NAME = "jakarta.security.enterprise-api-2.0.0.jar"
    def API_URL = "https://jakarta.oss.sonatype.org/content/repositories/staging/jakarta/security/enterprise/jakarta.security.enterprise-api/2.0.0/${DOWNLOAD_API_JAR_NAME}"

    def IMPL_JAR_NAME = "jakarta.security.enterprise.jar"
    def DOWNLOAD_IMPL_JAR_NAME = "jakarta.security.enterprise-2.0.1.jar"
    def IMPL_URL = "https://jakarta.oss.sonatype.org/content/repositories/staging/org/glassfish/soteria/jakarta.security.enterprise/2.0.1/${DOWNLOAD_IMPL_JAR_NAME}"

    def IMPL_JAR_NAME2 = "soteria.spi.bean.decorator.weld.jar"
    def DOWNLOAD_IMPL_JAR_NAME2 = "soteria.spi.bean.decorator.weld-2.0.1.jar"
    def IMPL_URL2 = "https://jakarta.oss.sonatype.org/content/repositories/staging/org/glassfish/soteria/soteria.spi.bean.decorator.weld/2.0.1/${DOWNLOAD_IMPL_JAR_NAME2}"

    def GF_URL = "https://ci.eclipse.org/glassfish/view/GlassFish/job/glassfish_build-and-test-using-jenkinsfile/job/master/lastSuccessfulBuild/artifact/bundles/glassfish.zip"

    def TCK_BUNDLE_NAME = "jakarta-security-tck-3.0.0"
    def TCK_BUNDLE_URL = "https://download.eclipse.org/es/jakartaee10/staged/eftl/${TCK_BUNDLE_NAME}.zip"

    env.ANT_VERSION='1.9.15'
    env.TOOLS_PREFIX='/opt/tools'
    env.JAVA_PREFIX="${TOOLS_PREFIX}/java/openjdk"
    env.MVN_HOME="${TOOLS_PREFIX}/apache-maven/latest"
    env.JAVA_HOME="${JAVA_PREFIX}/jdk-17/latest"
    env.PATH="${MVN_HOME}/bin:${JAVA_HOME}/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"

    env.ANT_OPTS="-Djavax.xml.accessExternalSchema=all"
    env.JAVA_TOOL_OPTIONS ='-XX:+IgnoreUnrecognizedVMOptions -XX:+UnlockExperimentalVMOptions '

    stage("Init") {
        // https://go.cloudbees.com/docs/cloudbees-documentation/cjoc-user-guide/index.html#cluster-copy-artifacts
        dir ("download") {
            // If any artefacts from other jobs need to be used, they can be placed here
            // copyRemoteArtifacts(from: "${GF_BUILD_JOB}")
            // copyRemoteArtifacts(from: "${TS_JTE_BUILD_JOB}")
        }
        sh '''#!/bin/bash -x

            mkdir download

           '''
    }

    stage("Grab GF") {
        env.GF_URL = "${GF_URL}"
        sh '''#!/bin/bash -ex

           # Add support for downloading GF from an arbitrary location

           '''
    }

    stage("Grab API and IMPL") {
        env.API_URL = "${API_URL}"
        env.API_JAR_NAME = "${API_JAR_NAME}"
        env.IMPL_URL = "${IMPL_URL}"
        env.IMPL_JAR_NAME = "${IMPL_JAR_NAME}"
        env.IMPL_URL2 = "${IMPL_URL2}"
        env.IMPL_JAR_NAME2 = "${IMPL_JAR_NAME2}"
        sh '''#!/bin/bash -ex

           '''
    }

    stage("Grab TCK") {
        env.TCK_BUNDLE_URL = "${TCK_BUNDLE_URL}"
        env.TCK_BUNDLE_NAME = "${TCK_BUNDLE_NAME}"
        sh '''#!/bin/bash -ex
            cd ${WORKSPACE}/download 
            wget -q ${TCK_BUNDLE_URL} -O ${TCK_BUNDLE_NAME}.zip

            cd ${WORKSPACE}
            unzip ${WORKSPACE}/download/${TCK_BUNDLE_NAME}.zip

            cat ${TCK_BUNDLE_NAME}/pom.xml
           '''
    }

    stage ("Configure ts.jte") {

        sh '''#!/bin/bash -ex

           '''
    }

     stage("Configure TCK") {
        sh '''#!/bin/bash -ex

           '''
    }

    stage ("Run TCK tests") {
        env.TCK_BUNDLE_NAME = "${TCK_BUNDLE_NAME}"
        sh '''#!/bin/bash -x

        ls -altrh
        echo ${TCK_BUNDLE_NAME}
        cd ${TCK_BUNDLE_NAME}

        mvn clean install -Pstaging,glassfish-ci-managed -Dglassfish.version=7.0.0-M4 surefire-report:failsafe-report-only -Daggregate=true | tee ${WORKSPACE}/run.log

           '''       
    } 

    stage ("Create summary.txt, API, and run.log artifacts") {
        env.TCK_BUNDLE_NAME = "${TCK_BUNDLE_NAME}"
        sh '''#!/bin/bash -ex
            cd ${WORKSPACE}/${TCK_BUNDLE_NAME}

            # Should not be needed, but some reason is.
            mvn site -Pstaging -pl -:glassfish-external-tck-security surefire-report:failsafe-report-only -Daggregate=true -DskipITs=true -DskipSurefireReport=true -Dglassfish.version=7.0.0-M4 -T2

            cd ${WORKSPACE}
            ls -altrh

            ls ${TCK_BUNDLE_NAME}
            ls ${WORKSPACE}/${TCK_BUNDLE_NAME}
            ls ${WORKSPACE}/${TCK_BUNDLE_NAME}/target

            cat ${TCK_BUNDLE_NAME}/target/site/failsafe-report.html | sed '/table/,/table/!d;//d' | sed '/Package/q' | sed -n 's:.*<td.*>\\(.*\\)</td>.*:\\1:p' > tmp_result.txt

            PASSED_COUNT=`cat tmp_result.txt |  sed '1q;d'`
            ERROR_COUNT=`cat tmp_result.txt |  sed '2q;d'`
            FAILED_COUNT=`cat tmp_result.txt |  sed '3q;d'`

            echo "********************************************************************************"  >> summary.txt
            echo "Completed running ${PASSED_COUNT} tests."                                          >> summary.txt
            echo "Number of Tests Failed      = ${FAILED_COUNT}"                                     >> summary.txt
            echo "Number of Tests with Errors = ${ERROR_COUNT}"                                      >> summary.txt
            echo "********************************************************************************"  >> summary.txt

            cat run.log | sed -e '1,/Completed running/d' >> summary.txt

            SHA256_API=`sha256sum ${WORKSPACE}/${TCK_BUNDLE_NAME}/target/glassfish7/glassfish/modules/${API_JAR_NAME} | awk '{print $1}'`
            SHA256_IMPL=`sha256sum ${WORKSPACE}/${TCK_BUNDLE_NAME}/target/glassfish7/glassfish/modules/${IMPL_JAR_NAME} | awk '{print $1}'`
            SHA256_TCK=`sha256sum ${WORKSPACE}/download/${TCK_BUNDLE_NAME}.zip | awk '{print $1}'`

            echo ERROR_COUNT=${ERROR_COUNT}
            echo FAILED_COUNT=${FAILED_COUNT}
            echo PASSED_COUNT=${PASSED_COUNT}

            echo ----

            echo SHA256_API=${SHA256_API} | tee -a summary.txt
            echo SHA256_IMPL=${SHA256_IMPL} | tee -a summary.txt
            echo SHA256_TCK=${SHA256_TCK}  | tee -a summary.txt

            echo ----

            echo TCK_download=${TCK_BUNDLE_URL} | tee -a summary.txt

            echo ----

            OS1=`lsb_release -a || true`
            OS2=`cat /etc/issue.net || true`
            OS3=`cat /etc/*_version || true`
            OS4=`cat /etc/*-release || true`

            if [ ! -z "${OS1}" ]; then
                echo OS1=${OS1} | tee -a summary.txt
            fi
            if [ ! -z "${OS2}" ]; then
                echo OS2=${OS2} | tee -a summary.txt
            fi
            if [ ! -z "${OS3}" ]; then
                echo OS3=${OS3} | tee -a summary.txt
            fi
            if [ ! -z "${OS4}" ]; then
                echo OS4=${OS4} | tee -a summary.txt
            fi

            JDK_VERSION=$(java -version 2>&1 || true)

            echo JDK_VERSION=${JDK_VERSION} | tee -a summary.txt

           '''

        archiveArtifacts artifacts: "summary.txt", fingerprint: true
        archiveArtifacts artifacts: "run.log", fingerprint: true
    }
}

Maybe we can do a minor maintenance release of the TCKs after EE 10. E.g. a 4.0.1 for security to clean up everything a little.

scottmarlow commented 9 months ago

Should these tasks be carried into a Faces 4.1 TCK task list and this issue closed?

arjantijms commented 9 months ago

Should these tasks be carried into a Faces 4.1 TCK task list and this issue closed?

At a glance I don't think anything needs to be carried over. These are all very specific items from the old to new TCK migration. I'll just close the issue now then.

jakartaee / faces

Prepare new Faces TCK #1600