vaadin / docs

Official documentation for Vaadin and Hilla.
https://vaadin.com/docs
Other
29 stars 195 forks source link

Spring Playwright testing does not work as described #3336

Closed edler-san closed 6 months ago

edler-san commented 7 months ago

Tell us about your request

Following https://vaadin.com/docs/latest/testing/playwright I ran into issues executing the tests. Playwright seems to not find the package.json even after running a production build to create it and the test fails both in an IDE and the terminal. I tested with OpenJDK 17.0.10, Node 21.7.2 (even though that is not used, see logs), Vaadin version 24.3.8 and 24.0.0.

[INFO] Scanning for projects...
[INFO]
[INFO] ---------------------< org.example:playwrightdemo >---------------------
[INFO] Building My Application 1.0-SNAPSHOT
[INFO]   from pom.xml
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- resources:3.3.1:resources (default-resources) @ playwrightdemo ---
[INFO] Copying 1 resource from src\main\resources to target\classes
[INFO] Copying 1 resource from src\main\resources to target\classes
[INFO]
[INFO] --- compiler:3.11.0:compile (default-compile) @ playwrightdemo ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- resources:3.3.1:testResources (default-testResources) @ playwrightdemo ---
[INFO] skip non existing resourceDirectory C:\Users\anon\Downloads\playwrightdemo\src\test\resources
[INFO]
[INFO] --- compiler:3.11.0:testCompile (default-testCompile) @ playwrightdemo ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- surefire:3.1.2:test (default-test) @ playwrightdemo ---
[INFO] Using auto detected provider org.apache.maven.surefire.junitplatform.JUnitPlatformProvider
[INFO]
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.example.TrivialPlaywrightTest
09:27:40.230 [main] INFO org.springframework.test.context.support.AnnotationConfigContextLoaderUtils -- Could not detect default configuration classes for test class [org.example.TrivialPlaywrightTest]: TrivialPlaywrightTest does not declare any static, non-private, non-final, nested classes annotated with @Configuration.
09:27:40.358 [main] INFO org.springframework.boot.test.context.SpringBootTestContextBootstrapper -- Found @SpringBootConfiguration org.example.Application for test class org.example.TrivialPlaywrightTest
09:27:45.053 [main] INFO org.springframework.boot.devtools.restart.RestartApplicationListener -- Restart disabled due to context in which it is running
 __  __             _
|  \/  | _   _     / \    _ __   _ __
| |\/| || | | |   / _ \  | '_ \ | '_ \
| |  | || |_| |  / ___ \ | |_) || |_) |
|_|  |_| \__, | /_/   \_\| .__/ | .__/
         |___/           |_|    |_|

2024-04-11T09:27:45.351+02:00  INFO 30380 --- [           main] org.example.TrivialPlaywrightTest        : Starting TrivialPlaywrightTest using Java 17.0.10 with PID 30380 (started by anon in C:\Users\anon\Downloads\playwrightdemo)
2024-04-11T09:27:45.351+02:00  INFO 30380 --- [           main] org.example.TrivialPlaywrightTest        : No active profile set, falling back to 1 default profile: "default"
2024-04-11T09:27:46.808+02:00  INFO 30380 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port 0 (http)
2024-04-11T09:27:46.825+02:00  INFO 30380 --- [           main] o.apache.catalina.core.StandardService   : Starting service [Tomcat]
2024-04-11T09:27:46.825+02:00  INFO 30380 --- [           main] o.apache.catalina.core.StandardEngine    : Starting Servlet engine: [Apache Tomcat/10.1.19]
2024-04-11T09:27:46.908+02:00  INFO 30380 --- [           main] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring embedded WebApplicationContext
2024-04-11T09:27:46.908+02:00  INFO 30380 --- [           main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 1528 ms
2024-04-11T09:27:50.410+02:00  INFO 30380 --- [           main] c.v.f.s.VaadinServletContextInitializer  : Search for subclasses and classes with annotations took 3194 ms
2024-04-11T09:27:50.873+02:00  INFO 30380 --- [           main] c.v.b.d.startup.DevModeStartupListener   : Starting dev-mode updaters in C:\Users\anon\Downloads\playwrightdemo folder.
2024-04-11T09:27:50.948+02:00  INFO 30380 --- [           main] c.v.f.s.f.s.FullDependenciesScanner      : Visited 105 classes. Took 50 ms.
2024-04-11T09:27:50.953+02:00  INFO 30380 --- [           main] c.v.f.s.frontend.BundleValidationUtil    : Checking if a development mode bundle build is needed
2024-04-11T09:27:51.382+02:00  INFO 30380 --- [           main] c.v.f.s.frontend.BundleValidationUtil    : A development mode bundle build is not needed
2024-04-11T09:27:51.461+02:00  INFO 30380 --- [onPool-worker-1] c.v.f.s.frontend.TaskCopyFrontendFiles   : Copying frontend resources from jar files ...
2024-04-11T09:27:51.605+02:00  INFO 30380 --- [onPool-worker-1] c.v.f.s.frontend.TaskCopyFrontendFiles   : Visited 20 resources. Took 141 ms.
2024-04-11T09:27:52.789+02:00  INFO 30380 --- [           main] c.v.f.s.DefaultDeploymentConfiguration   :
Vaadin is running in DEVELOPMENT mode - do not use for production deployments.
2024-04-11T09:27:52.862+02:00  INFO 30380 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port 58164 (http) with context path ''
2024-04-11T09:27:52.877+02:00  INFO 30380 --- [           main] org.example.TrivialPlaywrightTest        : Started TrivialPlaywrightTest in 7.839 seconds (process running for 13.453)
OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended
2024-04-11T09:27:53.979+02:00  INFO 30380 --- [o-auto-1-exec-1] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring DispatcherServlet 'dispatcherServlet'
2024-04-11T09:27:53.979+02:00  INFO 30380 --- [o-auto-1-exec-1] o.s.web.servlet.DispatcherServlet        : Initializing Servlet 'dispatcherServlet'
2024-04-11T09:27:53.979+02:00  INFO 30380 --- [o-auto-1-exec-1] o.s.web.servlet.DispatcherServlet        : Completed initialization in 0 ms
2024-04-11T09:27:54.088+02:00  INFO 30380 --- [o-auto-1-exec-1] c.vaadin.flow.spring.SpringInstantiator  : The number of beans implementing 'I18NProvider' is 0. Cannot use Spring beans for I18N, falling back to the default behavior
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 45.60 s <<< FAILURE! -- in org.example.TrivialPlaywrightTest
[ERROR] org.example.TrivialPlaywrightTest.testClicking -- Time elapsed: 32.76 s <<< ERROR!
com.microsoft.playwright.TimeoutError:
Error {
  message='Timeout 30000ms exceeded.
  name='TimeoutError
  stack='TimeoutError: Timeout 30000ms exceeded.
    at ProgressController.run (C:\Users\anon\AppData\Local\Temp\playwright-java-18005208858064934987\package\lib\server\progress.js:78:26)
    at Frame.click (C:\Users\anon\AppData\Local\Temp\playwright-java-18005208858064934987\package\lib\server\frames.js:996:23)
    at FrameDispatcher.click (C:\Users\anon\AppData\Local\Temp\playwright-java-18005208858064934987\package\lib\server\dispatchers\frameDispatcher.js:158:30)
    at FrameDispatcher._handleCommand (C:\Users\anon\AppData\Local\Temp\playwright-java-18005208858064934987\package\lib\server\dispatchers\dispatcher.js:94:40)
    at DispatcherConnection.dispatch (C:\Users\anon\AppData\Local\Temp\playwright-java-18005208858064934987\package\lib\server\dispatchers\dispatcher.js:365:39)
}
Call log:
- waiting for locator("//vaadin-button[contains(text(),'Click Me')]")

        at com.microsoft.playwright.impl.WaitableResult.get(WaitableResult.java:52)
        at com.microsoft.playwright.impl.ChannelOwner.runUntil(ChannelOwner.java:120)
        at com.microsoft.playwright.impl.Connection.sendMessage(Connection.java:130)
        at com.microsoft.playwright.impl.ChannelOwner.sendMessage(ChannelOwner.java:106)
        at com.microsoft.playwright.impl.FrameImpl.clickImpl(FrameImpl.java:252)
        at com.microsoft.playwright.impl.FrameImpl.lambda$click$8(FrameImpl.java:243)
        at com.microsoft.playwright.impl.LoggingSupport.lambda$withLogging$0(LoggingSupport.java:36)
        at com.microsoft.playwright.impl.LoggingSupport.withLogging(LoggingSupport.java:47)
        at com.microsoft.playwright.impl.ChannelOwner.withLogging(ChannelOwner.java:89)
        at com.microsoft.playwright.impl.LoggingSupport.withLogging(LoggingSupport.java:35)
        at com.microsoft.playwright.impl.FrameImpl.click(FrameImpl.java:243)
        at com.microsoft.playwright.impl.LocatorImpl.click(LocatorImpl.java:156)
        at com.microsoft.playwright.Locator.click(Locator.java:2387)
        at org.example.TrivialPlaywrightTest.testClicking(TrivialPlaywrightTest.java:33)
        at java.base/java.lang.reflect.Method.invoke(Method.java:568)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
Caused by: com.microsoft.playwright.TimeoutError: Error {
  message='Timeout 30000ms exceeded.
  name='TimeoutError
  stack='TimeoutError: Timeout 30000ms exceeded.
    at ProgressController.run (C:\Users\anon\AppData\Local\Temp\playwright-java-18005208858064934987\package\lib\server\progress.js:78:26)
    at Frame.click (C:\Users\anon\AppData\Local\Temp\playwright-java-18005208858064934987\package\lib\server\frames.js:996:23)
    at FrameDispatcher.click (C:\Users\anon\AppData\Local\Temp\playwright-java-18005208858064934987\package\lib\server\dispatchers\frameDispatcher.js:158:30)
    at FrameDispatcher._handleCommand (C:\Users\anon\AppData\Local\Temp\playwright-java-18005208858064934987\package\lib\server\dispatchers\dispatcher.js:94:40)
    at DispatcherConnection.dispatch (C:\Users\anon\AppData\Local\Temp\playwright-java-18005208858064934987\package\lib\server\dispatchers\dispatcher.js:365:39)
}
Call log:
- waiting for locator("//vaadin-button[contains(text(),'Click Me')]")

        at com.microsoft.playwright.impl.Connection.dispatch(Connection.java:254)
        at com.microsoft.playwright.impl.Connection.processOneMessage(Connection.java:211)
        at com.microsoft.playwright.impl.ChannelOwner.runUntil(ChannelOwner.java:118)
        ... 15 more

node:internal/modules/cjs/loader:1144
  const err = new Error(message);
              ^

Error: Cannot find module './../../package.json'
Require stack:
- C:\Users\anon\AppData\Local\Temp\playwright-java-18005208858064934987\package\lib\utils\userAgent.js
- C:\Users\anon\AppData\Local\Temp\playwright-java-18005208858064934987\package\lib\utils\index.js
- C:\Users\anon\AppData\Local\Temp\playwright-java-18005208858064934987\package\lib\cli\programWithTestStub.js
- C:\Users\anon\AppData\Local\Temp\playwright-java-18005208858064934987\package\cli.js
    at Module._resolveFilename (node:internal/modules/cjs/loader:1144:15)
    at Module._load (node:internal/modules/cjs/loader:985:27)
    at Module.require (node:internal/modules/cjs/loader:1235:19)
    at require (node:internal/modules/helpers:176:18)
    at getPlaywrightVersion (C:\Users\anon\AppData\Local\Temp\playwright-java-18005208858064934987\package\lib\utils\userAgent.js:91:54)
    at HarTracer.stop (C:\Users\anon\AppData\Local\Temp\playwright-java-18005208858064934987\package\lib\server\har\harTracer.js:414:54)
    at Tracing.abort (C:\Users\anon\AppData\Local\Temp\playwright-java-18005208858064934987\package\lib\server\trace\recorder\tracing.js:203:21)
    at CRBrowserContext._didCloseInternal (C:\Users\anon\AppData\Local\Temp\playwright-java-18005208858064934987\package\lib\server\browserContext.js:191:18)
    at CRBrowserContext._browserClosed (C:\Users\anon\AppData\Local\Temp\playwright-java-18005208858064934987\package\lib\server\browserContext.js:183:10)
    at CRBrowser._didClose (C:\Users\anon\AppData\Local\Temp\playwright-java-18005208858064934987\package\lib\server\browser.js:109:52) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    'C:\\Users\\anon\\AppData\\Local\\Temp\\playwright-java-18005208858064934987\\package\\lib\\utils\\userAgent.js',
    'C:\\Users\\anon\\AppData\\Local\\Temp\\playwright-java-18005208858064934987\\package\\lib\\utils\\index.js',
    'C:\\Users\\anon\\AppData\\Local\\Temp\\playwright-java-18005208858064934987\\package\\lib\\cli\\programWithTestStub.js',
    'C:\\Users\\anon\\AppData\\Local\\Temp\\playwright-java-18005208858064934987\\package\\cli.js'
  ]
}

Node.js v20.11.0
The batch file cannot be found.

[INFO]
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR]   TrivialPlaywrightTest.testClicking:33 » Timeout Error {
  message='Timeout 30000ms exceeded.
  name='TimeoutError
  stack='TimeoutError: Timeout 30000ms exceeded.
    at ProgressController.run (C:\Users\anon\AppData\Local\Temp\playwright-java-18005208858064934987\package\lib\server\progress.js:78:26)
    at Frame.click (C:\Users\anon\AppData\Local\Temp\playwright-java-18005208858064934987\package\lib\server\frames.js:996:23)
    at FrameDispatcher.click (C:\Users\anon\AppData\Local\Temp\playwright-java-18005208858064934987\package\lib\server\dispatchers\frameDispatcher.js:158:30)
    at FrameDispatcher._handleCommand (C:\Users\anon\AppData\Local\Temp\playwright-java-18005208858064934987\package\lib\server\dispatchers\dispatcher.js:94:40)
    at DispatcherConnection.dispatch (C:\Users\anon\AppData\Local\Temp\playwright-java-18005208858064934987\package\lib\server\dispatchers\dispatcher.js:365:39)
}
Call log:
- waiting for locator("//vaadin-button[contains(text(),'Click Me')]")

[INFO]
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  49.100 s
[INFO] Finished at: 2024-04-11T09:28:26+02:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.1.2:test (default-test) on project playwrightdemo:
[ERROR]
[ERROR] Please refer to C:\Users\anon\Downloads\playwrightdemo\target\surefire-reports for the individual test results.
[ERROR] Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException

If you think this issue is important, add a 👍 reaction to help the community and maintainers prioritize this issue.

mstahv commented 6 months ago

I tested and noticed there was for some reason one "M" instead of "m". Fixed that in https://github.com/vaadin/docs/commit/5f85b8d810e22c9936752363fc3dd6561b657a9d

But I'm not 100% sure if that was your issue, because there seems to be some weird package.json errors in your log. Let us know if the issue continues if you fix the selector or is this somethign else?

edler-san commented 6 months ago

Thanks for looking into that! With the update I now run into a Spring configuration issue. I am using a simple SpringBoot starter from start.vaadin.com with a "Hello World" view. I attached it for convenience. playwright.zip

10:40:37.167 [ForkJoinPool-1-worker-1] INFO org.springframework.test.context.support.AnnotationConfigContextLoaderUtils -- Could not detect default configuration classes for test class [com.example.TrivialPlaywrightTest]: TrivialPlaywrightTest does not declare any static, non-private, non-final, nested classes annotated with @Configuration.
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.552 s <<< FAILURE! -- in com.example.TrivialPlaywrightTest

[ERROR] com.example.TrivialPlaywrightTest -- Time elapsed: 0.552 s <<< ERROR!
java.lang.IllegalStateException: Unable to find a @SpringBootConfiguration, you need to use @ContextConfiguration or @SpringBootTest(classes=...) with your test
[...]
mstahv commented 6 months ago

Did you try what the error message suggested? That would probably be adding "classes=Application.class" (and related import) to the SpringBootTest annotation. Alternatively move the test to the same package where you have the Application class (but still in src/test/java source directory).

edler-san commented 6 months ago

I actually did that previously, yet the test still crashed and burned. Moving the test class to src\test\java\com\example\application\views\helloworld\TrivialPlaywrightTest.java and adding the view class to the test gives:

[ERROR] Errors:
[ERROR]   TrivialPlaywrightTest.testClicking » IllegalState Failed to load ApplicationContext for [WebMergedContextConfiguration@485f70bd testClass = com.example.application.views.helloworld.TrivialPlaywrightTest, locations = [], classes = [com.example.application.views.helloworld.HelloWorldView], contextInitializerClasses = [], activeProfiles = [], propertySourceDescriptors = [], propertySourceProperties = ["org.springframework.boot.test.context.SpringBootTestContextBootstrapper=true", "server.port=0"], contextCustomizers = [org.springframework.boot.test.context.filter.ExcludeFilterContextCustomizer@297bc4d8, org.springframework.boot.test.json.DuplicateJsonObjectContextCustomizerFactory$DuplicateJsonObjectContextCustomizer@3300e7d8, org.springframework.boot.test.mock.mockito.MockitoContextCustomizer@0, org.springframework.boot.test.web.client.TestRestTemplateContextCustomizer@372be360, org.springframework.boot.test.autoconfigure.actuate.observability.ObservabilityContextCustomizerFactory$DisableObservabilityContextCustomizer@1f, org.springframework.boot.test.autoconfigure.properties.PropertyMappingContextCustomizer@0, org.springframework.boot.test.autoconfigure.web.servlet.WebDriverContextCustomizer@49c3453b, org.springframework.boot.test.context.SpringBootTestAnnotation@48004231], resourceBasePath = "src/main/webapp", contextLoader = org.springframework.boot.test.context.SpringBootContextLoader, parent = null]
[INFO]
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0

The underlying issue seems to be:

Caused by: org.springframework.context.ApplicationContextException: Unable to start web server
        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.onRefresh(ServletWebServerApplicationContext.java:165)
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:618)
        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:146)
        at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:754)
        at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:456)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:334)
        at org.springframework.boot.test.context.SpringBootContextLoader.lambda$loadContext$3(SpringBootContextLoader.java:137)
        at org.springframework.util.function.ThrowingSupplier.get(ThrowingSupplier.java:58)
        at org.springframework.util.function.ThrowingSupplier.get(ThrowingSupplier.java:46)
        at org.springframework.boot.SpringApplication.withHook(SpringApplication.java:1454)
        at org.springframework.boot.test.context.SpringBootContextLoader$ContextLoaderHook.run(SpringBootContextLoader.java:553)
        at org.springframework.boot.test.context.SpringBootContextLoader.loadContext(SpringBootContextLoader.java:137)
        at org.springframework.boot.test.context.SpringBootContextLoader.loadContext(SpringBootContextLoader.java:108)
        at org.springframework.test.context.cache.DefaultCacheAwareContextLoaderDelegate.loadContextInternal(DefaultCacheAwareContextLoaderDelegate.java:225)
        at org.springframework.test.context.cache.DefaultCacheAwareContextLoaderDelegate.loadContext(DefaultCacheAwareContextLoaderDelegate.java:152)
        ... 21 more

Caused by: org.springframework.boot.web.context.MissingWebServerFactoryBeanException: No qualifying bean of type 'org.springframework.boot.web.servlet.server.ServletWebServerFactory' available: Unable to start AnnotationConfigServletWebServerApplicationContext due to missing ServletWebServerFactory bean
        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.getWebServerFactory(ServletWebServerApplicationContext.java:216)
        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.createWebServer(ServletWebServerApplicationContext.java:186)
        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.onRefresh(ServletWebServerApplicationContext.java:162)
        ... 35 more

The commonly suggest solution to add @SpringBootApplication to the Application.java does not apply as the annotation is already there (and the application works just fine with mvn spring-boot:run). What am I missing?

mstahv commented 6 months ago

I tested with your attached project and all fine there with both solutions I suggested 🤷‍♂️ If you have our support package, try to get somebody from there to help, althought this is purely Spring Boot configuation issue, rather than anything related to Vaadin-Playwright setup.

edler-san commented 6 months ago

Thanks for testing. Let's see if I can find a solution.