explore adding first-class support for playwright

[ptrthomas]

for context - karate has support for Playwright via the (somewhat undocumented) wire-protocol, and we were (one of) the first to use that approach (sep 2020): https://twitter.com/ptrthomas/status/1307678474627244032

but it is experimental, and no one stepped forward to help improve it. some teams tried it - but it does have limitations / bugs: https://stackoverflow.com/q/68490053/143475

since then playwright has put a lot of effort into an official java library: https://github.com/microsoft/playwright-java - it is being very actively developed

plan here is to see if we can come out with karate-playwright

• I don't think we can embed the playwright dependencies into karate-core so it has to be a separate optional dependency
• the principle is it should be possible to switch a karate feature (that e.g. works for chrome native) to playwright with just a driver config change. I really think this is doable
• supporting playwright is a good thing for test-automation users
  • option to switch browser execution engines (chrome / CDP, webdriver, playwright)
  • compatibility with cloud executors like saucelabs, browserstack, lambdatest
  • simple scripting approach of karate (no async, await or promises)
  • future-proof your investment in UI automation
  • take advantage of playwright capabilities when possible

[f-delahaye]

Is anyone looking at this? If not, i'll try to give it a go. I'm no Playwright expert, but their api does look close to what Karate supports. Locators/selectors and a few other things might be tricky but we'll see.

Just a few questions:

• i guess this should be implemented as a subclass of Driver, right?
• if so, your second point ("it should be possible to switch a karate feature") is just about adding a new type, isn't it? Similar to type: chrome or type: playwright?
• speaking of which... what do we do with the existing PlaywrightDriver? Do we keep it for backward compatibility (especially if the new implementation requires an optional dependency)? Or is that not much of a concern?
• Is there a test suite for Driver implementations? I'm sure there are lots of subleties in selectors, for exemple, that would need to be tested carefully...

Thanks

[ptrthomas]

@f-delahaye you can go ahead. don't worry about the existing driver, we can throw it away if needed. I would prefer that playwright support is added as a new karate-playwright dependency, but don't let that stop you with starting off with what you find comfortable in a PR. yes to all your other points. there is a test-suite here: https://github.com/karatelabs/karate/tree/master/karate-e2e-tests

[f-delahaye]

Thanks @ptrthomas. I have a preliminary version that passes the tests, but needs some rework / clean up.

It aims at using the Playwright apis as much as possible, especially Locators but that raises a few points where your feedback is welcome:

• waitFor methods in Playwright can wait for the element to be either ATTACHED or VISIBLE. I went with the latter.
• Locators natively auto-wait/auto retry (every 30ms, apparently) and I could not find how to configure / switch this off. Most actions (input, clear, select) do have a force flag but for higher level checks. So how does that work with Karate's retry functionality? We could just set the timeout to retryCount * retryDelay and let PW sort it out ... What would be a use case for retry(3, 2000) over retry(6000), for example?
• PW comes with pseudo classes for all friendly locators (right-of, ...) BUT I could not get them to work with xpath selectors. For example: rightOf('//div').find('{}Some text') would fail rightOf('div').find('{}Some text') or rightOf('{}Some text').find('{}Some other text') would work, though. If its a blocker, I'll look into implementing friendly locators in javascript. It would also make the code a bit clearer, imo.

Sorry, that's a lot of questions!

[ptrthomas]

@f-delahaye sounds great !

• VISIBLE sounds good to me and the most common intent
• 30 ms ? wow that's weird. we could start out by announcing that retry is not applicable for playwright. first thought that comes to mind is if we can run an async thread on a pool that calls playwright - and then keep checking if playwright returned on the main thread that does the karate retry / polling stuff
• do CSS locators work ? that's good enough for version 1 then. internally we do some hacking with XPath, so you may get some clues if you can step-through and debug the java code

[f-delahaye]

Thanks!

My bad, the 30ms thing is probably a bit misleading. waitUntil('#mydiv', "function() {console.log(new Date()); return myEvaluationFunction") prints a new line every 30ms but that's in the browser. Things might be different on the java side.

Anyway ... my version now supports:

• css as well as xpath for friendly locators.
• a simple retry mechanism, with the caveat that retry(2, 3000) will try, wait for 3 seconds at most (it will in fact return as soon as the element is available), and if needed, try again, and wait for 3 seconds at most. Not exactly what the doc states, but it's the easiest to implement using Playwright's auto-retry/auto-wait. If not desirable, let me know and I will look at your suggestion.
• very basic implementation of intercept.

• cookies, however, Playwright will fail if they don't have a domain or a path/url.

  Code is available here https://github.com/f-delahaye/karate/commit/d81a05bed636b45e0d0afc753849bd2b1b44fe09

Note that the new PlaywrightDriver will take precedence over the "legacy" one if karate-playwright is added as a dependency in the pom.xml before karate-core.

Testing-wise, LocalPlaywrightRunner passes all the tests except 04.feature (for reason explained above) so it has now been skipped for driver 'playwright', and some lines of 13.feature (i suspect Wikipedia have actually changed their layout, driver 'chrome' fails there too) which have now been commented out.

I have changed 00.feature to re-enable some of the tests for PW, and also to add 18.feature and 19.feature. I'm not sure if it was a good idea. 19.feature for example fails with LocalSingleRunner at locate('//i').rightOf().find('//span') but that does not look like something Karate supports so maybe that particular test is not relevant...

I didn't run any tests other than LocalPlaywrightRunner (and LocalSingleRunner). Not sure if my home pc can run all the docker stuff...

Is that enough to create the PR? Or else, what are the next steps?

[ptrthomas]

@f-delahaye I think you can just go ahead with a PR. karate 1.5.X will change the maven groupId to io.karatelabs and I've just merged the jdk17 branch to develop so watch out for that. if you have localplaywrightrunner or singlerunner working, that is good ! I can look at the docker side later

[ptrthomas]

this has been merged ! thanks @f-delahaye

this may need some documentation and testing before release, there will certainly be 1.5.0.RC2 in a week or two. having regression tests using the docker container that embeds chrome will be good to have

[anildhiman88]

Hi @ptrthomas When you are planning to release this fix in karate-core?

[ptrthomas]

@anildhiman88 this is available right now in Karate 1.5.0.RC2 (just released) - note that the artifact grouo-id has changed to io.karatelabs

[f-delahaye]

Thanks @ptrthomas

Agree about updating the documentation. I believe the main points are in the javadoc. Will try to create a README out of it and improve it a bit.

Definitely agree about testing. I had an opportunity to use this code in a real life (very very simple) case and while it worked, I'm not fully comfortable yet. Aside from bugs, one of my concerns is the implementation of delay/retry already mentioned in a previous comment. So, basically, feedback is welcome. And if delay/retry does not look right, please reach out to me.

[ptrthomas]

posted on social media

LinkedIn: https://www.linkedin.com/feed/update/urn:li:activity:7137691423725780992 Twitter: https://twitter.com/getkarate/status/1731926887595913593

[azope25]

Hi @f-delahaye / @ptrthomas , When I was testing karate-playwright on my local for my tests, getting below "Timeout exceeded" error as by default set to 100ms.

org.graalvm.polyglot.PolyglotException: Error {
  message='Timeout 100ms exceeded.

 logs 
waiting for locator("xpath=//div[contains(normalize-space(text()),'xyz')]").first()
  locator resolved to <div class="more ng-binding"> xyz </div>
attempting click action
  waiting for element to be visible, enabled and stable
  element is visible, enabled and stable
  scrolling into view if needed
  done scrolling
  performing click action

  name='TimeoutError
  stack='TimeoutError: Timeout 100ms exceeded.

Also getting below error while using Driver Java api in my feature file.

My java code:- if(driver.exists("{button}ReadMe")){  ........

Error
org.graalvm.polyglot.PolyglotException: Error {
  message='Unsupported token "{" while parsing selector "{button}ReadMe"
  name='Error
  stack='Error: Unsupported token "{" while parsing selector "{button}ReadMe"

Can you please help me how to set timeout? Already configure karate retry in config file

Thanks

[f-delahaye]

@azope25 Thanks for the report! I will look at fixing the second issue. For the first one, would you be able to use retry as a workaround? Also, any chance you could share the scenario causing the timeout so i can investigate further?

[azope25]

@f-delahaye , I already configured karate.configure('retry',{ count:10, interval:3000}); in my karate-config.js file. Is anything need to do extra here?

I can not share my scenario due to limitation but I will try to create some dummy project and try to reproduce issue and share

I think for slow loading application is causing the issue as timeout is only 100ms but not sure

Thanks

[f-delahaye]

I think for slow loading application is causing the issue as timeout is only 100ms but not sure

@azope25 I don't disagree but assuming the timeout happened on a action, the doc states "By default, all actions such as click() will not be re-tried - and this is what you would stick to most of the time, for tests that run smoothly and quickly", so I assume no delay in addition to no retry. But I probably got that wrong. Maybe @ptrthomas can clarify this and what the "default" interval should be, if any ... 3s?
Anyway, there is definitely a bug here because karate.configure('retry',{ count:10, interval:3000}) should have forced a delay. Will look at it as well. In the meantime, * retry(10, 3000).click(...) might help (although it's not a global setting)

[azope25]

In the meantime, * retry(10, 3000).click(...) might help (although it's not a global setting)

@f-delahaye This workaround is not working for me.

js failed:
>>>>
01: retry(3,3000).waitFor(icons.search_icon).click()
<<<<
org.graalvm.polyglot.PolyglotException: Error {
  message='Timeout 100ms exceeded.

[f-delahaye]

My understanding is that:

• by default, actions do not retry and do not wait. So, a 100 ms timeout is fine(could be 0, but 0 has another meaning in PW), no reason to increase it.

• Looking at the documentation and other Drivers implementations, global retry settings do not apply to actions. So, this is not a bug, as far as I can tell.

• However, retry(3,3000).waitFor(icons.search_icon).click() timing out is. There's indeed an inconsistency between waitFor (which waits for an element to be _attached_) and click (which requires an element to be _visible_). I have been able to reproduce the issue consistently when an element is attached but only visible more than 100ms later, or randomly when both events are close to 100 ms of each other. @azope25 PR submitted with a fix for this scenario. If that does not fix your timeout, then I guess its a different use case, please try to provide a reproducer. You might also want to try retry(3,3000).click(icons.search_icon) (without the waitFor, to remove the inconsistency between waitFor and click) PR should also fix the Unsupported token "{" while parsing selector "{button}ReadMe" issue.

[azope25]

@f-delahaye Thank you for detail explanation. I will try at my end

[azope25]

@f-delahaye My Observations regarding below points are:-

Looking at the documentation and other Drivers implementations, global retry settings do not apply to actions. So, this is not a bug, as far as I can tell.

As per my knowledge, current action methods(e.g. click,clear etc) behavior in Karate is it wait for global retry setting defined in karate.config file like karate.configure('retry',{ count:10, interval:3000}) and override global retry if we explicitly use retry along with action methods. By looking into code written for playwright, Can we use resolveLocator().click(new ClickOptions().setTimeout(driver.waitTimeout())); instead of resolveLocator().click(new ClickOptions().setTimeout(driver.actionWaitTimeout()));. I tested this code with global retry setting and explicit use of retry function and its working as expected as karate methods works with other driver implementation.

Also observed that playwright throw exception (strict mode violation ) if more than one locator found but in karate it returns first locator by default. To fix this, Can we use like Locator resolveLocator(PlaywrightToken token) { return token.toLocator().first(); }

Please correct me if I misunderstood.

[f-delahaye]

Upon further testing, it looks like global retryCount is not used, but global retryInterval is! i.e.the other drivers try the action, wait retryInterval ms, try again and fail. So I will change actionWaitTimeout() accordingly. Good point, @azope25, thanks!

Indeed, resolveLocator is designed (among other things) to avoid the strict mode violation. I thought it was called internally in all cases, did you find some where it is not?

[azope25]

Indeed, resolveLocator is designed (among other things) to avoid the strict mode violation. I thought it was called internally in all cases, did you find some where it is not?

@f-delahaye I think ofRoot(locator) method called almost in every action/wait methods which internally calls 'root' method from 'PlaywrightToken' which is calling PlaywrightToken constructor which sets 'first' flag to false and due to this toLocator() method is returning locator instead of locator.first(). public Element click(String locator) { return new PlaywrightElement(this, ofRoot(locator)).click(); }

public static PlaywrightToken root(PlaywrightDriver driver, String token) { return new PlaywrightToken(driver, KarateTokenParser.toPlaywrightToken(token), null); }

public Locator toLocator() { Locator locator; if (getParent() == null) { locator = driver.rootLocator(getPlaywrightToken()); } else { locator = getParent().toLocator().locator(getPlaywrightToken()); } return (isFirst()) ? locator.first() : locator; } Possible Solution:- Can we call root method using PlayWright constructor with First flag set to true like public static PlaywrightToken root(PlaywrightDriver driver, String token) { return new PlaywrightToken(driver, KarateTokenParser.toPlaywrightToken(token), null, true); }

Also in some methods like text, value, attribute uses rootLocator which is not implemented for first locator , i guess.

Please correct me if I misunderstood

[f-delahaye]

You're right again! I don't know how I missed that, that's embarrassing...

I will take a closer look at the problem/your proposed solution.

[azope25]

Also in some methods like text, value, attribute uses rootLocator which is not implemented for first locator , i guess.

@f-delahaye For rootLocator, Can we do return this.root.locator(locator).first();

[f-delahaye]

@azope25 yes, that's probably the easiest and quickest workaround. Do note however that scriptAll and locateAll probably will be broken with this change. I will have to refactor a few things for a complete fix.

Regarding passing true in the constructor, i'm a bit concerned about potential side effects. I think first() should be delayed as late as possible.

[ptrthomas]

@f-delahaye just posting here for future reference - something I ran into when I tried to run the build on docker:

command I ran: docker run -it --rm -v "$(pwd)":/src -w /src -v "$HOME/.m2":/root/.m2 maven:3-amazoncorretto-17 mvn clean install -P pre-release -pl -karate-robot

error:

[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running com.intuit.karate.playwright.driver.PlaywrightDriverTest
OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended
03:51:52.538 [main] DEBUG com.intuit.karate.shell.Command -- found / verified free local port: 4444
03:51:52.544 [main] DEBUG com.intuit.karate.playwright.driver.null_1709697112524 -- Playwright browsers will not be installed.
/tmp/playwright-java-14683935091175296828/node: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by /tmp/playwright-java-14683935091175296828/node)
/tmp/playwright-java-14683935091175296828/node: /lib64/libc.so.6: version `GLIBC_2.28' not found (required by /tmp/playwright-java-14683935091175296828/node)
java.io.IOException: Stream closed
    at java.base/java.lang.ProcessBuilder$NullOutputStream.write(ProcessBuilder.java:445)
    at java.base/java.io.OutputStream.write(OutputStream.java:162)
    at java.base/java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:81)
    at java.base/java.io.BufferedOutputStream.flush(BufferedOutputStream.java:142)
    at com.microsoft.playwright.impl.WriterThread.run(PipeTransport.java:167)
java.io.IOException: Stream closed
    at java.base/java.lang.ProcessBuilder$NullOutputStream.write(ProcessBuilder.java:445)
    at java.base/java.io.OutputStream.write(OutputStream.java:162)
    at java.base/java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:81)
    at java.base/java.io.BufferedOutputStream.flush(BufferedOutputStream.java:142)
    at java.base/java.io.FilterOutputStream.close(FilterOutputStream.java:182)
    at com.microsoft.playwright.impl.PipeTransport.close(PipeTransport.java:93)
    at com.microsoft.playwright.impl.PipeTransport.poll(PipeTransport.java:71)
    at com.microsoft.playwright.impl.Connection.processOneMessage(Connection.java:200)

[f-delahaye]

Damn! Is that a blocking issue?

[ptrthomas]

@f-delahaye no not a blocking issue. must be some weirdness on linux + docker. so no worries for now

[anildhiman88]

Hi @ptrthomas @f-delahaye , I have a very small query related to passing the context parameters in playwright options. i.e. playwrightOptions :{{igboreHTTPSErrors: true}}. But no luck, i believe it is not passed correctly to the context parameters in playwright driver.

[ptrthomas]

@f-delahaye this is also not a blocker, but I ran into another CI failure trying to get the build to work for Java 22 https://github.com/karatelabs/karate/issues/2542

[ERROR] Tests run: 4, Failures: 0, Errors: 4, Skipped: 0, Time elapsed: 3.155 s <<< FAILURE! -- in com.intuit.karate.playwright.driver.PlaywrightElementTest
[ERROR] com.intuit.karate.playwright.driver.PlaywrightElementTest.shiftTextEnter -- Time elapsed: 2.245 s <<< ERROR!
org.mockito.exceptions.base.MockitoException: 

Mockito cannot mock this class: class com.intuit.karate.playwright.driver.PlaywrightDriver.

If you're not sure why you're getting this error, please open an issue on GitHub.

am going to comment this out for now, FYI

[f-delahaye]

Hi @ptrthomas @f-delahaye , I have a very small query related to passing the context parameters in playwright options. i.e. playwrightOptions :{{igboreHTTPSErrors: true}}. But no luck, i believe it is not passed correctly to the context parameters in playwright driver.

Hi @anildhiman88 Sorry, I missed your comment. Indeed, ignoredHTTPSErrors is not supported. Playwright has many options, at different levels (browserType, browserContext...) and I'm not sure how to support them all in an easy and generic way. I'll try to find a solution.

[ptrthomas]

1.5.0 released