[discuss] test-execution progress reporting and changes in karate 1.0

[ptrthomas]

a few things to keep track of:

• in the long term com.intuit.karate.Main will be the only CLI, for now com.intuit.karate.cli.Main exists for backwards compatibility
• no more cucumber reports
• the reports directory is going to default to target/karate-reports
• if it already exists, karate will rename the existing dir to target/karate-reports-<timestamp> - I'm not sure what impact this has, for now I'm assuming that any browsers pointed to for e.g. karate-summary.html will not notice anything amiss
• renamed timeline.html to karate-timeline.html
• renamed results-json.txt to karate-results-json.txt
• we had discussed the need for keeping track of progress, so this file is new: karate-progress-json.txt right now it only contains {"featuresRemaining":42} but I'd like your thoughts on what else is needed. we can count the number of scenarios done. note that even the feature count is approximate, because of tags, and scenarios can "explode" dynamically at runtime because of outlines and dynamic data sources, so it will be very hard to derive a "starting total". but I'm hoping the featuresRemaining is a good number to estimate how much time is remaining.

[kirksl]

@ptrthomas thank you for the update. when do you anticipate all of this being released?

i was also thinking about this recently especially your last bullet point about tracking execution progress. a suggestion was recently made (referencing the link below) to use the Karate RuntimeHook to enable similar capabilities to the vscode ide.

curious what you see as pros/cons with RuntimeHook approach vs karate-progress-json.txt implementation. i'm thinking about which might provider better flexiblity in terms of reporting details, better abstraction for long term viability, etc.

https://github.com/ivangsa/karate-runner-fork/issues/1

[ptrthomas]

@kirksl I'm thinking this goes live mid jan-2021

the intent for karate-progress-json.txt is only to solve for rendering a progress-bar during test execution in VS code, I think we had discussed this some time back. so to show the number of features left as a progress-bar.

so the question is - is it possible to render a progress bar and if this is important. else I would de-prioritize this

[kirksl]

@ptrthomas yes we are actually rendering a progress bar today, but it is more of a spinner at the top of the "Tests" view in the extension. shown here https://recordit.co/7DnQGfyzty

we would definitely like to enhance this spinner to compete with other IDEs that show test execution progress(eg webstorm, intellij, etc)

featuresRemaining will work and definitely suffice for now. the only side effect which is probably an edge case would be if the user has lots of scenarios in each feature file, especially few feature files, the progress bar would not move gracefully. if equally accurate and easy enough to track scenariosRemaining would be preferable.

also curious reviewing our past conversations were you able to solve writing to the same json file when parallel runner is invoked? i mentioned a queue might be the way to go.

[ptrthomas]

@kirksl yes writing to the same json file is working. we try to lock it but don't wait and move on

[kirksl]

@ptrthomas great. let me know if you need more feedback. also when there is a viable branch to test the noted changes.

[ptrthomas]

@kirksl actually you should be able to test this in karate develop right now

[kirksl]

@ptrthomas built develop locally per dev guide. set pom.xml to 2.0.0 in test project. getting "cannot find symbol" errors on build. eg ExecutionContext, ScenarioContext, ExecutionHook, ScenarioResult, Resource

is this expected? noticed karate-apache is not built now as well which i was referencing.

[ptrthomas]

@kirksl there's no more karate-apache, just remove it from any dependencies: https://github.com/intuit/karate/wiki/1.0-upgrade-guide#breaking-changes

[kirksl]

@ptrthomas thanks for the guide.

nearly complete updating the demo project in my repo to conform to karate 1.0. wondering are mocks supposed to be setup via Runner.clientFactory( ) now? if yes can you suggest best design to handle env detection and conditionally setting up mocks. link below is what we have done in the past. single runner, user spec's karate.env, if "dev" setup mock in karate-config.

https://github.com/kirksl/karate-maven-gradle/blob/master/src/test/java/karate-config.js#L16

[ptrthomas]

@kirksl you mean the mock-servlet right

• so now this is pure java
• has to be constructed before calling the Runner
• you have to use the Runner class and builder and call the clientFactory() method
• since you know the environment when you call the Runner (note the new karateEnv() method) you can now perform the conditional logic in the Java code of the JUnit test

example: https://github.com/intuit/karate/blob/41fef7186f97505c6b98c117d2772b28d6a8b667/karate-mock-servlet/src/test/java/demo/MockSpringMvcServletTest.java#L45

[kirksl]

@ptrthomas thanks. i saw that example and was coding to it. thanks for confirming.

i am seeing a possible bug where one of the hooks are not being called with the RuntimeHook.

eg Runner.Builder rb = Runner.builder(); rb.path("classpath:org/company"); rb.tags("~@ignore"); rb.hook(new KarateHook()); Results results = rb.parallel(1);

these hooks are getting called beforeScenario afterScenario beforeFeature afterFeature afterSuite beforeStep afterStep

this one is not beforeSuite

[ptrthomas]

@kirksl good catch, thanks now added

[kirksl]

@ptrthomas posting some updates to your initial list...

in the long term com.intuit.karate.Main will be the only CLI, for now com.intuit.karate.cli.Main exists for backwards compatibility when do you plan to drop backwards compatibility support? i'll keep the old reference for now to avoid breaking people.

no more cucumber reports no change needed in extension. i will remove all references and code however from the demo project karate-maven-gradle in my repo to help instigate getting away from the old reports.

the reports directory is going to default to target/karate-reports if it already exists, karate will rename the existing dir to target/karate-reports- - I'm not sure what impact this has, for now I'm assuming that any browsers pointed to for e.g. karate-summary.html will not notice anything amiss renamed timeline.html to karate-timeline.html renamed results-json.txt to karate-results-json.txt for all items here i've updated the "target reports" glob pattern that populates the "build reports" UI in the extension to support the old and new files. should be seamless.

we had discussed the need for keeping track of progress, so this file is new: karate-progress-json.txt right now it only contains {"featuresRemaining":42} but I'd like your thoughts on what else is needed. we can count the number of scenarios done. note that even the feature count is approximate, because of tags, and scenarios can "explode" dynamically at runtime because of outlines and dynamic data sources, so it will be very hard to derive a "starting total". but I'm hoping the featuresRemaining is a good number to estimate how much time is remaining. just started implementing this and realized i need a way to calculate a "% of completion" which drives the status bar. can you add "featuresTotal: ###" to this? alternatively maybe i could extrapolate this value from the first read of featuresRemaining

[ptrthomas]

@kirksl i need a way to calculate a "% of completion" can you start by trying to extrapolate the featuresRemaining. in practice it seems to stabilize after the first few tests kick off and all "tagged" scenarios are in the queue. and once you suggest an algorithm, I don't mind putting it in the karate / server side to make it simpler for everyone

[kirksl]

@ptrthomas i've implemented the code in the extension to track test progress. i was unable to test as the results file is unfortunately always reporting "0" remaining.

[ptrthomas]

@kirksl I put a log statement in Suite.saveProgressJson() and it seems to work fine. try long running scenarios with sleeps in them

[kirksl]

@ptrthomas please let me know if you'd like to keep this open. i've updated the karate runner plugin to support the current state of karate 1.0 which is also released and ready in the vscode marketplace.

[ptrthomas]

@kirksl oh thanks - this can be closed, I think all the items have been covered !