search

serenity-bdd / serenity-core

Serenity BDD is a test automation library designed to make writing automated acceptance tests easier, and more fun.
http://serenity-bdd.info
Other
718 stars 516 forks source link

Serenity json file is written differently between IDE and command line running #1020

Closed nbarrett closed 5 years ago

nbarrett commented 6 years ago

I've noticed that the rendering of the final HTML reports in the areas of capabilities/requirements/stories is incorrect (e.g. the tests appear 'ignored' in the execution statistics) when I run our Serenity test suite from the command line. However when I run tests from inside the IDE (and therefore within the same module as our tests), the rendering is correct.

To clarify - we use BUCK as our build tool rather than maven or gradle, and all BUCK test commands are run from the monorepo root directory, not the module directory. Although the tests execute fine in terms of pass/fails, I've noticed that the serenity json file gets written differently between the two run modes.

Do you have a suggestion as to how I can influence the correct formatting of the json please?

Examples using the same feature/scenario

  "userStory": {
    "id": "feature-one",
    "storyName": "feature one",
    "path": "test1.feature",
    "type": "feature"
  },
  "featureTag": {
    "name": "feature one",
    "type": "feature"
  },
  "title": "one - scenario one",
  "description": "",
  "tags": [
    {
      "name": "Sequential",
      "type": "capability"
    },
    {
      "name": "nickb",
      "type": "tag"
    },
    {
      "name": "Testslice1/Feature one",
      "type": "feature"
    },
    {
      "name": "Sequential/Testslice1",
      "type": "feature"
    }
  ],
 "userStory": {
    "id": "feature-one",
    "storyName": "feature one",
    "path": "pipeline/sequential/testslice1/test1.feature",
    "type": "feature"
  },
  "featureTag": {
    "name": "Testslice1/feature one",
    "type": "feature"
  },
  "title": "one - scenario one",
  "description": "",
  "tags": [
    {
      "name": "nickb",
      "type": "tag"
    }
  ],
wakaleo commented 6 years ago

That looks like a problem with generating the requirements tags; something similar was happening with some of the previous Cucumber versions (due to the Cucumber 2.x API changes). Have you tried with the latest versions?

nbarrett commented 6 years ago

Thanks for your quick response @wakaleo. Actually, I did some further investigation and I eventually worked out that this happens if the current working directory is not the same as the test module directory. So I've got it to work by calling this method in the @BeforeClass method:

    public static void switchWorkingDirectory() {
        String currentDirectory = System.getProperty("user.dir");
        if (!currentDirectory.endsWith(TESTS_DIRECTORY)) {
            String workingDirectory = currentDirectory + TESTS_DIRECTORY;
            LOGGER.info("Switching working directory from {} -> {}", currentDirectory, workingDirectory);
            System.setProperty("user.dir", workingDirectory);
        }
    }

Maybe I should write a blog post about how to get Serenity running outside of maven/gradle? there are a few learning experiences!

nbarrett commented 6 years ago

Actually the above change didn't work as the web page folders (See screenshot below) got written to the execution directory, not the directory specified in serenity.outputDirectory. Also only the folders were created and nothing was in them!

image

So, I'm back to the drawing board. I am running Serenity 1.5.7 as 1.6.x has quite a few new dependencies to manually sort out as with buck as there is no transitive support. I don't want to move to Cucumber 2.0 yet but do want to get the latest Serenity updates. Is that possible?

wakaleo commented 6 years ago

It would be tricky if you are using Cucumber. 1.6.x includes the Cucumber 2.x support, which is a massive and very backward-incompatible upgrade, so you would need to upgrade everything. It might be worth trying, though, as the issue could be fixed in the latest versions. That said, I'm not familiar with Buck but have a look at the Maven and Gradle plugins to see what else you might need to do.

nbarrett commented 6 years ago

Okay, I've tried to upgrade my project to use the Serenity 1.7.3, Cucumber 2.1, Serenity Cucumber 1.6.4 in order to see whether the reported problem in this issue still happens but unfortunately the scenario no longer runs as two exceptions occur:


java.lang.NullPointerException
    at java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1011)
    at java.util.concurrent.ConcurrentHashMap.putIfAbsent(ConcurrentHashMap.java:1535)
    at net.thucydides.core.steps.StepEventBus.eventBusFor(StepEventBus.java:59)
    at cucumber.runtime.formatter.SerenityReporter.assureTestSuiteFinished(SerenityReporter.java:615)
    at cucumber.runtime.formatter.SerenityReporter.handleTestRunFinished(SerenityReporter.java:313)
    at cucumber.runtime.formatter.SerenityReporter.lambda$new$6(SerenityReporter.java:135)
    at cucumber.runner.EventBus.send(EventBus.java:28)
    at cucumber.api.junit.Cucumber$1.evaluate(Cucumber.java:109)
    at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
    at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
    at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
    at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
    at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
    at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
    at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)

I think I might raise a separate Serenity issue on this, as I think the real problem I'm having at the moment is the degree of change required on my client codebase in order to move up to Cucumber 2.x just for the purpose of running the latest Serenity code. There was lots of compilation errors I found in modules outside of the one I am testing that I have yet to address, but my gut feel is that we will have to stay on Cucumber 1.2.5 for quite a while yet. I think you might have other Serenity users that are also committed to staying on Cucumber 1.x that will also not be able to upgrade to the latest serenity too in order to work out whether their bugs are fixed. Could the version of cucumber be pluggable in some way to the latest version of Serenity?

wakaleo commented 6 years ago

The first issue might be a dependency issue (there is no Serenity code in the stack dump). Not sure about the second, but it could be related to the new Cucumber API (which needs an event bus per feature, not per thread).

The API has changed so drastically that I don't see how you could have both working together in a pluggable way, but feel free to take a look.

nbarrett commented 6 years ago

I wouldn't attempt to try and create the pluggable aspect I referred to as that would be way outside my area of knowledge - that's more an architectural aspect of Serenity I suspect. Even though I was only intending to run a single scenario, given that I specified unique tag in my test runner, I was surprised to see one log messages for every feature in my codebase e.g.

13-Nov-2017 19:01:54.077 INFO  [main] net.serenitybdd.core.Serenity - Test Suite Started: <name of feature>
(etc)

This doesn't seem right does it?

wakaleo commented 6 years ago

The others may be in DEBUG, though - best to check the CucumberReporter class to be sure.

nbarrett commented 6 years ago

Okay, given the suggestion in #102, the update to Serenity core, junit, report-resources, screenplay, screenplay-webdriver -> 1.7.3, Selenium -> 3.7.1, leaving serenity-cucumber @ 1.5.4, worked okay in terms of test execution, however the original problem raised in this issue still remains (not surprising given above comment). Some progress though as I can at least take Serenity core updates! 👍

nbarrett commented 6 years ago

Oh, hold on there is a new problem that has cropped up during the aggregation (I'm on cucumber core 1.2.5):

java.lang.NoSuchMethodError: cucumber.runtime.model.CucumberFeature.load(Lcucumber/runtime/io/ResourceLoader;Ljava/util/List;)Ljava/util/List;

    at net.thucydides.core.requirements.model.cucumber.CucumberParser.loadFeatureNarrative(CucumberParser.java:47)
    at net.thucydides.core.requirements.FileSystemRequirementsTagProvider.loadFromFeatureFile(FileSystemRequirementsTagProvider.java:528)
    at net.thucydides.core.requirements.FileSystemRequirementsTagProvider.readRequirementsFromStoryOrFeatureFile(FileSystemRequirementsTagProvider.java:495)
    at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
    at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
    at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
    at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
    at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
    at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
    at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
    at net.thucydides.core.requirements.FileSystemRequirementsTagProvider.loadStoriesFrom(FileSystemRequirementsTagProvider.java:476)
    at net.thucydides.core.requirements.FileSystemRequirementsTagProvider.getRequirements(FileSystemRequirementsTagProvider.java:133)
    at net.thucydides.core.requirements.FileSystemRequirementsTagProvider.readChildrenFrom(FileSystemRequirementsTagProvider.java:593)
    at net.thucydides.core.requirements.FileSystemRequirementsTagProvider.requirementFromDirectoryName(FileSystemRequirementsTagProvider.java:551)
    at net.thucydides.core.requirements.FileSystemRequirementsTagProvider.readRequirementFrom(FileSystemRequirementsTagProvider.java:487)
    at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
    at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
    at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
    at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
    at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
    at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
    at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
    at net.thucydides.core.requirements.FileSystemRequirementsTagProvider.loadCapabilitiesFrom(FileSystemRequirementsTagProvider.java:469)
    at net.thucydides.core.requirements.FileSystemRequirementsTagProvider.getRequirements(FileSystemRequirementsTagProvider.java:132)
    at net.thucydides.core.requirements.reports.FileSystemRequirmentsOutcomeFactory.buildRequirementsOutcomesFrom(FileSystemRequirmentsOutcomeFactory.java:37)
    at net.thucydides.core.reports.html.HtmlAggregateStoryReporter.generateReportsForTestResultsIn(HtmlAggregateStoryReporter.java:152)
    at net.thucydides.core.reports.html.HtmlAggregateStoryReporter.generateReportsForTestResultsFrom(HtmlAggregateStoryReporter.java:134)
wakaleo commented 6 years ago

Ah yes, the Cucumber API is used in serenity-core to read the feature files, and since it has changed, so will the serenity code. Not sure what to do about that - maybe have a look at the FileSystemRequirementsTagProvider and see if you can make a more pluggable strategy that adapts to the Cucumber API?

nbarrett commented 6 years ago

okay, sure - will do my best to work this out. The problem is that the latest net.thucydides.core.requirements.model.cucumber.CucumberParser calls CucumberFeature.load(resourceLoader, featurePaths), but Cucumber 1.2.5 only has these method signatures available:

Here's a screenshot of the call site:

image

nbarrett commented 6 years ago

Before I go too far, my suggestion for the pluggable solution would be to add a cucumber 1.2.5 compatible version of CucumberParser into a branch off [serenity-cucumber v1.5.13](https://github.com/serenity-bdd/serenity-cucumber/releases/tag/v1.5.13 - 28872f8) , and then make a change to serenity core (master) to instantiate this rather than the local version if it's on the classpath. How does that sound? I'm not sure whether it's possible to create a point release of a legacy version of a serenity library though? (e.g. v1.5.13.1)

wakaleo commented 6 years ago

The current release build wouldn't be able to build off a legacy version unfortunately - that's why there isn't a legacy support branch for selenium 2.x

nbarrett commented 6 years ago

Ah okay... where would you suggest I create a cucumber 1.2.5 compatible version of CucumberParser? It can't be in serenity-core as it wouldn't compile there...

wakaleo commented 6 years ago

I am wondering if it wouldn't be easier to port/shadow the 1.2.5 classes (it's only the ones use to parse the feature files, so hopefully there shouldn't be too many) into a local package?

nbarrett commented 6 years ago

Hi @wakaleo - I've spent some time working on your suggestion of importing the legacy cucumber classes into a local package and found the number of classes ran into over 100 files which really suggests to me that this is not the 'right' approach. More importantly this approach would not work in normal serenity usage (gradle/maven), given that all dependencies are pulled in transitively therefore both the shadowed and depended upon classes would be on the classpath at the same time. To me it seems that it would be better for serenity core to not have a cucumber dependency at all and instead pull in 'wrapped' versions of the cucumber classes from the serenity-cucumber project at runtime, rather than compile time as the latter would cause a circular dependency. I did a quick check and I only found 3 production serenity core classes that have cucumber or gherkin dependencies as follows:

There are a few tests that also have cucumber dependencies but perhaps these belong in the cucumber serenity project instead? Do you think it's sensible / possible to remove the hardcoded dependency from serenity core to enable users like me that can't move to the latest version of cucumber to still work on the latest serenity core? I don't mind having a go at doing this providing you think this is worthwhile?

wakaleo commented 6 years ago

Hi Nick,

You could try to extract the logic, but be careful not to break the requirements reporting - those classes (FailureAnalysisConfiguration and FailureAnalysis) are quite important. The file system requirements code uses the Cucumber code to read directory-based requirements structures - historically, there have been may ways to represent requirements, but if this could be made optional and dependent on the presence of the cucumber dependency, it might still work.

On 21 November 2017 at 17:02, Nick Barrett notifications@github.com wrote:

Hi @wakaleo https://github.com/wakaleo - I've spent some time working on your suggestion of importing the legacy cucumber classes into a local package and found the number of classes ran into over 100 files which really suggests to me that this is not the 'right' approach. More importantly this approach would not work in normal serenity usage (gradle/maven), given that all dependencies are pulled in transitively therefore both the shadowed and depended upon classes would be on the classpath at the same time. To me it seems that it would be better for serenity core to not have a cucumber dependency at all and instead pull in 'wrapped' versions of the cucumber classes from the serenity-cucumber project at runtime, rather than compile time as the latter would cause a circular dependency. I did a quick check and I only found 3 production serenity core classes that have cucumber or gherkin dependencies as follows:

  • net.thucydides.core.model.failures.FailureAnalysisConfiguration & FailureAnalysis
    • cucumber.api.PendingException
  • net.thucydides.core.requirements.model.cucumber.CucumberParser
    • cucumber.runtime.io.MultiLoader
    • cucumber.runtime.model.CucumberFeature
    • gherkin.ast.GherkinDocument
    • gherkin.ast.Tag

There are a few tests that also have cucumber dependencies but perhaps these belong in the cucumber serenity project instead? Do you think it's sensible / possible to remove the hardcoded dependency from serenity core to enable users like me that can't move to the latest version of cucumber to still work on the latest serenity core? I don't mind having a go at doing this providing you think this worthwhile?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/serenity-bdd/serenity-core/issues/1020#issuecomment-346092923, or mute the thread https://github.com/notifications/unsubscribe-auth/AADassLxr47EOMUqf6Q7QnDsumozdGiGks5s4wIsgaJpZM4QadKA .

--

John Smart | Wakaleo Consulting | +44 7398 832273 Making smart teams collaborate better http://johnfergusonsmart.com | john.smart@wakaleo.com

We love breaking down silos and helping smart teams collaborate better! Ask about our tailored on-site workshops https://johnfergusonsmart.com/training/ in Agile Product Planning http://johnfergusonsmart.com/programs-courses/next-generation-agile-delivery-product-planning/, BDD Requirements Discovery http://johnfergusonsmart.com/programs-courses/mastering-agile-requirements-behaviour-driven-development/, BDD, TDD and Clean Coding http://johnfergusonsmart.com/programs-courses/bdd-tdd-clean-coding/, and Advanced BDD Test Automation http://johnfergusonsmart.com/programs-courses/advanced-bdd-test-automation/ .

Need some help with Serenity BDD? Check out our Serenity BDD training and support packages here https://johnfergusonsmart.com/serenity-bdd/.

nbarrett commented 6 years ago

Wow..an update on this....I eventually managed to upgrade my project to the latest Cucumber[2.3]/Serenity[1.9.2] and the rendering is still not working when we run tests from the command line. I did manage to turn up some logging however and saw that when we run from the IDE I get this:

13-Mar-2018 17:53:39.025 DEBUG [pool-3-thread-1] n.t.c.r.RequirementsTagProvider - Requirements found:[Requirement{name='Parallel', type='capability' parent = 'null', cardNumber='null'}, Requirement{name='Sequential', type='capability' parent = 'null', cardNumber='null'}]
13-Mar-2018 17:53:39.039 DEBUG [pool-3-thread-1] n.t.c.r.BaseRequirementsService - Requirement found for test outcome LCTRS notification is sent at the End of Assessment Period for a claimant that claims Council Tax Reduction through UC-[]: Optional[Requirement{name='Lads notifications can be generated and forwarded to data hub', type='feature' parent = 'Sequential', cardNumber='null'}]

and when I run from the command line (when it doesn't work properly) I get this:

13-Mar-2018 18:24:13.181 DEBUG [pool-3-thread-1] n.t.c.r.RequirementsTagProvider - Requirements found:[]

So the inability to resolve the requirements directory seems to be the root of the problem. I've even tried setting the property as follows:

-Dserenity.requirements.dir=/Users/nickbarrett/dev/universe/automated-tests/business-assurance-tests/src/test/resources/pipeline

...but that doesn't make any difference either... Any further suggestions please @wakaleo ?

wakaleo commented 6 years ago

For some reason, when you run on the command line, it is not finding the requirements directory (most likely it is expecting something in src/test/resources/features), and the system property is being detected in the IDE but not by Gradle. Which is odd; maybe something to do with the Gradle working directory?

nbarrett commented 6 years ago

Actually I made a minor breakthrough with this problem in that when I hard-code the serenity.requirements.dir value in serenity.conf then it works! So the problem in this issue seems to be same as #1071 - which is that some system properties don't override the physical values. Any suggestions about that one please?! 🙏

wakaleo commented 6 years ago

Maybe have a look at how and when the environment variables are loaded in the SystemEnvironmentVariables class.

wakaleo commented 6 years ago

This might be fixed in 1.9.9

wakaleo commented 5 years ago

Reproduced and fixed in 2.0.7 (getAbsolutePath() works differently depending on where you run it from)