Wrong allure report [cucumber]

IlgizVafin commented 8 years ago

Hi!

I use webdriverio + cucumber + allure-reporter. Thank you for awesome tool!

"wdio-allure-reporter": "0.0.2"
"webdriverio": "^4.2.10"

But, I notice that allure report is wrong. Here some details:

My example feature (example of real-life scenario)
- last step I expect that title is "example title2" is broken to test screenshot

basereporter.stats content. There are only one test I press next button
allure report.
- only one test I press next button;
- wrong duration of steps (each step accumulate duration);
- all output with screenshot and commands for failed step in first step "I go to the site". Expected output and attachments per each step.

just-boris commented 8 years ago

Hello, first of all, please try to upgrade wdio-allure-reporter to latest version 0.1.1. It contains so many conceptual changes, so your issue might been fixed.

IlgizVafin commented 8 years ago

Thanks!

New version looks good! But I have java.lang.NumberFormatException exception when I try to generate allure-report:

stdout: Command aborted due to exception {}.
 org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1)
    at org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:404)
    at org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:166)
    at org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:153)
    at ru.yandex.qatools.allure.command.ReportGenerate.runUnsafe(ReportGenerate.java:48)
    at ru.yandex.qatools.allure.command.AbstractCommand.run(AbstractCommand.java:52)
    at ru.yandex.qatools.allure.CommandLine.main(CommandLine.java:46)

 stderr: Exception in thread "main" java.lang.NumberFormatException: For input string: ""
    at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
    at java.lang.Long.parseLong(Long.java:601)
    at java.lang.Long.valueOf(Long.java:803)
    at com.sun.xml.internal.bind.DatatypeConverterImpl._parseLong(DatatypeConverterImpl.java:118)
    at ru.yandex.qatools.allure.model.TestCaseResult_JaxbXducedAccessor_stop.parse(TransducedAccessor_field_Long.java:50)
    at com.sun.xml.internal.bind.v2.runtime.unmarshaller.StructureLoader.startElement(StructureLoader.java:195)
    at com.sun.xml.internal.bind.v2.runtime.unmarshaller.UnmarshallingContext._startElement(UnmarshallingContext.java:559)
    at com.sun.xml.internal.bind.v2.runtime.unmarshaller.UnmarshallingContext.startElement(UnmarshallingContext.java:538)
    at com.sun.xml.internal.bind.v2.runtime.unmarshaller.SAXConnector.startElement(SAXConnector.java:153)
    at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.startElement(AbstractSAXParser.java:509)
    at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.scanStartElement(XMLNSDocumentScannerImpl.java:380)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2787)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:606)
    at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:118)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:510)
    at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:848)
    at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:777)
    at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
    at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1213)
    at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:643)
    at com.sun.xml.internal.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal0(UnmarshallerImpl.java:243)
    at com.sun.xml.internal.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal(UnmarshallerImpl.java:221)
    at com.sun.xml.internal.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal(UnmarshallerImpl.java:276)
    at javax.xml.bind.JAXB.unmarshal(JAXB.java:242)
    at ru.yandex.qatools.allure.commons.AllureFileUtils.unmarshal(AllureFileUtils.java:59)
    at ru.yandex.qatools.allure.commons.AllureFileUtils.unmarshal(AllureFileUtils.java:50)
    at ru.yandex.qatools.allure.commons.AllureFileUtils.unmarshal(AllureFileUtils.java:40)
    at ru.yandex.qatools.allure.data.io.TestSuiteReader$TestSuiteResultIterator.next(TestSuiteReader.java:48)
    at ru.yandex.qatools.allure.data.io.TestSuiteReader$TestSuiteResultIterator.next(TestSuiteReader.java:1)
    at ru.yandex.qatools.allure.data.io.TestCaseReader$TestCaseResultIterator.nextSuite(TestCaseReader.java:46)
    at ru.yandex.qatools.allure.data.io.TestCaseReader$TestCaseResultIterator.hasNext(TestCaseReader.java:56)
    at ru.yandex.qatools.allure.data.AllureReportGenerator.generate(AllureReportGenerator.java:61)
    at ru.yandex.qatools.allure.data.AllureReportGenerator.generate(AllureReportGenerator.java:53)
    at ru.yandex.qatools.allure.AllureMain.main(AllureMain.java:48)

 Process exited with code 1

just-boris commented 8 years ago

Well, I guess I know the reason of the issue, but could you please to post the content of *-testuite.xml file that you have in output directory?

IlgizVafin commented 8 years ago

allure-results.zip

just-boris commented 8 years ago

Thank you!

The exact issue in the following screenshot:

As you can see, all tests after broken doesn't have stop time and status. I will try to reproduce this in a small demo-project which uses cucumber specs, webdriverio and wdio-allure-reporter. @IlgizVafin could you help me to create it? I don't have so much experience with cucumber specs, so any help will be appreciated.

IlgizVafin commented 8 years ago

OK! I'll try to create a small demo-project in few days.

IlgizVafin commented 8 years ago

Boris, try to use this demo project: https://github.com/IlgizVafin/webdriverio-demo-project

I created 2 features: Passed.feature and Broken.feature. Broken scenario tries to click non-existent button and gets the error An element could not be located on the page using the given search parameters.

In *-testsuite.xml we can see the same picture

I hope this project will help you!

dezdem0na commented 8 years ago

@christian-bromann @just-boris regarding my error, here is xml file https://gist.github.com/anonymous/0d42dde2196423caa5fd65e05f925eae

<step start="1473677045071" stop="" status="">

adamnock commented 8 years ago

I can recreate this same issue using webdriverio ^4.2.12 + mocha ^0.4.2 + wdio-allure-reporter ^0.1.1

XML source of error:

correlates to the step: browser.frame(browser.element("iframe[name='gsft_nav']").value); Its the only step element produced which has no stop or status, subsequent elements are complete.

just-boris commented 8 years ago

I am not sure that issue was resolved completely

@IlgizVafin can you test against current master branch?

IlgizVafin commented 8 years ago

@just-boris, hello!

You are right! I have error in my demo-project with cucumber specs:

some details:

test:start handler is invoked for every step in Broken.feature (even steps after broken step) and writes to *-testsuite.xml new <test-case> with empty attributes (stop, status);
_this.getAllure(test.cid).pendingCase(test.title); writes to *-testsuite.xml new <test-case> with correct attributes.

just-boris commented 8 years ago

Thank you! Actually, I have checked out your demo project, but couldn't reproduce the issue.

Your additional information is useful, I will try again with this.

IlgizVafin commented 8 years ago

How can I help you?

just-boris commented 8 years ago

I tried to reproduce this, but got a bit different result.

In my case, the xml for broken feature was not created at all.

As usually, I added a logging hook here and got some details about events that were passed to reporter.

   handleEvent (...args) {
+      let event = args[0]
+      let params = args[1]
+      console.log(event, params && params.title)
       if (this.listeners(args[0]).length) {

suite:start Passed feature
suite:start Go to github.com and press signup button
  test:start I go to https://github.com
  test:pass I go to https://github.com

  test:start I press button "Sign up for GitHub"
  test:pass I press button "Sign up for GitHub"

  test:start I expect page title "Join GitHub · GitHub"
  test:pass I expect page title "Join GitHub · GitHub"
suite:end Go to github.com and press signup button
suite:end Passed feature

suite:start Broken feature
suite:start Go to github.com and press non-existent button
  test:start I go to https://github.com
  test:pass I go to https://github.com

  test:start I press button "Non-existent button"
  test:fail I press button "Non-existent button"

(The indentation here is mine, for readability).

As you can see, there is no suite:end event, that nesessary to tell reporter that suite is finished and xml can be dropped to disk.

Now I am going to figure out, what happens in wdio-cucumber-framework. Especially, why handleAfterScenarioEvent is not called.

IlgizVafin commented 8 years ago

Try to use:

"cucumber": "^1.3.0"
"wdio-cucumber-framework": "^0.2.5"

Older versions of packages give strange behaviour.

just-boris commented 8 years ago

Well, now this is better. The events dump is the following:

test:start I expect page title "Join GitHub · GitHub"
test:pending I expect page title "Join GitHub · GitHub"

The issue here is that cucumber has a different behavior, rather than mocha framework, cucumber emits test:start event anyway.

I think we can fix this if we will change test:pending to test:skipped event. Allure has also skipped aka cancelled state for testcases which were started but not finished. Semathically it is more correct, also friendly allure-cucumber project does this too

IlgizVafin commented 8 years ago

Good news!

We are waiting for a new release! :) Can I help you?

christian-bromann commented 8 years ago

@just-boris @IlgizVafin just released recent changes. Does this resolve the issue?

just-boris commented 8 years ago

I guess it is not.

First of all, the fix in webdriverio/wdio-cucumber-framework#34 should be shipped. Then I need to update wdio-allure-reporter to support the new event and also release it. Then the issue with cucumber will be closed.

anarwal commented 8 years ago

@just-boris it is not just cucumber, I face same the issue which is reported above in mocha framework also

just-boris commented 8 years ago

@anarwal that's true. So, if you have example, how to reproduce this issue using mocha framework, welcome to #29 That issue will be focused on fixing integration with Mocha.

anarwal commented 7 years ago

@just-boris Sure I will put in a small project to show it.

lj9142 commented 7 years ago

@just-boris do you know when the fix for this will be added? Thanks

just-boris commented 7 years ago

Now we have stale pull-request https://github.com/webdriverio/wdio-cucumber-framework/pull/34 Once it will be merged and released, issue should be gone

tripleratamacue commented 7 years ago

this is awesome guys, I just got the same error, was reading all comments, hope next release is the solution. Pura vida.

mygithub07 commented 7 years ago

Got the same error Exception in thread "main" java.lang.NumberFormatException: For input string: "" today when running allure generate allure-results My setup is

wdio v4.0.5 "wdio-mocha-framework": "^0.2.9" wdio-allure-reporter allure version gives me 1.4.23 So as i understood from the discussion above, there is a pending fix for this issue and next release should fix it ? (for mocha also) ?

dayanamcc commented 7 years ago

same issue Exception in thread "main" java.lang.NumberFormatException: For input string: "" when running allure generate allure-results

wdio v4.6.2 "wdio-cucumber-framework": "^0.2.15", "wdio-allure-reporter": "^0.1.2", allure version = 1.5.0

please suggest any solution.

its true the issue talked above , broken test that doesn't have stop time= "" and status = "" makes that allure can not generate the report.

vpattonjr commented 7 years ago

Exception in thread "main" java.lang.NumberFormatException: For input string: "" What's the status on this? I really like using the Allure reporter with WebdriverIO, but this keeps popping up.

just-boris commented 7 years ago

allure-commandline@2.0.0 has been released. It should fix the error for you. Also esure, that you have allure-js-commons@1.3.0 in your node modules. It is already within dependencies version range of this package, so it should be updated with next npm install.

Please, try v2 and check whether this error happens or not.

evless commented 7 years ago

@just-boris, hello! I install allure-commandline@2.0.0 and allure-js-commons@1.3.0, but exception was not lost. :(

Exception:

Exception in thread "main" java.lang.NumberFormatException: For input string: "" at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) at java.lang.Long.parseLong(Long.java:601) at java.lang.Long.valueOf(Long.java:803)

Modules: npm list

allure-commandline@2.1.1 allure-js-commons@1.3.0 wdio-allure-reporter@0.1.1

just-boris commented 7 years ago

@evless Needed more information. Are you using Cucumber? Do you have test steps marked as pending?

evless commented 7 years ago

@just-boris Sure!

Yep, i use cucumber
Yep, have test steps marked as pending allure-results.zip

vpattonjr commented 7 years ago

allure-commandline@2.0.0 fixed the issue, thx :)

just-boris commented 7 years ago

@evless so far I couldn't reproduce your problem. I have created a demo proejct: https://github.com/just-boris/wdio-cucumber-demo There I tried to do different steps and falilures, but everything worked as expected.

Also I noticed, that your package.json has an older version of wdio-cucumber-framework. The version that you are using is ~1 year old, the actual one is v0.3.1. Maybe, that can be a reason

christian-bromann commented 7 years ago

allure-commandline@2.0.0 fixed the issue, thx :)

Closing then. Thanks

webdriverio-boneyard / wdio-allure-reporter

Wrong allure report [cucumber] #26