pytest-bdd 5.0.0: error on generating report with failed tests with scenario outlines

[sqandrew]

[//]: # ( . Note: for support questions, please use Stackoverflow or Gitter. . This repository's issues are reserved for feature requests and bug reports. . . In case of any problems with Allure Jenkins plugin please use the following repository . to create an issue: https://github.com/jenkinsci/allure-plugin/issues . . Make sure you have a clear name for your issue. The name should start with a capital . letter and no dot is required in the end of the sentence. An example of good issue names: . . - The report is broken in IE11 . - Add an ability to disable default plugins . - Support emoji in test descriptions )

I'm submitting a ...

• [V] bug report
• [V] feature request

What is the current behavior?

report doesn't include failed tests with scenario outline (pytest-bdd 5.0.0) it fails with com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot deserialize instance of java.lang.String out of START_OBJECT token

If the current behavior is a bug, please provide the steps to reproduce and if possible a minimal demo of the problem

1. pytest-bdd 5.0.0
2. run scenario outlines with examples which fails
3. generate report

What is the expected behavior?

failed tests appears in report

Please tell us about your environment:

• Test framework: pytest-bdd@5.0.0
• Allure adaptor: allure-pytest-bdd@2.9.45 (and other lower versions)

Other information

it worked OK with previous pytest-bdd versions

pytest-bdd 5.0.0 changes https://pytest-bdd.readthedocs.io/en/latest/#id1

[andronikosnedos-temboo]

I've come across the same issue, it fails to generate the report even for successful tests when using Scenario Outline.

com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot deserialize value of type `java.lang.String` from Object value (token `JsonToken.START_OBJECT`)
 at [Source: (sun.nio.ch.ChannelInputStream); line: 1, column: 855] (through reference chain: io.qameta.allure.model.TestResult["parameters"]->java.util.ArrayList[0]->io.qameta.allure.model.Parameter["value"])
    at com.fasterxml.jackson.databind.exc.MismatchedInputException.from(MismatchedInputException.java:59)

[JagadeeshJayachandran]

Same issue, when this change will be available please

[sqandrew]

please someone take a look to PRs

[leibson]

I see the same thing.
allure-pytest-bdd==2.9.45 pytest-bdd==5.0.0

Sharing some sample data.

I also checked out https://pytest-bdd.readthedocs.io/en/latest/#migration-from-4-x-x, and I am following the guidelines, for example:

@allure.step
@then(parsers.parse('return code is "{returnCode}"'))

f204af76-9e79-44ee-a7d5-26e214885178-result.json

{"name": "Log in [my_email-my_password-Api Tester]", "status": "passed", "steps": [{"name": "Given Email is \"my_email\" and password is \"my_password\"", "status": "passed", "start": 1654072367059, "stop": 1654072367059}, {"name": "When I log in", "status": "passed", "start": 1654072367059, "stop": 1654072368071}, {"name": "Then Result shows name is \"Api Tester\"", "status": "passed", "start": 1654072368072, "stop": 1654072368072}], "parameters": [{"name": "_pytest_bdd_example", "value": {"email": "my-email", "password": "my_password", "name": "Api Tester"}}], "start": 1654072367058, "stop": 1654072368072, "uuid": "0a81e533-5799-e47d-6b56-d7ccd0c31b1f", "historyId": "0a81e5335799e47d6b56d7ccd0c31b1f", "fullName": "features\\visitor_page.feature:Log in", "labels": [{"name": "host", "value": "ILGUYL-LT1"}, {"name": "thread", "value": "5072-MainThread"}, {"name": "framework", "value": "pytest-bdd"}, {"name": "language", "value": "cpython3"}, {"name": "feature", "value": "Visitor Page for Tigapo"}]}

Could not read test result file reports\f204af76-9e79-44ee-a7d5-26e214885178-result.json
com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot deserialize value of type `java.lang.String` from Object value (token `JsonToken.START_OBJECT`)
 at [Source: (sun.nio.ch.ChannelInputStream); line: 1, column: 520] (through reference chain: io.qameta.allure.model.TestResult["parameters"]->java.util.ArrayList[0]->io.qameta.allure.model.Parameter["value"])
        at com.fasterxml.jackson.databind.exc.MismatchedInputException.from(MismatchedInputException.java:59)
        at com.fasterxml.jackson.databind.DeserializationContext.reportInputMismatch(DeserializationContext.java:1601)
        at com.fasterxml.jackson.databind.DeserializationContext.handleUnexpectedToken(DeserializationContext.java:1375)
        at com.fasterxml.jackson.databind.DeserializationContext.handleUnexpectedToken(DeserializationContext.java:1280)
        at com.fasterxml.jackson.databind.DeserializationContext.extractScalarFromObject(DeserializationContext.java:872)
        at com.fasterxml.jackson.databind.deser.std.StringDeserializer.deserialize(StringDeserializer.java:62)
        at com.fasterxml.jackson.databind.deser.std.StringDeserializer.deserialize(StringDeserializer.java:11)
        at com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeAndSet(MethodProperty.java:129)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:324)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:187)
        at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer._deserializeFromArray(CollectionDeserializer.java:355)
        at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:244)
        at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:28)
        at com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeAndSet(MethodProperty.java:129)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:324)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:187)
        at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:322)
        at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4593)
        at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3585)
        at io.qameta.allure.allure2.Allure2Plugin.readTestResult(Allure2Plugin.java:352)
        at java.util.stream.ReferencePipeline$3$1.accept(Unknown Source)
        at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(Unknown Source)
        at java.util.stream.AbstractPipeline.copyInto(Unknown Source)
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source)
        at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(Unknown Source)
        at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(Unknown Source)
        at java.util.stream.AbstractPipeline.evaluate(Unknown Source)
        at java.util.stream.ReferencePipeline.forEach(Unknown Source)
        at io.qameta.allure.allure2.Allure2Plugin.readResults(Allure2Plugin.java:111)
        at io.qameta.allure.ReportGenerator.lambda$readResults$0(ReportGenerator.java:47)
        at java.util.ArrayList.forEach(Unknown Source)
        at io.qameta.allure.ReportGenerator.readResults(ReportGenerator.java:47)
        at java.util.stream.ReferencePipeline$3$1.accept(Unknown Source)
        at java.util.stream.ReferencePipeline$2$1.accept(Unknown Source)
        at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(Unknown Source)
        at java.util.stream.AbstractPipeline.copyInto(Unknown Source)
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source)
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(Unknown Source)
        at java.util.stream.AbstractPipeline.evaluate(Unknown Source)
        at java.util.stream.ReferencePipeline.collect(Unknown Source)
        at io.qameta.allure.ReportGenerator.generate(ReportGenerator.java:69)
        at io.qameta.allure.ReportGenerator.generate(ReportGenerator.java:58)
        at io.qameta.allure.Commands.generate(Commands.java:102)
        at io.qameta.allure.Commands.serve(Commands.java:127)
        at io.qameta.allure.CommandLine.run(CommandLine.java:167)
        at java.util.Optional.orElseGet(Unknown Source)
        at io.qameta.allure.CommandLine.main(CommandLine.java:92)

[olerudyshyn]

The same issue: allure-pytest-bdd==2.9.45 pytest-bdd==5.0.0

Could not read test result file allure-results/55e4a1df-d110-4fe7-8ddb-4acb6bf0f380-result.json
com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot deserialize value of type `java.lang.String` from Object value (token `JsonToken.START_OBJECT`)
 at [Source: (sun.nio.ch.ChannelInputStream); line: 1, column: 866] (through reference chain: io.qameta.allure.model.TestResult["parameters"]->java.util.ArrayList[0]->io.qameta.allure.model.Parameter["value"])
    at com.fasterxml.jackson.databind.exc.MismatchedInputException.from(MismatchedInputException.java:59)
    at com.fasterxml.jackson.databind.DeserializationContext.reportInputMismatch(DeserializationContext.java:1601)
    at com.fasterxml.jackson.databind.DeserializationContext.handleUnexpectedToken(DeserializationContext.java:1375)
    at com.fasterxml.jackson.databind.DeserializationContext.handleUnexpectedToken(DeserializationContext.java:1280)
    at com.fasterxml.jackson.databind.DeserializationContext.extractScalarFromObject(DeserializationContext.java:872)
    at com.fasterxml.jackson.databind.deser.std.StringDeserializer.deserialize(StringDeserializer.java:62)
    at com.fasterxml.jackson.databind.deser.std.StringDeserializer.deserialize(StringDeserializer.java:11)
    at com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeAndSet(MethodProperty.java:129)
    at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:324)
    at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:187)
    at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer._deserializeFromArray(CollectionDeserializer.java:355)
    at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:244)
    at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:28)
    at com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeAndSet(MethodProperty.java:129)
    at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:324)
    at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:187)
    at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:322)
    at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4593)
    at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3585)
    at io.qameta.allure.allure2.Allure2Plugin.readTestResult(Allure2Plugin.java:352)
    at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
    at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
    at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
    at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
    at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
    at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
    at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
    at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596)
    at io.qameta.allure.allure2.Allure2Plugin.readResults(Allure2Plugin.java:111)
    at io.qameta.allure.ReportGenerator.lambda$readResults$0(ReportGenerator.java:47)
    at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
    at io.qameta.allure.ReportGenerator.readResults(ReportGenerator.java:47)
    at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
    at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179)
    at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
    at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
    at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
    at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
    at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
    at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
    at io.qameta.allure.ReportGenerator.generate(ReportGenerator.java:69)
    at io.qameta.allure.ReportGenerator.generate(ReportGenerator.java:58)
    at io.qameta.allure.Commands.generate(Commands.java:102)
    at io.qameta.allure.Commands.serve(Commands.java:127)
    at io.qameta.allure.CommandLine.run(CommandLine.java:167)
    at java.base/java.util.Optional.orElseGet(Optional.java:364)
    at io.qameta.allure.CommandLine.main(CommandLine.java:92)

[aspenboy]

I'm having exactly the same issue, but using pytest-bdd==6.0.0

[JagadeeshJayachandran]

Any update on this issue, please?

[junkei-okinawa]

It seems that it occurs because the content of json output is different between allure-pytest and allure-pytest-bdd.

dosn't works

{"name": "ユーザーが各Yahooサービスページに移動できる [\\u30b7\\u30e7\\u30c3\\u30d4\\u30f3\\u30b0]", "status": "passed", "steps": [{"name": "Given YahooJapanトップページ", "status": "passed", "start": 1658995157898, "stop": 1658995164567}, {"name": "When サイドパネルの「ショッピング」をクリック", "status": "passed", "start": 1658995164569, "stop": 1658995167515}, {"name": "And 「ショッピング」サイトに遷移していることを確認", "status": "passed", "start": 1658995167515, "stop": 1658995167578}, {"name": "Then Yahoo! Japanのリンクをクリックできる", "status": "passed", "start": 1658995167579, "stop": 1658995169701}, {"name": "And Yahoo! Japanトップページに遷移していることを確認", "status": "passed", "start": 1658995169701, "stop": 1658995169783}], "parameters": [{"name": "_pytest_bdd_example", "value": {"service": "ショッピング"}}], "start": 1658995157882, "stop": 1658995169783, "uuid": "db3f2b15-d90a-aeff-050e-6426cf56685e", "historyId": "db3f2b15d90aaeff050e6426cf56685e", "fullName": "tests\\test_yahoofinance.feature:ユーザーが各Yahooサービスページに移動できる", "labels": [{"name": "host", "value": "S-NPC-5340"}, {"name": "thread", "value": "9292-MainThread"}, {"name": "framework", "value": "pytest-bdd"}, {"name": "language", "value": "cpython3"}, {"name": "feature", "value": "Yahooジャパン"}]}

It's worked

{"uuid": "97defbbd-ca84-439c-bd76-83a8e0bdf1b5", "befores": [{"name": "page", "status": "passed", "start": 1658993243274, "stop": 1658993244353}], "afters": [{"name": "page::0", "status": "passed", "start": 1658993255567, "stop": 1658993255567}], "start": 1658993243274, "stop": 1658993255567}

[JagadeeshJayachandran]

@junkei-okinawa, any work around when using allure-pytest-bdd, please.

[soprano8086]

hit this issue. pytest-bdd=6.0.1 allure=2.18.1 allure-pytest-bdd=2.9.45

[olerudyshyn]

I have workaround for this issue. Following code should be placed in conftest.py:

from allure_commons.lifecycle import AllureLifecycle
from allure_commons.model2 import TestResult
from allure_commons import plugin_manager

def custom_write_test_case(self, uuid=None):
    test_result = self._pop_item(uuid=uuid, item_type=TestResult)
    if test_result:
        if test_result.parameters:
            adj_parameters = []
            for param in test_result.parameters:
                if param.name != '_pytest_bdd_example':
                    # do not include parameters with "_pytest_bdd_example"
                    adj_parameters.append(param)
            test_result.parameters = adj_parameters

        plugin_manager.hook.report_result(result=test_result)

AllureLifecycle.write_test_case = custom_write_test_case

[DianPermana]

same issue on :

pytest-bdd=6.0.1 allure-pytest-bdd=2.10.0

[robbrad]

@delatrie Still not fixed for me

pytest-bdd = "^6.1.1" allure-pytest-bdd = "^2.12.0" pytest-xdist = {extras = ["psutil"], version = "^3.2.0"}

error is :

Could not read test result file .\build\allure-results\d48b4a4f-9c33-4fb7-85cd-9dccd37d03b2-result.json
com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot deserialize value of type `java.lang.String` from Object value (token `JsonToken.START_OBJECT`)

[delatrie]

@robbrad Hi! Sorry for the delay. The fix just got released. Check v2.13.0!

[robbrad]

@delatrie No issues at all my bad for not checking the releases

For some reason I thought the commit to master was enough to be out there in the wild

[CuriousQA]

@delatrie I am still encountering issues when running the pytest-bdd allure report with the latest fix

[delatrie]

Hi, @CuriousQA !

Did you remove all the previously generated *.json files before running a new version of allure-pytest-bdd?

[CuriousQA]

Yes @delatrie , I created a new folder and ran the tests.

[delatrie]

I will take a look. Please, provide me with the following:

• the feature file
• the test file (.py)
• the content of the .json file from the error message
• CLI command you used to run the test

[CuriousQA]

Thanks @delatrie . I will create a mock feature/test file and share it with you.

[CuriousQA]

@delatrie , The issue I encountered is when the feature files has Scenario Outline with Examples.

command: pytest -m cucumber -W ignore --alluredir reports3

Test_Allure_report.zip

[delatrie]

@CuriousQA Hello again!

I was unable to reproduce the effect you have observed.

Check this repo: https://github.com/delatrie/allure-python-633-repro. It has detailed report on how exactly I tried to reproduce the issue. The actual commands and the output are available in the workflow run here on Lunux and on Windows.

Note, that unlike in the files you have attached, there are no _pytest_bdd_example parameters in the *-result.json files (Lunix: 1, 2, 3, 4; Windows: 1, 2, 3, 4).

Please, compare with how you have run the tests.

For now, it looks to me that an older version of allure-pytest-bdd is still sitting in the environment you used to run the tests. Maybe you can try to create a new environment from scratch.

What also catches my eye is the fact that the versions of allure-pytest and allure-pytest-bdd differ from each other on your screenshot of the pip list output. When I force pip to install allure-pytest==2.12.0 while upgrading allure-pytest-bdd and allure-commons to 2.13.0, pytest crashes on my machine because allure-pytest 2.12.0 just can't work with allure-commons 2.13.0 (even if we disable allure-pytest plugin with -p no:allure_pytest argument). The fact that you was able to run the tests and get the results also speaks in favor of some issue with your environment.

[delatrie]

Alright, I'm closing this as no response was given in two weeks. But you're always welcome to notify me here on additional details on your issue, if any!