Add filename and line no. as context in failed test runs for feature files

[wgwz]

Summary

This issue is a proposal to add context to test failure output for feature files, i.e. filename and line no. of the failed scenario.

Problem Definition

This would improve the dev UX while debugging failed tests. Knowing the failed test you could more easily search for the file, and find the scenario you need to debug. I'm not sure if this is even possible, it is just an idea.

Proposal

Output at present:

        --- FAIL: TestMsgBridgeReceive/MsgBridgeReceive/an_error_is_returned_if_class_id_is_not_formatted (0.00s)
            msg_bridge_receive_test.go:60: 
                        Error Trace:    msg_bridge_receive_test.go:60
                                                                value.go:556
                                                                value.go:339
                                                                run_step.go:98
                                                                run_scenario.go:112
                                                                run_scenario.go:65
                                                                run_doc.go:33
                        Error:          Error message not equal:
                                        expected: "class ID didn't match the format: expected A00, got foo: parse error"
                                        actual  : "class ID didn't match the format: expected A00, got foo: invalid request"
                        Test:           TestMsgBridgeReceive/MsgBridgeReceive/an_error_is_returned_if_class_id_is_not_formatted

Example Desired output:

        --- FAIL: TestMsgBridgeReceive/MsgBridgeReceive/an_error_is_returned_if_class_id_is_not_formatted (0.00s)
            msg_bridge_receive_test.go:60: 
                        Error Trace:    msg_bridge_receive_test.go:60
                                                                value.go:556
                                                                value.go:339
                                                                run_step.go:98
                                                                run_scenario.go:112
                                                                run_scenario.go:65
                                                                run_doc.go:33
                        Error:          Error message not equal:
                                        expected: "class ID didn't match the format: expected A00, got foo: parse error"
                                        actual  : "class ID didn't match the format: expected A00, got foo: invalid request"
                        Test:           TestMsgBridgeReceive/MsgBridgeReceive/an_error_is_returned_if_class_id_is_not_formatted
                        Location:       x/ecocredit/core/features/msg_bridge_receive.feature:59

Note the added Location where x/ecocredit/core/features/msg_bridge_receive.feature:59 leads to:

 59   Scenario: an error is returned if class id is not formatted
 60     Given the message
 61     """
 62     {
 63       "issuer": "regen1depk54cuajgkzea6zpgkq36tnjwdzv4ak663u6",
 64       "class_id": "foo"
 65     }
 66     """
 67     When the message is validated
 68     Then expect the error "class ID didn't match the format: expected A00, got foo: invalid request"

---

For Admin Use

• [ ] Not duplicate issue
• [ ] Appropriate labels applied
• [ ] Appropriate contributors tagged
• [ ] Contributor assigned/self-assigned

[ryanchristo]

Providing the location of the scenario within the feature file might be nice and not something we would implement within regen ledger but rather within https://github.com/regen-network/gocuke.

That being said, this seems a bit redundant of Error Trace and Test:

Error Trace:    msg_bridge_receive_test.go:60

The go test file and the line on which the error was thrown msg_bridge_receive_test.go:60.

Test:           TestMsgBridgeReceive/MsgBridgeReceive/an_error_is_returned_if_class_id_is_not_formatted

The go test that runs the scenarios in the feature file TestMsgBridgeReceive, the feature name MsgBridgeReceive defined in the feature file, and the scenario: an_error_is_returned_if_class_id_is_not_formatted.

I'm going to transfer this issue to the gocuke repository, which is where it would need to be addressed.

[wgwz]

I feel it makes getting to the feature file under test just a bit easier but that said I can see how you can craft the search differently using the feature name.

While the scenario is listed, the fact that it has underscores makes it not something you can copy and paste into a search to get to the scenario in the feature file (which has spaces and not underscores)

[aaronc]

It's a bit hard to retrieve the exact file and line number given the current way that gherkin is transformed into their internal pickle representation. Not impossible, but hard.

What would be pretty easy is outputting literal step text as steps as executed. Would that help?

[wgwz]

Thanks for looking at this! Yup, I think that would satisfy the requirement of wanting some specific text to search for, that brings me directly to scenario under test when there is a failure :-)