catchorg / Catch2

A modern, C++-native, test framework for unit-tests, TDD and BDD - using C++14, C++17 and later (C++11 support is in v2.x branch, and C++03 on the Catch1.x branch)
https://discord.gg/4CWS9zD
Boost Software License 1.0
18.29k stars 3.01k forks source link

-r console report displays BDD-style test results out of order #886

Open wfs opened 7 years ago

wfs commented 7 years ago

Description

  1. What I did

  2. What I wanted to happen Output to be in correct order that matches SCENARIO test code. Note : "-r xml" param outputs in correct order.

    $ ./udacity_cpp_for_programmers  -r console -d yes --order lex "Scenario: vectors can be sized and resized"
    0.000 s: Scenario: vectors can be sized and resized
    0.000 s:    Given: A vector with some items
    0.000 s:     When: the size is increased
    0.000 s:     Then: the size and capacity change
    0.000 s: Scenario: vectors can be sized and resized
    0.000 s:    Given: A vector with some items
    0.000 s:     When: the size is reduced
    0.000 s:     Then: the size changes but not capacity
    0.000 s: Scenario: vectors can be sized and resized
    0.000 s:    Given: A vector with some items
    0.000 s:     When: more capacity is reserved
    0.000 s:     Then: the capacity changes but not the size
    0.000 s: Scenario: vectors can be sized and resized
    0.000 s:    Given: A vector with some items
    0.000 s:     When: less capacity is reserved
    0.000 s:     Then: neither size nor capacity are changed
    ===============================================================================
    All tests passed (16 assertions in 1 test case)
  3. What actually happened Output was out of order ...

0.000 s:     Then: the size and capacity change
0.000 s:     When: the size is increased
0.000 s:    Given: A vector with some items
0.000 s: Scenario: vectors can be sized and resized
0.000 s:     Then: the size changes but not capacity
0.000 s:     When: the size is reduced
0.000 s:    Given: A vector with some items
0.000 s: Scenario: vectors can be sized and resized
0.000 s:     Then: the capacity changes but not the size
0.000 s:     When: more capacity is reserved
0.000 s:    Given: A vector with some items
0.000 s: Scenario: vectors can be sized and resized
0.000 s:     Then: neither size nor capacity are changed
0.000 s:     When: less capacity is reserved
0.000 s:    Given: A vector with some items
0.000 s: Scenario: vectors can be sized and resized
===============================================================================
All tests passed (16 assertions in 1 test case)

Steps to reproduce

horenmar commented 7 years ago

This is ~likely~ caused by how -d yes is handled.