search

getgauge / gauge

Light weight cross-platform test automation
https://gauge.org
Apache License 2.0
2.99k stars 344 forks source link

Gauge list --scenarios may not be returning all scenarios #2312

Open menty666 opened 1 year ago

menty666 commented 1 year ago

Describe the bug I tried to use gauge list --scenarios to list all the scenarios, trying to get a count of how many existed in our project. It gave me 242 scenarios.

We used a powershell script to sweep the directory and get the list, and it yields what may be the real count, at 355. In other words, the command line tool may be missing some scenarios.

To Reproduce Steps (or project) to reproduce the behavior:

  1. Within the project folder, call gauge list --scenarios.
  2. Copy the output into something like Excel or Notepad++ and count note the number of rows/lines.
  3. Issue the following from within the same project folder: 
    
Start-Transcript
#Using a transcript is just a quick and easy way to get an output file

$file_location = "" #Change the location

$spec_files = Get-ChildItem $file_location -Filter "*.spec" -Recurse #gets all the spec files

$scenarios = @() $total_count = 0

this block gets the lines starting with

foreach ($_ in $specfiles) { $FileContent = Get-Content $.FullName foreach ($line in Get-Content $_.FullName) { if ($line -like '##') { $scenarios += $line $total_count += 1 } }

#This line just outputs the list of files that have matches in them, and provides the count per file
$Matches = Select-String -InputObject $FileContent -Pattern "##" -AllMatches

#output the per file data
Write-Host $_.FullName " >>> " $Matches.Matches.Count

}

Output the list of found scenarios

foreach ($ in $scenarios) { Write-Host $ }

Output the count of scnenarios

Write-Host "Total scenarios: $($scenarios.Count)"

Stop-Transcript

4. Note the counts between the official command line tool results and the quick and dirty Powershell script will differ

**Expected behavior**
The counts between the two methods should match

**Screenshots**
N/A

<!--- please complete the following information -->
**Versions:**
<!--- If you are using Gauge please paste the output of "gauge -v"  -->

Gauge version: 1.4.3 Commit Hash: f98dd40

Plugins

html-report (4.1.4) json-report (0.3.7) python (0.3.17) screenshot (0.1.0) xml-report (0.2.3)


 - OS Win10
 - Runner info:  Python 3.10.4
 - IDE info VS Code/Powershell ISE
sriv commented 1 year ago

Thanks for the detailed report @menty666

I tried to replicate this on gauge-tests (https://github.com/getgauge/gauge-tests). Here's what I saw:

 gauge list --scenarios | sort
All level Execution hooks
AND aggregation of hooks
basic context execution
Basic spec with multiple scenarios, passing
Basic spec with one scenario, passing
Capture screenshot using custom screenshot grabber during test execution
Concept calling itself
Concept execution failure
Context steps should appear for all scenarios
Continue execution on any error/exception in "Continue on Failure" step
Continue execution only for mentioned error/exception in "Continue on Failure" step
Continue on failure in a concept step
Continue on failure in a nested concept step
Continue on failure in setup and teardown
Cyclic concept dependency
Datatable driven parallel execution should execute before spec hooks per row
Datatable driven parallel execution with hook failure
Datatable execution with row number
Datatable execution with row range
Duplicate concept definition
Duplicate step implementation should atleast give a warning
Execute a spec from a folder which is not specs
Execute a spec in a sub folder
Execute multiple specs from a folder
Execute specs before parallel execution
Execute specs parallelly
Execute specs parallelly for each datatable
Execute specs parallelly in n streams
Execute specs parallely in n streams
Execute specs sequentially if datatable row values not used
Execute teardown if some other step fails in setup
External table parameter (csv)
Failed teardown execution
Formatting merges scenario with same heading
Formatting should not remove steps, scenarios from spec
Hook failure at scenario level
Hook failure at spec level
Hook failure at step level
Hook failure at suite level
Implementation file in Custom Compile Directory
Implementation files in Multiple Custom Compile Directory
Initializing a new project
Initializing a new project without existing gitignore
Initializing unknown language project
Inline simple parameters
Inline table parameter
Invoke a concept with no parameters
Nested Concept execution
Nested Concept execution with data table
Normal context step - with no cof fails then tear down steps are expected to execute
OR aggregation of hooks
Passing tagged execution
Passing teardown execution
Repeat last run with a failure
Repeat last run with log level debuging
Repeat last run with rerun failed specs after correcting failure
Repeat last run with spec directory
Repeat last successful run
Rerun failed after correcting failure
Rerun failed scenarios
Rerun failed specs after correcting failure
Rerun failed specs with log level debuging
Rerun failed specs with spec directory
Rerun specs with failed flags
Resolve concepts when executing from nested directories in specs
Scenario and step level hooks
Scenario data store refreshes after every scenario run
Scenario execution by specifying line number
Scenario in concepts
[Scenarios]
Scenarios with context and teardown having Unused data table value
Scenarios with context Using data table value
Scenarios with teardown Using data table value
Scenarios with Unused data table value
Scenarios with Using data table values
Scenario with no heading
Screenshot using custom logic
Setting multiple values for package_to_scane env
Setting single value for package_to_scane env
Should retry failing scenario
Simple Datatable Execution
Simple Datatable With special characters
Skip spec if all scenarios are skipped
Spec data store persists data between scenario runs
Spec execution with unimplemented step in context step
Spec execution with unimplemented step in context step and scenario
Spec execution with unimplemented step in context step, scenario and tear down
Spec execution with unimplemented step in scenarios
Spec fails if it has even one scenario failing irrespective of the passing and skipped scenarios in it
Spec run using fail safe does not fail if irrespective of the passing and skipped scenarios in it
Spec should be skipped if it has all scenarios with unimplemented steps
Spec with no heading
step should not be empty
Stop execution on failure due to unspecified errors
Stop execution on failure of some other step
Suite and Spec Level Hooks
Suite data store persists data between spec runs
Tagged expression
Tagged expression with ! operator
Tagged expression with & operator
Tagged expression with (!&|) operator
Tagged expression with | operator
Test concept failure Path
Unimplemented teardown execution
Validation Failure
Validation Success
When a parameter not defined is used in a step it should give a parse error

Ignoring [Scenarios] heading in the output, the line count = 106

Now running your script:

$pwsh /tmp/scenarios.ps1 
<snip>
## Initializing a new project
## Initializing a new project without existing gitignore
## Initializing unknown language project
## Implementation file in Custom Compile Directory
## Implementation files in Multiple Custom Compile Directory
## Setting single value for package_to_scane env
## Setting multiple values for package_to_scane env
## Scenario data store refreshes after every scenario run
## Spec data store persists data between scenario runs
## Suite data store persists data between spec runs
## Execute specs parallelly
## Execute specs parallelly in n streams
## Execute specs before parallel execution
## Execute specs parallelly for each datatable
## Execute specs parallely in n streams
## Execute specs sequentially if datatable row values not used
## Datatable driven parallel execution should execute before spec hooks per row
## Datatable driven parallel execution with hook failure
## Simple Datatable Execution
## Datatable execution with row number
## Datatable execution with row range
## Simple Datatable With special characters
## Scenarios with Unused data table value
## Scenarios with context and teardown having Unused data table value
## Scenarios with Using data table values
## Scenarios with context Using data table value
## Scenarios with teardown Using data table value
## Hook failure at suite level
## Hook failure at spec level
## Hook failure at scenario level
## Hook failure at step level
## Suite and Spec Level Hooks
## Scenario and step level hooks
## All level Execution hooks
## Should retry failing scenario
## Tagged expression with & operator
## Tagged expression with | operator
## Tagged expression with ! operator
## Tagged expression with (!&|) operator
## Tagged expression
## Passing tagged execution
## AND aggregation of hooks
## OR aggregation of hooks
## AND aggregation of hooks
## OR aggregation of hooks
## AND aggregation of hooks
## OR aggregation of hooks
## Continue on failure in a concept step
## Continue on failure in a nested concept step
## Continue on failure in setup and teardown
## Execute teardown if some other step fails in setup
## Continue execution on any error/exception in "Continue on Failure" step
## Stop execution on failure of some other step
## Continue execution only for mentioned error/exception in "Continue on Failure" step
## Stop execution on failure due to unspecified errors
## Rerun failed scenarios
## Rerun failed after correcting failure
## Rerun failed specs after correcting failure
## Rerun failed specs with log level debuging
## Rerun failed specs with spec directory
## Repeat last run with a failure
## Repeat last successful run
## Repeat last run with rerun failed specs after correcting failure
## Repeat last run with log level debuging
## Repeat last run with spec directory
## Rerun specs with failed flags
## Screenshot using custom logic
## Capture screenshot using custom screenshot grabber during test execution
## basic context execution
## Normal context step - with no cof fails then tear down steps are expected to execute
## Context steps should appear for all scenarios
## Passing teardown execution
## Passing teardown execution
## Unimplemented teardown execution
## Failed teardown execution
## Context steps should appear for all scenarios
## Passing teardown execution
## Duplicate step implementation should atleast give a warning
## step should not be empty
## When a parameter not defined is used in a step it should give a parse error
## Basic spec with one scenario, passing
## Basic spec with multiple scenarios, passing
## Spec fails if it has even one scenario failing irrespective of the passing and skipped scenarios in it
## Spec should be skipped if it has all scenarios with unimplemented steps
## Spec run using fail safe does not fail if irrespective of the passing and skipped scenarios in it
## Formatting should not remove steps, scenarios from spec
## Formatting merges scenario with same heading
## Execute multiple specs from a folder
## Execute a spec from a folder which is not specs
## Execute a spec in a sub folder
## Spec execution with unimplemented step in scenarios
## Spec execution with unimplemented step in context step
## Spec execution with unimplemented step in context step and scenario
## Spec with no heading
## Scenario with no heading
## Skip spec if all scenarios are skipped
## Spec execution with unimplemented step in scenarios
## Spec execution with unimplemented step in context step
## Spec execution with unimplemented step in context step, scenario and tear down
## Resolve concepts when executing from nested directories in specs
## Test concept failure Path
## Duplicate concept definition
## Nested Concept execution
## Nested Concept execution with data table
## Concept execution failure
## Invoke a concept with no parameters
## Duplicate concept definition
## Concept calling itself
## Cyclic concept dependency
## Scenario in concepts
   |## Scenario Heading|comment|comment|
## Scenario execution by specifying line number
## Validation Failure
## Validation Success
## Inline table parameter
## External table parameter (csv)
## Inline simple parameters

The line count is 116, ignoring |## Scenario Heading|comment|comment| which is not a scenario, rather a table row.

Looking at the diff:

diff --git a/scenarios.txt b/scenarios.txt
index f2cfff9..814faf5 100644
--- a/scenarios.txt
+++ b/scenarios.txt
@@ -1,5 +1,7 @@
 All level Execution hooks
 AND aggregation of hooks
+AND aggregation of hooks
+AND aggregation of hooks
 basic context execution
 Basic spec with multiple scenarios, passing
 Basic spec with one scenario, passing
@@ -7,6 +9,7 @@ Capture screenshot using custom screenshot grabber during test execution
 Concept calling itself
 Concept execution failure
 Context steps should appear for all scenarios
+Context steps should appear for all scenarios
 Continue execution on any error/exception in "Continue on Failure" step
 Continue execution only for mentioned error/exception in "Continue on Failure" step
 Continue on failure in a concept step
@@ -18,6 +21,7 @@ Datatable driven parallel execution with hook failure
 Datatable execution with row number
 Datatable execution with row range
 Duplicate concept definition
+Duplicate concept definition
 Duplicate step implementation should atleast give a warning
 Execute a spec from a folder which is not specs
 Execute a spec in a sub folder
@@ -49,8 +53,12 @@ Nested Concept execution
 Nested Concept execution with data table
 Normal context step - with no cof fails then tear down steps are expected to execute
 OR aggregation of hooks
+OR aggregation of hooks
+OR aggregation of hooks
 Passing tagged execution
 Passing teardown execution
+Passing teardown execution
+Passing teardown execution
 Repeat last run with a failure
 Repeat last run with log level debuging
 Repeat last run with rerun failed specs after correcting failure
@@ -67,12 +75,12 @@ Scenario and step level hooks
 Scenario data store refreshes after every scenario run
 Scenario execution by specifying line number
 Scenario in concepts
-Scenario with no heading
 Scenarios with context and teardown having Unused data table value
 Scenarios with context Using data table value
 Scenarios with teardown Using data table value
 Scenarios with Unused data table value
 Scenarios with Using data table values
+Scenario with no heading
 Screenshot using custom logic
 Setting multiple values for package_to_scane env
 Setting single value for package_to_scane env
@@ -82,9 +90,11 @@ Simple Datatable With special characters
 Skip spec if all scenarios are skipped
 Spec data store persists data between scenario runs
 Spec execution with unimplemented step in context step
+Spec execution with unimplemented step in context step
 Spec execution with unimplemented step in context step and scenario
 Spec execution with unimplemented step in context step, scenario and tear down
 Spec execution with unimplemented step in scenarios
+Spec execution with unimplemented step in scenarios
 Spec fails if it has even one scenario failing irrespective of the passing and skipped scenarios in it
 Spec run using fail safe does not fail if irrespective of the passing and skipped scenarios in it
 Spec should be skipped if it has all scenarios with unimplemented steps
@@ -96,8 +106,8 @@ Suite and Spec Level Hooks
 Suite data store persists data between spec runs
 Tagged expression
 Tagged expression with ! operator
-Tagged expression with (!&|) operator
 Tagged expression with & operator
+Tagged expression with (!&|) operator
 Tagged expression with | operator
 Test concept failure Path
 Unimplemented teardown execution

So the extra lines are duplicates (or order is mixed up). The duplicate lines are these:

AND aggregation of hooks
AND aggregation of hooks
Context steps should appear for all scenarios
Duplicate concept definition
OR aggregation of hooks
OR aggregation of hooks
Passing teardown execution
Passing teardown execution
Spec execution with unimplemented step in context step

these seem to arise because the same scenario name is used in different specs, and gauge list is filtering it out as duplicate.

This brings a question - in case two specifications have same scenarios, how could it be best shown in the console output? Spec execution with unimplemented step in scenarios

menty666 commented 1 year ago

Thank you for having a look!

It's definitely a real use case, though. Despite asking them not to for reporting reasons, we have Test Developers that will use the same scenario name in multiple files. i.e.

SomeProcess_Desktop.spec SomeProcess_Mobile.spec

They might both contain a test scenario named ## Amount field should only allow numeric input since the functionality is the same, but the platform differs

In that case, if we're looking for the total number of scenarios, and used the gauge list functionality to get the list, we'd definitely want to include both in that list for counting purposes.