search

testomatio / reporter

Testomatio Reporter
123 stars 21 forks source link

Reporter does not read all cases from NUnit report xml file #346

Closed JanVokroj closed 2 months ago

JanVokroj commented 4 months ago

Describe the bug When Nunit generates report, where some cases are in nested suite. After use of native testomat reporter tool npx report-xml "results.xml" --lang=c#. Testomat web UI shows only cases in the deepest test suite of xml.

Steps to reproduce

  1. Use Specflow C# to generate C# code readable by NUnit from gherkin feature file, where file contains few normal scenarios and one scenario outline with examples.
  2. Run NUnit console with generated tests library
  3. After run is finished, upload report via testomat reporter tool npx report-xml "results.xml" --lang=c#

Expected behavior All cases are reported to testomat run. No matter in which suite they are.

Screenshots

Feature file example image

Current xml report contains 7 cases but only 3 are reported image

This is how it looks in testomat UI image

xml report

`<?xml version="1.0" encoding="utf-8" standalone="no"?>

Loading plugin \LivingDoc.SpecFlowPlugin.dll Output was deleted due to sensitive information ]]> done: LoginContext.ILoginWithUsernameAndPassword("*admin", "1234") (1.4s) Given I navigate to settings page "Credit" -> done: NavigationSteps.INavigateToSettingsPage("Credit") (0.7s) And I disable credit -> done: SettingsCreditContext.IDisableCredit() (0.0s) And I navigate to settings page "Projects" -> done: NavigationSteps.INavigateToSettingsPage("Projects") (0.3s) And I disable projects -> done: SettingsProjectsContext.IDisableProjects() (0.0s) And I navigate to settings page "Quota" -> done: NavigationSteps.INavigateToSettingsPage("Quota") (0.5s) And I disable quota -> done: SettingsQuotaContext.IDisableQuota() (0.0s) When I navigate to main menu page "Queues" -> done: NavigationSteps.INavigateToMainMenuPage("Queues") (0.4s) When I create queue --- table step argument --- | property | value | | name | queue | -> done: QueuesContext.ICreateQueue() (4.0s) When I select queue "queue" -> done: QueuesContext.ISelectQueue("queue") (0.4s) And I click on dropdown list "ref:queues.actions" to command "Edit" -> done: DropdownSteps.IClickOnDropdownListToCommand("ref:queues.actions", "Edit") (1.2s) And I click to "data:tab1" -> done: ClickSteps.IClickTo("data:tab1") (0.1s) When I select from list "prop:ud_type" value "1" -> done: SelectListSteps.ISelectFromListValue("prop:ud_type", "1") (0.5s) And I set group state for all groups to "expanded" -> done: GroupSteps.ISetGroupStateForAllGroupsTo("expanded") (2.0s) Then I have checkbox "ref:queues.receiving.mobilePrint.enabled" "editable" -> done: CheckboxSteps.IHaveCheckboxReadonly("ref:queues.receiv...", "editable") (0.0s) When I select from list "prop:ud_type" value "256" -> done: SelectListSteps.ISelectFromListValue("prop:ud_type", "256") (0.1s) And I set group state for all groups to "expanded" -> done: GroupSteps.ISetGroupStateForAllGroupsTo("expanded") (2.0s) Then I have checkbox "ref:queues.receiving.mobilePrint.enabled" "readonly" -> done: CheckboxSteps.IHaveCheckboxReadonly("ref:queues.receiv...", "readonly") (0.0s) ##teamcity[testFinished name='|[C211256|] Allow mobile print behavior' duration='16222'] ]]> done: LoginContext.ILoginWithUsernameAndPassword("*admin", "1234") (2.2s) When I navigate to main menu page "Queues" -> done: NavigationSteps.INavigateToMainMenuPage("Queues") (0.3s) When I select queue "Email" -> done: QueuesContext.ISelectQueue("Email") (0.3s) And I click on dropdown list "ref:queues.actions" to command "Edit" -> done: DropdownSteps.IClickOnDropdownListToCommand("ref:queues.actions", "Edit") (0.5s) Then I "do not have" element "ref:queue.desktopClient" presented -> done: GeneralSteps.IHaveElementPresented("do not have", "ref:queue.desktop...") (0.0s) When I create queue --- table step argument --- | property | value | | name | queue | -> done: QueuesContext.ICreateQueue(
) (4.7s) And I select queue "queue" -> done: QueuesContext.ISelectQueue("queue") (0.8s) And I click on dropdown list "ref:queues.actions" to command "Edit" -> done: DropdownSteps.IClickOnDropdownListToCommand("ref:queues.actions", "Edit") (1.0s) And I switch queue properties tab to " client" -> done: QueuesContext.ISwitchQueuePropertiesTabTo(" client") (0.9s) Then I "have" element "ref:queue.desktopClient" presented -> done: GeneralSteps.IHaveElementPresented("have", "ref:queue.desktop...") (0.0s) ##teamcity[testFinished name='|[C211257|] Desktop Client tab presence' duration='14645'] ]]> done: LoginContext.ILoginWithUsernameAndPassword("*admin", "1234") (3.6s) Given I navigate to main menu page "Queues" -> done: NavigationSteps.INavigateToMainMenuPage("Queues") (0.5s) When I create queue --- table step argument --- | property | value | | name | queueBeforeEdit | | type | 1 | | priority | 2 | | protocol | 3 | | port | 9588 | | use fiery if available | true | -> done: QueuesContext.ICreateQueue(
) (8.4s) Then I have queue --- table step argument --- | property | value | | name | queueBeforeEdit | | type | 1 | | priority | 2 | | protocol | 3 | | port | 9588 | | use fiery if available | true | -> done: QueuesContext.IHaveQueue(
) (0.2s) When I click to "cmd:ClosePane" -> done: ClickSteps.IClickTo("cmd:ClosePane") (0.8s) When I reload current page -> done: BrowserSteps.IReloadCurrentPage() (0.9s) When I select from context menu cmd "Edit" on row with id from table "tblqueues:name:queueBeforeEdit" -> done: DatagridViewSteps.ISelectFromContextMenuCmdOnRowWithIdFromTable("Edit", "tblqueues:name:qu...") (1.1s) And I wait until properties tab is opened -> done: BrowserSteps.IWaitUntlilPropertiesTabIsOpened() (0.4s) Then I "have" focus on "prop:name" -> done: TypeTextSteps.IHaveFocusOn("have", "prop:name") (0.0s) When I edit queue with name "queueBeforeEdit" --- table step argument --- | property | value | | name | queueAfterEdit | | type | 0 | | priority | 1 | | protocol | 2 | | port | 9598 | | use fiery if available | false | -> done: QueuesContext.IEditQueueWithName("queueBeforeEdit",
) (7.5s) Then I have queue --- table step argument --- | property | value | | name | queueAfterEdit | | type | 0 | | priority | 1 | | protocol | 2 | | port | 9598 | | use fiery if available | false | -> done: QueuesContext.IHaveQueue(
) (0.1s) ##teamcity[testFinished name='|[C30918|] Edit queue' duration='26678'] ]]> done: LoginContext.ILoginWithUsernameAndPassword("*admin", "1234") (3.5s) Given I navigate to main menu page "Queues" -> done: NavigationSteps.INavigateToMainMenuPage("Queues") (0.5s) When I create queue --- table step argument --- | property | value | | name | queue | -> done: QueuesContext.ICreateQueue(
) (4.7s) When I select queue "queue" -> done: QueuesContext.ISelectQueue("queue") (0.4s) When I click on dropdown list "ref:queues.actions" to command "Edit" -> done: DropdownSteps.IClickOnDropdownListToCommand("ref:queues.actions", "Edit") (0.9s) When I switch queue properties tab to "Rights" -> done: QueuesContext.ISwitchQueuePropertiesTabTo("Rights") (1.0s) When I click to "cmd:AddEntity" -> done: ClickSteps.IClickTo("cmd:AddEntity") (0.6s) Then I can add rights use queue for "All users" -> done: QueuesContext.HaveQueueWithPrinter("All users") (0.1s) ##teamcity[testFinished name='Check values of Current Users List (All users)' duration='14504'] ]]> done: LoginContext.ILoginWithUsernameAndPassword("*admin", "1234") (3.3s) Given I navigate to main menu page "Queues" -> done: NavigationSteps.INavigateToMainMenuPage("Queues") (0.4s) When I create queue --- table step argument --- | property | value | | name | queue | -> done: QueuesContext.ICreateQueue(
) (4.5s) When I select queue "queue" -> done: QueuesContext.ISelectQueue("queue") (0.5s) When I click on dropdown list "ref:queues.actions" to command "Edit" -> done: DropdownSteps.IClickOnDropdownListToCommand("ref:queues.actions", "Edit") (1.0s) When I switch queue properties tab to "Rights" -> done: QueuesContext.ISwitchQueuePropertiesTabTo("Rights") (0.7s) When I click to "cmd:AddEntity" -> done: ClickSteps.IClickTo("cmd:AddEntity") (0.3s) Then I can add rights use queue for "Managers" -> done: QueuesContext.HaveQueueWithPrinter("Managers") (0.1s) ##teamcity[testFinished name='Check values of Current Users List (Managers)' duration='13356'] ]]> done: LoginContext.ILoginWithUsernameAndPassword("*admin", "1234") (2.5s) Given I navigate to main menu page "Queues" -> done: NavigationSteps.INavigateToMainMenuPage("Queues") (0.4s) When I create queue --- table step argument --- | property | value | | name | queue | -> done: QueuesContext.ICreateQueue(
) (4.8s) When I select queue "queue" -> done: QueuesContext.ISelectQueue("queue") (0.5s) When I click on dropdown list "ref:queues.actions" to command "Edit" -> done: DropdownSteps.IClickOnDropdownListToCommand("ref:queues.actions", "Edit") (1.0s) When I switch queue properties tab to "Rights" -> done: QueuesContext.ISwitchQueuePropertiesTabTo("Rights") (0.6s) When I click to "cmd:AddEntity" -> done: ClickSteps.IClickTo("cmd:AddEntity") (0.4s) Then I can add rights use queue for "Unclassified" -> done: QueuesContext.HaveQueueWithPrinter("Unclassified") (0.1s) ##teamcity[testFinished name='Check values of Current Users List (Unclassified)' duration='12836'] ]]> done: LoginContext.ILoginWithUsernameAndPassword("*admin", "1234") (2.9s) Given I navigate to main menu page "Queues" -> done: NavigationSteps.INavigateToMainMenuPage("Queues") (0.4s) When I click on dropdown list "ref:queues.tools" to command "SaveAsCsv" -> done: DropdownSteps.IClickOnDropdownListToCommand("ref:queues.tools", "SaveAsCsv") (0.8s) When I wait until all background tasks are completed -> done: BrowserSteps.IWaitUntilAllBackgroundTasksAreCompleted() (3.1s) When I wait until downloads are completed -> done: ReportsContext.IWaitUntilDownloadsAreCompleted() (5.0s) Then I don't have any alert -> done: AlertSteps.IDontHaveAnyAlert() (0.2s) Then I should not see any error message -> done: ErrorMessageSteps.IShouldNotSeeAnyErrorMessage() (0.2s) Then I have "1" files in download folder with pattern "grid*" and size bigger than "1" bytes -> done: DownloadFolderSteps.IHaveFilesInDownloadFolder(1, "grid*", 1) (0.0s) ##teamcity[testFinished name='Save Queues' duration='14677'] ]]> `
DavertMik commented 4 months ago

@JanVokroj I just released a new reporter version

Please try to install 

"@testomatio/reporter": "1.4.2-beta-specflow-compat.1"

With this version

  • nesting of suites should work
  • using properties to detect names and tags for scenarios
  • correctly passing examples into parametrized tests
  • tested on your example

Please try it

JanVokroj commented 4 months ago

@DavertMik Thank you for quick reaction. I just tried, looks much better. All cases are reported. But now the ones which are scenario outlines with examples are marked as skipped and contains some reporter/runner error. image Here is my cmd : image

DavertMik commented 4 months ago

Thanks, @JanVokroj Good catch

Could you check this release

@testomatio/reporter@1.4.2-beta-specflow-compat.2
JanVokroj commented 4 months ago

Hi @DavertMik. Yep, it works now with the file I provided. xml is reported as expected, Thank you very much. But unfortunately I found another xml report. Where I'm facing the original issue :( That report was created by run of two feature files. So I think reason is, that xml file contains two 'main' suites ( TestFixtures ), where the second has nested suite with paramerized cases. Those are not reported.

xml report look like this. Everything except refreshing Filtered Log Cause Server Communication Error is reported. Whole xml contains 22 cases, but only 13 are reported. image

Here is how it looks in testomat image

xml file ( extension was renamed to .log ) log.log

DavertMik commented 4 months ago

Ok, I'm almost ready to present a fix for this

image

@JanVokroj please try version 1.4.2 which is going to be released in next hour

@testomatio/reporter@1.4.2
JanVokroj commented 4 months ago

@DavertMik , Looks very good now. I tried all small runs we have and all are reported correctly. Tomorrow I'll try just few more cases and let you know.

JanVokroj commented 3 months ago

Hello, I'm sorry for late response. But unfortunately I found another reports which are not uploaded correctly :(. Seems to me, there could be different approach how to load/parse xml, some recursion to get all test-case tags and then it's parents. Like going from the deepest to the top. But I'm really not able to determinate if that approach would be fine. I'm sure that your approach, from top to the deepest has also good reasons. Anyway, here is the acceptance.xml which gaves me 0 cases uploaded to testomat, but contains cca 275 case ( including scenario outlines, where one example is counted as one test-case ) acceptance.log Another report missing one case of each 'suite' and 'cases' are marked as retries. image But If I open the 'suite' the amount of examples is correct ( so this might be not related ...) image Policy.log

I'll try few more reports...

JanVokroj commented 3 months ago

I found another one :(. I don't know maybe I'm doing something wrong. In case you need any additional information, please, let me know. Right now I'm using @testomatio/reporter@1.4.2 This xml report contains about 85 cases total, but only last 13 are reported to testomat. language.log image

TetianaKhomenko commented 3 months ago

@DavertMik please take a look at the new details added by the user.

DavertMik commented 2 months ago

@JanVokroj I tested all provided cases and published new update for reporter to handle them

Please check latest reporter which is 1.4.7

https://github.com/testomatio/reporter/releases/tag/1.4.7

TetianaKhomenko commented 2 months ago

Hi @JanVokroj! We are closing this ticket as done (please see comment above). If there are any other issues please create new tickets.

JanVokroj commented 1 month ago

Hello @TetianaKhomenko , We will test in following weeks and let you know in case of any troubles. Thank you

  • © Githubissues.
  • Githubissues is a development platform for aggregating issues.