--section-id not working while import junit report

chooiyeewoon commented 5 months ago

TestRail CLI Version

1.8.0

CLI Environment

No response

TestRail Version

8.0.1

TestRail Instance Type

Professional Cloud

Current behavior

I am trying to import Cypress test result (Junit xml report) to TestRail and the test cases to be in a desired section. I specify the --section-id but it doesn't work.

Sample command:

trcli -y \
...
parse_junit \
--section-id "975" \
--suite-id "1" \
--plan-id "408" \
--run-id "417" \
--case-fields "custom_test_type:1" \
-f "reports/junit-report.xml"

Expectation: i want the test case to be located to below section. I did check via API and it returned correctly as per section id of 975 /index.php?/api/v2/get_section/975 Screenshot 2024-01-18 at 5 34 29 PM

What happens now: the test case have been uploaded to other section named Dashboard instead. looks like it is taking the <testsuite name="Dashboard" in my junit report and upload the test case to an existing Dashboard section.

Screenshot 2024-01-18 at 5 39 22 PM

How can i resolve this?

Desired behavior

when specify section id, it should upload the test cases to the specific section instead of matching the testsuite name in Junit xml report.

More Details

No response

dibyaranjan-pg commented 5 months ago

Can we get an update on this as we are kind of blocked because of this issue.

Testinator-X commented 5 months ago

@bitcoder can you check this?

bitcoder commented 4 months ago

Hi @dibyaranjan-pg , I did take some time to look at this and it seems it was discussed earlier on: https://github.com/gurock/trcli/issues/71#issuecomment-1442244893

https://github.com/gurock/trcli/releases/tag/v1.4.3

As the report can contain multiple <testsuite> elements, and each one maps to a section in TR, I'm not fully sure of how this should work. Should it be used to specify the parent section where then the other sections would be created in? I also don't know how this was intended to work initially as it is not very clear

bitcoder commented 4 months ago

Well, I tried the following:

I created a dummy section on a project (having id 111)
I uploaded a Junit XML report using:

trcli -y -h $TR_INSTANCE --project SFEXP -u $TR_USERNAME -p "$TR_PASSWORD" parse_junit -f junit_199.xml --title "results" --section-id 111

it created the frontend, backend,middleendsections under thedummy` section I created and that I specified on the command line.

This was the XML that I used:

<testsuites id="" name="" tests="15" failures="0" skipped="0" errors="0" time="16.813736">
<testsuite name="frontend" timestamp="2023-10-11T03:58:09.528Z" hostname="chromium" tests="5" failures="0" skipped="0" time="12.734" errors="0">
<testcase name="Surat Pesanan, Positive Case @testrail @dummy › [List Surat Pesanan] User able to see button &quot;Tambah&quot;" classname="e2e/dummy/testrail.spec.ts" time="2.858">
</testcase>
<testcase name="Surat Pesanan, Positive Case @testrail @dummy › [Add Surat Pesanan] User able to choose Jenis SP&quot;" classname="e2e/dummy/testrail.spec.ts" time="2.815">
</testcase>
<testcase name="Surat Pesanan, Positive Case @testrail @dummy › [Add Surat Pesanan] User able to choose SP Reguler" classname="e2e/dummy/testrail.spec.ts" time="2.367">
</testcase>
<testcase name="Surat Pesanan, Positive Case @testrail @dummy › [Add Surat Pesanan] User able to see field Apoteker Penanggung Jawab and this field disable to input" classname="e2e/dummy/testrail.spec.ts" time="1.946">
</testcase>
<testcase name="Surat Pesanan, Positive Case @testrail @dummy › [Tambah Surat Pesanan] [Detail Surat Pesanan] User can click button &quot;+&quot;" classname="e2e/dummy/testrail.spec.ts" time="2.748">
</testcase>
</testsuite>
<testsuite name="backend" timestamp="2023-10-11T03:58:09.528Z" hostname="firefox" tests="5" failures="0" skipped="0" time="19.474" errors="0">
<testcase name="Surat Pesanan, Positive Case @testrail @dummy › [List Surat Pesanan] User able to see button &quot;Tambah&quot;" classname="e2e/dummy/testrail.spec.ts" time="4.489">
</testcase>
<testcase name="Surat Pesanan, Positive Case @testrail @dummy › [Add Surat Pesanan] User able to choose Jenis SP&quot;" classname="e2e/dummy/testrail.spec.ts" time="4.54">
</testcase>
<testcase name="Surat Pesanan, Positive Case @testrail @dummy › [Add Surat Pesanan] User able to choose SP Reguler" classname="e2e/dummy/testrail.spec.ts" time="4.058">
</testcase>
<testcase name="Surat Pesanan, Positive Case @testrail @dummy › [Add Surat Pesanan] User able to see field Apoteker Penanggung Jawab and this field disable to input" classname="e2e/dummy/testrail.spec.ts" time="3.187">
</testcase>
<testcase name="Surat Pesanan, Positive Case @testrail @dummy › [Tambah Surat Pesanan] [Detail Surat Pesanan] User can click button &quot;+&quot;" classname="e2e/dummy/testrail.spec.ts" time="3.2">
</testcase>
</testsuite>
<testsuite name="middleend" timestamp="2023-10-11T03:58:09.528Z" hostname="webkit" tests="5" failures="0" skipped="0" time="12.824" errors="0">
<testcase name="Surat Pesanan, Positive Case @testrail @dummy › [List Surat Pesanan] User able to see button &quot;Tambah&quot;" classname="e2e/dummy/testrail.spec.ts" time="2.619">
</testcase>
<testcase name="Surat Pesanan, Positive Case @testrail @dummy › [Add Surat Pesanan] User able to choose Jenis SP&quot;" classname="e2e/dummy/testrail.spec.ts" time="3.479">
</testcase>
<testcase name="Surat Pesanan, Positive Case @testrail @dummy › [Add Surat Pesanan] User able to choose SP Reguler" classname="e2e/dummy/testrail.spec.ts" time="2.698">
</testcase>
<testcase name="Surat Pesanan, Positive Case @testrail @dummy › [Add Surat Pesanan] User able to see field Apoteker Penanggung Jawab and this field disable to input" classname="e2e/dummy/testrail.spec.ts" time="1.028">
</testcase>
<testcase name="Surat Pesanan, Positive Case @testrail @dummy › [Tambah Surat Pesanan] [Detail Surat Pesanan] User can click button &quot;+&quot;" classname="e2e/dummy/testrail.spec.ts" time="3">
</testcase>
</testsuite>

Therefore, it seems that as of 1.9.0 the CLI is created sections under the specified section (by its id) and adding the cases there; the --section-id is used to specify the parent folder where all sections (which are derived from the <testsuite> elemnts) will be created.

I think this clarifies the current behaviour, which for me makes sense at first sight. Let me know you our thoughts.

chooiyeewoon commented 4 months ago

Hi @bitcoder, thank you for reply.

Yes the dummy sample you did is as expected. Sorry, maybe i am not being clear enough for my case.

Hope the image above helps.

Test cases are not uploaded to the section which i specify (section id = 975), where i expect it should create a Dashboard folder (with test cases created) under E2E Test Daily Run folder. correct?

now it seems that it look for existing section with name Dashboard and upload the test cases there. So in this case the section-id is being ignore.

my sample xml file.

<testsuites tests="2" failures="0" errors="0" skipped="0" time="24.799"><testsuite name="Root Suite" timestamp="2024-01-15T08:42:53" tests="0" file="cypress/e2e/smokeTests/desktop.cy.js" time="0" failures="0" errors="0" skipped="0">
  </testsuite>
  <testsuite name="Agent - Smoke test" timestamp="2024-01-15T08:42:53" tests="0" time="0" failures="0" errors="0" skipped="0">
  </testsuite>
  <testsuite name="Dashboard" timestamp="2024-01-15T08:43:02" tests="2" time="24.799" failures="0" errors="0" skipped="0">
    <testcase name="Agent - Smoke test Dashboard View subscription page:             - verify H1, url" time="15.103" classname="View subscription page:             - verify H1, url">
    </testcase>
    <testcase name="Agent - Smoke test Dashboard View credit usage history page:             - verify H1, url, table headers             - able filter data" time="9.696" classname="View credit usage history page:             - verify H1, url, table headers             - able filter data">
    </testcase>
  </testsuite>
</testsuites>

i tried upload other xml with test suite name that is not exists, with section-id=975 and it able to create/upload correctly. Let me know if i missed out anything.

bitcoder commented 4 months ago

thanks @chooiyeewoon , I confirm the behaviour you described and I was able to replicate it. If a section with the same name as you have in the testsuite element, then it will be used instead of the section you specify by id on the command line. Help would be appreciated to fix this in the code.

bitcoder commented 4 months ago

I made a very preliminary fix but it needs:

more testing
adding automated tests for it

It's a part of the code that I don't have a full understanding, so it may be not the best option.. needs more time.

You can try it out:

python3 -m pip uninstall trcli -y 
python3 -m pip install dist/trcli-1.9.0.tar.gz

trcli-1.9.0.tar.gz

What I tested (using the sample Junit XML you provided):

uploading to an existing section by its id: it created a "Dashboard" section under that section, and created the tests there
uploading without specifying the section id, and without having an existing Dashboard section: created a "Dashboard" section on the root
uploading without specifying the section id, and having an existing Dashboard section in the root: used the existing "Dashboard" section on the root and added the test cases to it
uploading without specifying the section id, and having an existing Dashboard section inside some other section: used the existing "Dashboard" section and added the test cases to it

chooiyeewoon commented 4 months ago

hi @bitcoder, great! i tried and it created under section that specified now. take your time, looking forward for a permanent fix. Thanks!

gurock / trcli