Unable to parse field title in TestRailCase tag. Title is empty.

marksaxer-cbs commented 3 months ago

TestRail CLI Version

1.9.4

CLI Environment

Python 3.11.8

TestRail Version

v8.0.3 Default 3070

TestRail Instance Type

Enterprise Cloud

Current behavior

If a junit-report.xml file contains a testcase name that exceeds 100 characters, the entire results upload fails.

Command

> trcli -y -h <host> --project '<project>' -u 'test.user@email.com' -k $API_TOKEN \
  parse_junit \
  --suite-id 123 \
  --run-id 1234 \
  -f "junit-report.xml"
TestRail CLI v1.9.4
Copyright 2024 Gurock Software GmbH - www.gurock.com
Parser Results Execution Parameters
> Report file: junit-report.xml
> Config file: None
> TestRail instance: https://host/ (user: test.user@email.com)
> Project:<project>
> Run title: None
> Update run: 1234
> Add to milestone: No
> Auto-create entities: True
Parsing JUnit report.
Unable to parse field title in TestRailCase tag. Title is empty.

Contents of "junit-report.xml"

<?xml version="1.0" encoding="utf-8"?>
  <testsuites><testsuite name="pytest" errors="0" failures="0" skipped="0" tests="1" time="1.888" timestamp="2024-03-11T15:40:16.888055" hostname="BLAHBLAHBLAH">
    <testcase classname="services.service.tests.boring_test" name="test_boring_123456789009876543211234567890098765432112345678900987654321123456789009876543211234567890" time="1.871">
    <properties>
      <property name="testrail_attachment" value="logs/service/boring_test/test_boring__2024_03_11T15_40_16.log" />.</properties>
    </testcase>
  </testsuite>
</testsuites>

The failure is thrown at: https://github.com/gurock/trcli/blob/62f87f2943bb649737c252c1354f5a366d65fc0d/trcli/readers/junit_xml.py#L167-L174

because TestRailCaseFieldsOptimizer.extract_last_words will return an empty string when case_name does not contain any delimiters [' ', '\t', ';', ':', '>', '/', '.'] and exceeds MAX_TESTCASE_TITLE_LENGTH or 100 characters.

It appears that, at least in the version we are using, TestRail UI accepts more than 100 characters for a test case title.

Desired behavior

I would like to be able to increase this limit or modify extract_last_words to always return a string with a length > 1. This could be accomplished either by:

including the _ underscore character in the delimiters list
increasing the value of MAX_TESTCASE_TITLE_LENGTH to match the users version of TestRail max title length
use the last MAX_TESTCASE_TITLE_LENGTH characters within the string if the string exceeds MAX_TESTCASE_TITLE_LENGTH

If those solutions are rejected, I'd like to implement:

better error handling to inform the user of the character limit
continue-on-error in order to upload as many results as possible

More Details

No response

marksaxer-cbs commented 3 months ago

This ticket duplicates #207 and was addressed in #209 by @bitcoder. However, I think the solution could be improved further. I am happy to pick up the work if we have a consensus on the solution

bitcoder commented 3 months ago

Hi @marksaxer-cbs , thanks for reporting! I agree with:

including the _ underscore character in the delimiters list
if the previous delimiters don't work, then as a fallback make a substring of it (i.e. use the last MAX_TESTCASE_TITLE_LENGTH characters within the string if the string exceeds MAX_TESTCASE_TITLE_LENGTH)

A PR would be appreciated. Please add a test(s) for it :)

mist3r3 commented 3 months ago

We have the exact same issue, The problem as well is that the error thrown is a bit irrelevant... "Unable to parse field title in TestRailCase tag. Title is empty."

pytest by default add "_" in delimiters..

bitcoder commented 3 months ago

After thinking a bit more I think just following the next approach works better:

use the last MAX_TESTCASE_TITLE_LENGTH characters within the string if the string exceeds MAX_TESTCASE_TITLE_LENGTH

@mist3r3 @marksaxer-cbs , Can you please both of you try this version and give some feedback on it?

trcli-1.9.5.tar.gz

python -m pip uninstall trcli -y 
python3 -m pip install trcli-1.9.5.tar.gz

bitcoder commented 3 months ago

feedback would be appreciated to assess if my fix addresses your needs

marksaxer-cbs commented 3 months ago

The fix looks good to me and the implemented solution, in my opinion, is more ideal than adding _ to the delimiters list.

mist3r3 commented 3 months ago

@bitcoder it's working for me, Thanks!! :)

mist3r3 commented 3 months ago

@bitcoder actually now that I checked it, I get truncated test case names...

bitcoder commented 3 months ago

@mist3r3 truncating names is expectable according with what was discussed above. What is the content of your element on the JUnit xml report and the title you're getting...and the name you would like to have?

mist3r3 commented 3 months ago

Well I get for example testing_this_as_an_example_for_testrail_trcli_in_github_test_test_test_tes_test it is truncating the beginning.. so I would get sting_this_as_an_example_for_testrail_trcli_in_github_test_test_test_tes_test as the test case name...

I would expect to see the whole test case name... At least in the UI.

bitcoder commented 3 months ago

@mist3r3 can you pls share your JUnit XML report?

mist3r3 commented 3 months ago

test-report.xml.zip

bitcoder commented 3 months ago

Thanks for sharing @mist3r3 . However, in that report,

the test you mentioned isn't there; there are other ones of course
all the testcases on that report are imported to TestRail and each test case title has the last 100 characters of the original test case name; so it looks working as expected . note that the idea is to not hit the limit of the title attribute on the system

The limit of the title at the system level is 250 chars. Eventually I can align the limit where TRCLI considers just the ending chars, from 100 chars to the 250 chars. I really don't like long test names but that's just my opinion; I can make the CLI aligned with the current implement limitation on the backend. Sounds good? @marksaxer-cbs @mist3r3 if so, I'll adjust the PR

bitcoder commented 3 months ago

Well, I've updated the PR. Can you please try this one? @mist3r3 , meanwhile I resubmitted the junit report you provided where I also did a small change.

trcli-1.9.5.tar.gz

bitcoder commented 3 months ago

I'm assuming it works for you right?

gurock / trcli