eportportal_client.errors.ResponseError: 4065: Cannot finish launch '5e304ff730c49e00015d6341' with calculated automatically status 'FAILED' as 'PASSED'

[testautomation]

TC to reproduce issue

*** Test Cases ***
Failing non-critical TC for reportportal
    [Documentation]     Since this test is marked as 'not-ready' and executed
    ...                 with `--noncritical not-ready` option, it should not
    ...                 impact the overall test result.
    [Tags]    not-ready
    Fail    Expected to FAIL as non-critical

results in error below

==============================================================================
Ci Test Dummy                                                                 
==============================================================================
Failing non-critical TC for reportportal :: Since this test is mar... | FAIL |
Expected to FAIL as non-critical
------------------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/osboxes/projects/ehrbase/tests/venv/lib/python3.7/site-packages/reportportal_client/service_async.py", line 234, in process_item
    getattr(self.rp_client, method)(**kwargs)
  File "/home/osboxes/projects/ehrbase/tests/venv/lib/python3.7/site-packages/reportportal_client/service.py", line 171, in finish_launch
    status=status)
  File "/home/osboxes/projects/ehrbase/tests/venv/lib/python3.7/site-packages/reportportal_client/service.py", line 167, in _finalize_launch
    return _get_msg(r)
  File "/home/osboxes/projects/ehrbase/tests/venv/lib/python3.7/site-packages/reportportal_client/service.py", line 40, in _get_msg
    return _get_data(response)["msg"]
  File "/home/osboxes/projects/ehrbase/tests/venv/lib/python3.7/site-packages/reportportal_client/service.py", line 52, in _get_data
    raise ResponseError(error_messages[0])
reportportal_client.errors.ResponseError: 4065: Cannot finish launch '5e3051fc30c49e00015d6418' with calculated automatically status 'FAILED' as 'PASSED'. Did you provide correct status in request?
Ci Test Dummy                                                         | PASS |
0 critical tests, 0 passed, 0 failed
1 test total, 0 passed, 1 failed
==============================================================================

and has wrong status in RP

Interestingly there is no error if you have the following test cases:

*** Test Cases ***
Failing TC for reportportal
                        Fail    Expected to FAIL for reportportal  *tags

Failing non-critical TC for reportportal
    [Documentation]     Since this test is marked as 'not-ready' and executed
    ...                 with `--noncritical not-ready` option, it should not
    ...                 impact the overall test result.
    [Tags]              not-ready
                        Fail    Expected to FAIL as non-critical

==============================================================================
Ci Test Dummy                                                                 
==============================================================================
Failing TC for reportportal                                           | FAIL |
Expected to FAIL for reportportal
------------------------------------------------------------------------------
Failing non-critical TC for reportportal :: Since this test is mar... | FAIL |
Expected to FAIL as non-critical
------------------------------------------------------------------------------
Ci Test Dummy                                                         | FAIL |
1 critical test, 0 passed, 1 failed
2 tests total, 0 passed, 2 failed
==============================================================================

but still wrong status in RP:

[HardNorth]

Please specify version of RP and agent are you using?

[HardNorth]

Well, consulted with a product owner. I guess you are using RP4. In that case yes, there is a substantial limitation. Please consider using RP5.

[DzmitryHumianiuk]

@testautomation hey, We had no idea that robot framework has this type of test case, I mean non-critical.

In RPv4 server will not allow to finish launch as PASSED if any of failed test occur.

In RPv5 it does accept.

The rest is just a decision how you want to see Non-critical test case after fail. As failed or as passed