testFailed message is not recognized as test failure when block is used

[cielavenir]

(Due to https://github.com/JetBrains/teamcity-messages/issues/230 , I had to decipher https://github.com/JetBrains/teamcity-messages/blob/master/tests/integration-tests/unittest_integration_test.py and got the raw command line python3 tests/guinea-pigs/unittest/subtest_named.py.)

From subtest_named I got the following teamcity message:

##teamcity[testCount timestamp='2020-03-27T14:45:59.159' count='1']
##teamcity[testStarted timestamp='2020-03-27T14:45:59.159' captureStandardOutput='true' flowId='__main__.NumbersTest.test_even (Test that numbers between 0 and 5 are all even_)' name='__main__.NumbersTest.test_even (Test that numbers between 0 and 5 are all even_)']
##teamcity[blockOpened timestamp='2020-03-27T14:45:59.159' flowId='__main__.NumbersTest.test_even (Test that numbers between 0 and 5 are all even_)' name='(i=0)' subTestResult='Success']
##teamcity[blockClosed timestamp='2020-03-27T14:45:59.159' flowId='__main__.NumbersTest.test_even (Test that numbers between 0 and 5 are all even_)' name='(i=0)']
##teamcity[blockOpened timestamp='2020-03-27T14:45:59.160' flowId='__main__.NumbersTest.test_even (Test that numbers between 0 and 5 are all even_)' name='(i=1)' subTestResult='Failure']
##teamcity[testStdErr timestamp='2020-03-27T14:45:59.160' flowId='__main__.NumbersTest.test_even (Test that numbers between 0 and 5 are all even_)' name='__main__.NumbersTest.test_even (Test that numbers between 0 and 5 are all even_)' out='SubTest failure: Traceback (most recent call last):|n  File "/home/mujin/mujin/jhbuildappteachworker/install/lib/python3.6/unittest/case.py", line 59, in testPartExecutor|n    yield|n  File "/home/mujin/mujin/jhbuildappteachworker/install/lib/python3.6/unittest/case.py", line 519, in subTest|n    yield|n  File "tests/guinea-pigs/unittest/subtest_named.py", line 16, in test_even|n    self.assertEqual(i % 2, 0)|n  File "/home/mujin/devel/teamcity-messages/teamcity/diff_tools.py", line 39, in _patched_equals|n    raise native_error|n  File "/home/mujin/devel/teamcity-messages/teamcity/diff_tools.py", line 32, in _patched_equals|n    old(self, first, second, msg)|n  File "/home/mujin/mujin/jhbuildappteachworker/install/lib/python3.6/unittest/case.py", line 821, in assertEqual|n    assertion_func(first, second, msg=msg)|n  File "/home/mujin/mujin/jhbuildappteachworker/install/lib/python3.6/unittest/case.py", line 814, in _baseAssertEqual|n    raise self.failureException(msg)|nAssertionError: 1 != 0|n|n']
##teamcity[blockClosed timestamp='2020-03-27T14:45:59.161' flowId='__main__.NumbersTest.test_even (Test that numbers between 0 and 5 are all even_)' name='(i=1)']
##teamcity[testFailed timestamp='2020-03-27T14:45:59.161' details='Failed subtests list: (i=1)' flowId='__main__.NumbersTest.test_even (Test that numbers between 0 and 5 are all even_)' message='One or more subtests failed' name='__main__.NumbersTest.test_even (Test that numbers between 0 and 5 are all even_)']
##teamcity[testFinished timestamp='2020-03-27T14:45:59.161' duration='1' flowId='__main__.NumbersTest.test_even (Test that numbers between 0 and 5 are all even_)' name='__main__.NumbersTest.test_even (Test that numbers between 0 and 5 are all even_)']

I made a teamcity job just to print this and running it caused the test passing. Teamcity summary definitely should show test failure.

test log in https://www.dropbox.com/sh/tvugn3zeebrhy8x/AABHy75s6Vu2Pflzl5HPslIya?dl=0

[cielavenir]

Forgot to mention that I use TeamCity Enterprise 9.0.4 (build 32407)

[cielavenir]

Hi, I do have issues that I cannot utilize teamcity-messages for reporting tests if I print something in fixtures, which creates teamcity-message-block.

[cielavenir]

How is this going? Pasting the teamcity message needs to report failure. Simple, isn't it?

If this is fixed in latest teamcity, could you provide an (command-line) option not to make block if test fixture has output? Currently I need to sed pytest_plugin.py, which is pretty bad idea.

[cielavenir]

with #232 I don't have to sed-patch pytest_plugin.py at least.

[madlexa]

Could you add a simple project with the problem?

[cielavenir]

I did paste the log for the project. As I'm not an admin of our teamcity server, I don't know how to post it here... Like this?

<?xml version="1.0" encoding="UTF-8"?>
<build-type xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" uuid="eef53c24-0255-4a9e-84bf-278be8b16c08" xsi:noNamespaceSchemaLocation="http://www.jetbrains.com/teamcity/schemas/9.0/project-config.xsd">
  <name>test-newline</name>
  <description />
  <settings>
    <parameters />
    <build-runners>
      <runner id="RUNNER_258" name="test" type="simpleRunner">
        <parameters>
          <param name="script.content"><![CDATA[cat subtest_named.log]]></param>
          <param name="teamcity.step.mode" value="default" />
          <param name="use.custom.script" value="true" />
        </parameters>
      </runner>
    </build-runners>
    <vcs-settings>
      <vcs-entry-ref root-id="aaa" />
    </vcs-settings>
    <requirements />
    <build-triggers />
    <cleanup />
  </settings>
</build-type>

[cielavenir]

subtest_named.log exists above "I got the following teamcity message"

[madlexa]

I'm sorry, I spelled it wrong. Could you give me 1 simple test to add to this library test base?

[cielavenir]

Well, could you tell me how to interact teamcity server from teamcity-messages test suite?

The thing is I don't know whether teamcity-messages writes unexpected message or teamcity server interpretation is wrong. (I posted the issue to teamcity-messages in case of former). I personally don't think that teamcity-message itself cannot judge the message itself's rightfulness (unless interacting with teamcity server).

[madlexa]

TeamCity server parses output from a build and searches special messages #teamcity[...] for logic. teamcity-messages only provides the output.

[cielavenir]

again, TC server does say success for the subtest_named test output even though it contains testFailed. Is this expected?

[madlexa]

I will check it out. Have you tested this on the new TeamCity version? Could you give me a python test with this error?

[cielavenir]

sorry, currently I can only access to TC 9.0.4.

The command is python3 tests/guinea-pigs/unittest/subtest_named.py, which is available at https://github.com/JetBrains/teamcity-messages/blob/master/tests/guinea-pigs/unittest/subtest_named.py .