Tests are not passed to TeamCity with Bazel remote caching

[konste]

The last fix made things substantially worse, by hiding test failures. When we have a build with some unit tests failed we want to see the build failed with the indication that it failed because of the tests, like this:

What we actually see is a completely green successful build:

with no indication that there is anything wrong with it! This is very misleading and error-prone.

At the very least please make the build fail if not all tests are passing, as it was before the last fix. But ideally, we should see tests failed and passed stats, as in the picture. And this stat should show up during the build, not only after the completion, as it happens for other build runners.

[NikolayPianikov]

Could you please attach a screenshot of the tab Common Failure Conditions.

Before this fix you had a state Tests failed: X, passed: Y; build failed but after this fix you should have Tests failed: X, passed: Y. Build was completed successfully because all tests were passed. It is expected behaviour:

But a build reports that it has failed tests. We had a discussion about that there. But anyway another TeamCity build runners work the same way.

Bazel provides a tests report after a build. You could select diagnostics verbosity level and find original Bazel messages in the build log. So I am not sure it is possible to report tests progress in real-time. But if know about it please share this info

[konste]

The problem is - we have tests failing, but still the build shows Success as in my screenshot above. We want to see Tests failed: X, passed: Y and the build icon red, as in your screenshot, but instead we get Succees with the green checkmark.

In the following screenshot the only difference between the build #453 and #454 is the plugin updated. The test is still failing.

[NikolayPianikov]

Thank you. It looks good. Could you attach a build log file with diagnostics verbosity level?

[NikolayPianikov]

Which version of TeamCity and Bazel are you using?

[konste]

Diag build log attached. TeamCity Enterprise 2019.2.2 (build 71923) Bazel 2.2.0 product_root_Linux_455.log.zip

[NikolayPianikov]

Thanks. Could you attach a screenshot of Overview page for this build?

[konste]

[NikolayPianikov]

So you have no information about tests at all?

[konste]

Unless I open Build Log I have no way to see test failures.

[NikolayPianikov]

And what about passed tests? Do you have Tests tab for the build?

[konste]

No tests tab for the build.

[konste]

To answer your question I am not sure it is possible to report tests progress in real-time. But if know about it please share this info I tried to run the build with --build_event_text_file option and in the resulting file I see events like

event {
id {
  test_result {
    label: "@lego//:test_tabsys"
    run: 1
    shard: 1
    attempt: 1
    configuration {
      id: "1b476220511947b79d025bbb49fa1fdc32ee46aed368223de5aa50720c305857"
    }
  }
}
test_result {
  test_action_output {
    name: "test.log"
    uri: "bytestream://10.176.116.122:2020/blobs/f9fee9b7c9cef38bb056d24e04d67aa7838620ce0a7d89369e29a937226027a1/1714"
  }
  test_action_output {
    name: "test.xml"
    uri: "bytestream://10.176.116.122:2020/blobs/143c15e5d2188abdf0962f9b851a7971d59c589fa7835a0bff77800b288257d5/2151"
  }
  test_attempt_duration_millis: 10840
  status: FAILED
  test_attempt_start_millis_epoch: 1584389752518
  execution_info {
    strategy: "processwrapper-sandbox"
  }
}
}

which come in as tests are passing or failing, while the build is still going, so it is possible to have that info in real time.

[NikolayPianikov]

Are you using Bazel Remote caching? I found a difference between my and your test reports but I'm going to continue investigation tomorrow.

[konste]

Thank you, Nikolay! We started using remote caching last week. I will try without it and report.

[konste]

Indeed it seems to be remote caching which makes the difference! When I turned remote caching off everything test related started to behave as it should. Here is diagnostic log without remote caching: product_root_Linux_457.log.zip Diag log WITH remote caching is attached above (build #455). I hope you can compare those and figure how remote caching disrupts your plugin functionality.

[NikolayPianikov]

@konste please try https://plugins.jetbrains.com/plugin/11248-bazel-build-support/update/80972

[NikolayPianikov]

I've added #10 task