tcplugins / tcWebHooks

WebHooks plugin for Teamcity. Supports many build states and payload formats.
https://netwolfuk.wordpress.com/category/teamcity/tcplugins/tcwebhooks/
157 stars 30 forks source link

Muted test webhook doesn't include test name #219

Closed matt-richardson closed 1 year ago

matt-richardson commented 1 year ago

I'm checking out the new test muted and test unmuted events, to see how we can tie this into our other systems.

For unmuted, we get the test name, but for muted, we dont get the test name.

Expected Behavior

buildEventType | TESTS_UNMUTED
mutedOrUnmutedTests | {4=[suiteName: nestedSuiteName: package_or_namespace.ClassName.TestName]}
buildEventType | TESTS_MUTED
mutedOrUnmutedTests | {4=[suiteName: nestedSuiteName: package_or_namespace.ClassName.TestName]}

(I'm not 100% sure what the 4 is about...)

Current Behavior

buildEventType | TESTS_UNMUTED
mutedOrUnmutedTests | {4=[suiteName: nestedSuiteName: package_or_namespace.ClassName.TestName]}
buildEventType | TESTS_MUTED
mutedOrUnmutedTests | {5=[]}

Steps to Reproduce (for bugs)

Create a basic teamcity project, add a command line runner with the following content:

echo "##teamcity[testSuiteStarted name='suiteName']"
echo "##teamcity[testSuiteStarted name='nestedSuiteName']"
echo "##teamcity[testStarted name='package_or_namespace.ClassName.TestName']"
echo "##teamcity[testFailed name='package_or_namespace.ClassName.TestName' message='The number must be 20000' details='junit.framework.AssertionFailedError: expected:<20000> but was:<10000>|n|r    at junit.framework.Assert.fail(Assert.java:47)|n|r    at junit.framework.Assert.failNotEquals(Assert.java:280)|n|r...']"
echo "##teamcity[testFinished name='package_or_namespace.ClassName.TestName']"
echo "##teamcity[testSuiteFinished name='nestedSuiteName']"
echo "##teamcity[testSuiteFinished name='suiteName']"

Add a webhook that triggers on mute and unmute. Mute / unmute the test.

Your Environment

Investigation so far

Looking at the code, in WebhookListener.java, it creates a new map, but we could potentially change it from

this.processTestEvent(BuildStateEnum.TESTS_MUTED, null, Collections.singletonMap(muteInfo, new ArrayList<>()));

to

this.processTestEvent(BuildStateEnum.TESTS_MUTED, null, Collections.singletonMap(muteInfo, muteInfo.getTests()));

and I think it should work... (Note that I haven't tested this at all)

netwolfuk commented 1 year ago

Hi @matt-richardson. Thanks for your detailed report (as per usual).

If I remember correctly, the list of tests was not available for muted.

I'll have a look at your suggestion just in case I missed something obvious.

netwolfuk commented 1 year ago

Thanks @matt-richardson. It looks like your suggestion works.

I've pushed to a branch here: https://teamcity.jetbrains.com/buildConfiguration/WebHooksAndOtherPlugins_TcWebHooks?branch=issue_219-muted-tests&buildTypeTab=overview&mode=builds

Can you give that a try and I'll look to do so a release later this week if it all goes well.

netwolfuk commented 1 year ago

Hmm. ~~Looking at the code in that file on line 121, I don't think it ever worked for muted.

It checks for a non-empty list of tests before executing the webhook. School boy error. 🫣~~

Ignore. It's too late. I'm not being sense.

matt-richardson commented 1 year ago

Happy to report that it works successfully for both muted and unmuted now :success:.

derivedBuildEventType | TESTS_MUTED
mutedOrUnmutedTests | {13=[suiteName: nestedSuiteName: package_or_namespace.ClassName.TestName]}
derivedBuildEventType | TESTS_UNMUTED
mutedOrUnmutedTests | {11=[suiteName: nestedSuiteName: package_or_namespace.ClassName.TestName]}
matt-richardson commented 1 year ago

~Another question with this... What is the 13= and 11= on these?~

~I'm using the legacy / deprecated name value pairs format to get these, so it may just be a side affect of that, but not entirely sure how they'll format in json?~

Nevermind: figured it out... It's the default .toString() implementation of MuteInfo - it just renders the id.

netwolfuk commented 1 year ago

I imagine it the result of calling toString on that value.

This is what I get.

I am doing the following in velocity

## Define macro called "showTests"
#macro( showTests $myTests)
#if ( $myTests.size() > 0 ) 
#foreach( $tests in $myTests.keySet() )##
#foreach( $test in $tests.getTests() )##
• #escapejson($test.name)
#end
#end
#else No Changes found #end
#end

It prints the bulleted list in the following screenshot when posted to Slack.

image

matt-richardson commented 1 year ago

I am looking at exactly that file right now :)

matt-richardson commented 1 year ago

I've successfully got this all working to my needs - I'm now able to get hooks with:

{
  "event": "TESTS_UNMUTED",
  "tests": [
    {
      "name": "suiteName: nestedSuiteName: package_or_namespace.ClassName.TestName",
      "id": "-8186798926076262107",
      "user": "matt@example.com"
    }
  ]
}

and

{
  "event": "TESTS_MUTED",
  "tests": [
    {
      "name": "suiteName: nestedSuiteName: package_or_namespace.ClassName.TestName",
      "id": "-8186798926076262107",
      "user": "matt@example.com"
    }
  ]
}

For posterity, this is based on the following template:

{
    "event": "$buildEventType",
    "tests": [
#foreach( $entry in $mutedOrUnmutedTests.entrySet() )##
#foreach( $test in $entry.value )##
      {
        "name": "#escapejson($test.name)",
        "id": "$test.testNameId",
        "user": "$entry.key.mutingUser.username"
      }#if( $foreach.hasNext ),#end
#end
#end
    ]
}
netwolfuk commented 1 year ago

Nice work. Thanks for the update. I'll get a new release out over the weekend. Thanks for your help on this.

netwolfuk commented 1 year ago

@matt-richardson If you get a chance, you could pop those into https://github.com/tcplugins/tcWebHooks/wiki/WebHook-Templates-%3A-The-Velocity-Templating-Engine

matt-richardson commented 1 year ago

Good call - I've updated the wiki 👍

Looks like 1.2.3 is out - I'll give that a go.

netwolfuk commented 1 year ago

Very nice. I like the foreach.hasNext to conditionally add the closing comma.

Thanks again. I think we can close this ticket.

netwolfuk commented 1 year ago

FYI. If you're interested in sharing a working template set, there is a project here: https://github.com/tcplugins/tcWebHooksTemplates

I really really need to update the instructions on that repo for exporting and importing templates, as it's all possible via the GUI now. No more curl and crazy REST malarky.

netwolfuk commented 1 year ago

I've updated the instructions at https://github.com/tcplugins/tcWebHooksTemplates

matt-richardson commented 1 year ago

I know I shouldn't add onto a closed issue, but.... i'm going to anyway :joy:

Im seeing a bunch of errors:

[2023-07-18 02:16:17,787] ERROR [@7438424f'; Normal executor 28] - jetbrains.buildServer.SERVER - AbstractWebHookExecutor :: trackingId: dadfab32-86b1-4bab-8c1a-afd38e6870e4 :: projectId: OctopusDeploy_OctopusServer_CorePlatform :: webhookId: id_905369674 :: templateId: buildmonitormutes, errorCode: 902, errorMessage: Template 'buildmonitormutes' does not support build state 'beforeBuildFinish'

The template is configured like this: image

But the webhook is showing all the boxes as ticked, but disabled: image

Am I holding it wrong?

netwolfuk commented 1 year ago

Looks like a UI issue. Can you please raise as a new ticket? I think it's because the template was re-selected in the UI from one that did support all event types, hence are selected but disabled.

You might be able to work around by opening and saving and opening the webhook again.