Send all parameters as valid json to elasticsearch

[Super8film87]

Hey,

its me again :)

Problem description: I've created a dummy build chain which use any agent which is available and I try to send all parameters of a build to elasticsearch. Most of the code you should already know - but somehow I've problems to receive the data in elasticsearch. in 20% of the builds it is working - but mostly it is not working. What is the best way to check that string is valid for all parameters that I send - so I did some replacements. Any best practice? #escapejson or #sanatize were not helping me :( First I missed the timeStamp format - once I found this stupid mistake I received the data with the correct timestamps but on the way to Elasticsearch loose data on logstash side.

I tested it with different settings - I see that (not 100% sure) it is from env. variables

Test Template I cannot use because timestamp Format is not used in it ->Contraint :(

My template:

##   #set ($currentBuild = $myBuild)
##   #set ($previousBuild = $myBuild.getPreviousFinished())
##   #set ($keepLooping = true)
##   #foreach($unused in [1..50]) ## only loop 50 times in case there are hundreds of failures
##     #if ($keepLooping)
##       #if ($previousBuild) ## Null check 
##         #if (!$previousBuild.getStatusDescriptor().isSuccessful())
##           #set ($keepLooping = false) ## Our $previousBuild isSuccessful, so the $currentBuild is the first failure
##           "$currentBuild.buildNumber  $currentBuild.getStatusDescriptor().isSuccessful()  $currentBuild.getFinishDate().Time " ## Output something when we find our first failure
##         #end
##       #else
##          #set ($keepLooping = false)
##          "unknown" ## text to display if getPreviousFinished returns null
##       #end
##       #set ($currentBuild = $previousBuild) ## set vars for next iteration
##       #set ($previousBuild = $previousBuild.getPreviousFinished()) ## set vars for next iteration
##     #end ## close $keepLooping
##   #end
## #end
#set($queueDuration = (${build.StartDate.Time} - ${build.QueuedDate.Time}) )
###set($buildDuration = (${build.FinishDate.Time} - ${build.StartDate.Time}) )
{
"build_start_time": "${buildStartTime}",
"build_start_time_milli": ${build.StartDate.Time},
"build_full_name" : "${buildFullName}",
"build_finish_time": "${currentTime}",
##set($log = (#tojson(${build.buildLog})))
"queueTime" : $queueDuration,
## "buildTime" : $buildDuration,
"build_event": "${notifyType}", 
"build_name": "${buildName}", 
"triggered_by": "${triggeredBy}", 
"build_result": "${buildResult}", 
"build_result_previous": "${buildResultPrevious}", 
"build_result_delta": "${buildResultDelta}",
"build_type_id" : "${buildTypeId}",
"build_projectName" : "${projectName}",
"build_Status" : "${buildStatus}",
"build_runners" : "${buildRunners}",
"build_status_url": "${buildStatusUrl}",
"build_state_description" : "${buildStateDescription}",
"build_change_file_count" : "${changeFileListCount}",
"build_branch_name" : "${branchName}",
"build_agent_name" : "${agentName}",
"build_agent_hostname" : "${agentHostname}",
"build_agent_os" : "${agentOs}",
"build_text" : "${text}",
"branch_display_name" : "${branchDisplayName}",
"projectName" : "${project.externalId}",
#set ($buildStatistics = $build.fullStatistics)
"successfulTests": $buildStatistics.passedTestCount, 
"failedTests" : $buildStatistics.failedTestCount,
"ignoredTests": $buildStatistics.ignoredTestCount,
"mutedTests"  : $buildStatistics.allTestRunCount,

## #foreach ($paramEntry in $build.parametersProvider.all.entrySet())
## #if (!$paramEntry.key.startsWith("env.A__z") && !$paramEntry.key.startsWith("env.TEAMCITY_CAPTURE_ENV") && !$paramEntry.key.startsWith("env.") && !$paramEntry.key.endsWith(":branchSpec") && $paramEntry.value)
##     #set($cleanKey = ${paramEntry.key.replace(".","_")})
##     #set($cleanKey_string = $cleanKey.replace('"',""))
##     #set($cleanValue = ${paramEntry.value.replace('"',"")})
##     ## "$cleanKey_string" : "#sanitise("$cleanValue")",
##     "$cleanKey_string" : "$cleanValue",
## #end
## #end
#foreach ($paramEntry in $build.parametersProvider.all.entrySet())
    #if (!$paramEntry.key.startsWith("env.A__z") && !$paramEntry.key.startsWith("env.TEAMCITY_CAPTURE_ENV") && $paramEntry.key.startsWith("teamcity.") && !$paramEntry.key.endsWith(":branchSpec") && $paramEntry.value)
        #set($cleanKey = ${paramEntry.key.replace(".","_")})
        #set($cleanKey_string = $cleanKey.replace('"',""))
        #set($cleanValue = ${paramEntry.value.replace('"',"")})
        ## "$cleanKey_string" : "#sanitise("$cleanValue")",
        "$cleanKey_string" : "$cleanValue",
    #end
#end

#foreach ($paramEntry in $build.parametersProvider.all.entrySet())
    #if (!$paramEntry.key.startsWith("env.A__z") && !$paramEntry.key.startsWith("env.TEAMCITY_CAPTURE_ENV") && $paramEntry.key.startsWith("system.") && !$paramEntry.key.endsWith(":branchSpec") && $paramEntry.value)
        #set($cleanKey = ${paramEntry.key.replace(".","_")})
        #set($cleanKey_string = $cleanKey.replace('"',""))
        #set($cleanValue = ${paramEntry.value.replace('"',"")})
        ## "$cleanKey_string" : "#sanitise("$cleanValue")",
        "$cleanKey_string" : "$cleanValue",
    #end
#end

## #foreach ($paramEntry in $build.parametersProvider.all.entrySet())
##     #if (!$paramEntry.key.startsWith("env.A__z") && !$paramEntry.key.startsWith("env.TEAMCITY_CAPTURE_ENV") && $paramEntry.key.startsWith("env.") && !$paramEntry.key.endsWith(":branchSpec") && $paramEntry.value)
##         #set($cleanKey = ${paramEntry.key.replace(".","_")})
##         #set($cleanKey_string = $cleanKey.replace('"',""))
##         #set($cleanValue = ${paramEntry.value.replace('"',"")})
##         ## "$cleanKey_string" : "#sanitise("$cleanValue")",
##         "$cleanKey_string" : "$cleanValue",
##     #end
## #end

## #foreach ($paramEntry in $build.parametersProvider.all.entrySet())
##     #if (!$paramEntry.key.startsWith("env.") && !$paramEntry.key.startsWith("system.") && !$paramEntry.key.startsWith("teamcity.") && !$paramEntry.key.endsWith(":branchSpec") && $paramEntry.value)
##         #set($cleanKey = ${paramEntry.key.replace(".","_")})
##         #set($cleanKey_string = $cleanKey.replace('"',""))
##         #set($cleanValue = ${paramEntry.value.replace('"',"")})
##         ## "$cleanKey_string" : "#sanitise("$cleanValue")",
##         "$cleanKey_string" : "$cleanValue",
##     #end
## #end

#foreach ($mapEntry in $statsValues.entrySet())
  #if ( $mapEntry.value )
    "$mapEntry.key": $mapEntry.value.intValue(),
  #end
#end
"build_status" : "${buildType.status}",
## Define macro called "getFirstFailedBuild"
##if buildtype is not composite
## "steps" :[
##     #foreach($buildStep in $build.getBuildPromotion().getBuildSettings().getBuildRunners())
##     #set($stepDuration = ($build.getStatisticValue("buildStageDuration:buildStep" + $buildStep.getId())))
##     {
##         "stepName" : "$buildStep.getName()",
##         "runnerType" : "$buildStep.getType()",
##         ##  "assosciated_parameters": "$buildStep.getParameters()",
##         ## "variant_parameters": "$buildStep.getOwnBuildParameters()",
##         "buildStepDuration" : "$stepDuration"
##     }#if($foreach.hasNext),#end
##     #end
## ]
"template_id" : "velocity_v1"
##"firstFailed" : #getFirstFailedBuild($build)
}```

[netwolfuk]

Hi @Super8film87

I copied the template above and it seems to work for my testing. I created a webhook with it pointed at http://localhost:8111/webhooks/endpoint.html

Then you can look at the results at http://teamcity:8111/webhooks/endpoint-viewer.html It looks like valid JSON to me, but would depend on the variables you have. Can you try that and share a value that doesn't appear to work?

[Super8film87]

Oh damn - this nice! didnt knew that!

{
  "build_start_time": "2023-08-04T16:49:43.410+02:00",
  "build_start_time_milli": 1691160583410,
  "build_full_name": "xxxx/ xxx / xx.xx.x - xxxxPipelines / xx.x.xxx/ xxx/ Pull_Request / xxx Generation of xx and xxx",
  "build_finish_time": "2023-08-04T16:49:50.229+02:00",
  "queueTime": 1395,
  "build_event": "buildFinished",
  "build_name": "xx xxof xxand xxx",
  "triggered_by": "xxA  xx xx- xxx",
  "build_result": "failure",
  "build_result_previous": "failure",
  "build_result_delta": "unchanged",
  "build_type_id": "xxxx01Projects_xxxxxxxDevelopment_xxxxxxxxx_PullRequest_xxxxxxxxx",
  "build_projectName": "Pull_Request",
  "build_Status": "FAILURE",
  "build_runners": "[Command Line, Command Line, Command Line, Command Line, Command Line]",
  "build_status_url": "http://xxxxxxx.com/viewLog.html?buildTypeId=xxxxxx=5465013",
  "build_state_description": "finished",
  "build_change_file_count": "0",
  "build_branch_name": "pull/34",
  "build_agent_name": "AGENTxxxxxx0cac0c813eb1a4773",
  "build_agent_hostname": "xx.xxx.xx.xxx.xx",
  "build_agent_os": "Windows Server 2019, version 10.0",
  "build_text": "xxxx/ xxxx/ xxxx.x - xxxxPipelines / xx.xx.xxx/ xxx/ Pull_Request / xxx xxx xxx of xxx and xxx has finished. Status: failure",
  "branch_display_name": "pull/34",
  "projectName": "xxxxs_xxxt_xxxF_xxx_xxx",
  "successfulTests": "0",
  "failedTests": "0",
  "ignoredTests": "0",
  "mutedTests": "0",
  "system_ec2_ami-manifest-path": "_unknown_",
  "system_ec2_reservation-id": "r-0419970d824efd93d",
  "system_ec2_proposed-agent-name": "Axxxxx-0cac0c813eb1a4773",
  "system_ec2_instance-id": "i-0cac0c813eb1a4773",
  "system_ec2_instance-type": "xxx.xxxe",
  "system_ec2_ami-id": "ami-xxxx",
  "system_ec2_local-ipv4": "x.1xxx.xx1x",
  "system_ec2_local-hostname": "ip-1x.1xxx.xx1x.eucal",
  "system_ec2_ami-launch-index": "0",
  "env_ComSpec": "C:_Windows_system32_cmd.exe",
  "env_USERNAME": "Administrator",
  "env_PATHEXT": ".COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC;.PY;.PYW",
  "env_USERDOMAIN": "EC2AMAZ-VTNR0T7",
  "env_POWERSHELL_DISTRIBUTION_CHANNEL": "MSI:Windows Server 2019 Datacenter",
  "env_TRACET_LICENSE": "cccc.inetpsa.com",
  "env_PUBLIC": "C:_Users_Public",
  "env_PSModulePath": "_ProgramFiles__WindowsPowerShell_Modules;C:_Windows_system32_WindowsPowerShell_v1.0_Modules;C:_Program Files _x86__AWS Tools_PowerShell_",
  "env_PROCESSOR_LEVEL": "6",
  "env_TEAMCITY_GIT_VERSION": "2.31.1.0",
  "env_CANoe_InstallDir": "C:_Program Files_Vector xxxx 16_Exec64_",
  "env_=C:": "c:_BuildAgent_bin",
  "env_TEAMCITY_JRE": "C:_Program Files_xxx Corretto_jdk11.0.16_9",
  "env_CommonProgramW6432": "C:_Program Files_Common Files",
  "env_WRAPPER_FILE_SEPARATOR": "_",
  "env_ALLUSERSPROFILE": "C:_ProgramData",
  "env_ReportViewerDivaAPIs_InstallDir": "C:_Program Files_Common Files_xxxxx Test Report Viewer xxx xxxI Assemblies_",
  "env_ProgramData": "C:_ProgramData",
  "env_ProgramFiles(x86)": "C:_Program Files _x86_",
  "env_ProgramFiles": "C:_Program Files",
  "env_JDK_11_x64": "C:_Program Files_Amazon Corretto_jdk11.0.16_9",
  "env_PROCESSOR_ARCHITECTURE": "AMD64",
  "env_NUMBER_OF_PROCESSORS": "48",
  "env_DSPACE_RTT_GLOBALS": "C:_Program Files_Common Files_xxxx_Real-Time Testing Globals_Main",
  "env_PROCESSOR_IDENTIFIER": "Intel64 Family 6 Model 106 Stepping 6, GenuineIntel",
  "env_USERPROFILE": "C:_Users_Administrator",
  "env_test_variable": "TRUE",
  "env_PICO_DRIVER": "C:_Program Files_Vector_CANoe_CANalyzer_Scope_Drivers",
  "env_APPDATA": "C:_Users_Administrator_AppData_Roaming",
  "env_windir": "C:_Windows",
  "env_DOTNET_ROOT": "C:_Program Files_dotnet",
  "env_EC2LAUNCH_TELEMETRY": "1",
  "env_BUILD_NUMBER": "76",
  "env_WRAPPER_OS": "windows",
  "env_JDK_HOME": "C:_Program Files_Amazon Corretto_jdk11.0.16_9",
  "env_SystemRoot": "C:_Windows",
  "env_SONAR_TOKEN": "_______",
  "env_vVIRTUALtarget_InstallDir": "C:_Program Files_Vector vVIRTUALtarget 6_Exec64_",
  "env_WRAPPER_ARCH": "x86",
  "env_TEAMCITY_BUILD_PROPERTIES_FILE": "C:_TC_temp_buildTmp_teamcity.build.parameters",
  "env_CANoe4SWSE_InstallDir64": "C:_Program Files_Vector CANoe4SW Server Edition 16_Exec64_",
  "env_JDK_11_0_x64": "C:_Program Files_Amazon Corretto_jdk11.0.16_9",
  "env_TEAMCITY_GIT_PATH": "git.exe",
  "env_CONAN_LOGIN_USERNAME": "xxxxx",
  "env_ReportViewerSelector_InstallDir": "C:_Program Files_Vector xxxTest Report xxxSelector_",
  "env_Path": "C:_Program Files_Fortify_Fortify_SCA_and_Apps_22.2.1_bin;C:_Program Files_Python310_Scripts_;C:_Program Files_Python310_;C:_Program Files_Python39_Scripts_;C:_Program Files_Python39_;C:_Tools_Git_cmd;C:_Windows_system32_config_systemprofile_AppData_Local_Microsoft_WindowsApps;C:_Program Files_Amazon Corretto_jdk11.0.16_9_bin;C:_Windows_system32;C:_Windows;C:_Windows_System32_Wbem;C:_Windows_System32_WindowsPowerShell_v1.0_;C:_Windows_System32_OpenSSH_;C:_Program Files_Amazon_cfn-bootstrap_;C:_Program Files_PowerShell_7_;C:_Program Files_Microsoft VS Code_bin;C:_Program Files_Python38;C:_portable_tools;C:_portable_tools_MakeSupport_04.01.00;C:_portable_tools_ta-tool-suite_22.1.1_win64;C:_portable_tools_ta-tool-suite_22.1.1_win64_TA Tool Suite x86_64;C:_portable_tools_MinGW_i686-8.1.0-release-win32-dwarf-rt_v6-rev0_mingw32_bin;C:_Apps_MATLAB_R2022a_runtime_win64;C:_Apps_MATLAB_R2022a_bin;C:_Program Files_Microsoft SQL Server_130_Tools_Binn_;C:_Program Files _x86__Windows Kits_8.1_Windows Performance Toolkit_;C:_Program Files_Vector_ASAP2 Tool-Set 16.0_Bin;C:_Program Files _x86__Vector CANdb++ 3.1_Exec32;C:_portable_tools_gradle-7.5.1;C:_portable_tools_gradle-7.5.1_bin;C:_Program Files_dotnet;C:_Program Files_dotnet_;C:_Tools_Git_usr_bin;C:_Tools_cpptest;C:_Program Files _x86__Microsoft Visual Studio_2017_WDExpress_MSBuild_15.0_Bin;C:_Program Files_CMake_bin;C:_Users_Administrator_AppData_Local_Microsoft_WindowsApps;C:_Users_Administrator_.dotnet_tools;C:_tools_Git_usr_bin",
  "env_DriverData": "C:_Windows_System32_Drivers_DriverData",
  "env_ProgramW6432": "C:_Program Files",
  "env_JRE_HOME": "C:_Program Files_Amazon Corretto_jdk11.0.16_9",
  "env_WRAPPER_PATH_SEPARATOR": ";",
  "env_PROCESSOR_REVISION": "6a06",
  "env_PROJECT_ACCOUNT_USERNAME": "_______",
  "env_DSPACE_RTT_TOOLS_6_0": "C:_Program Files_Common Files_xxxx_Real-Time Testing Tools 6.0_Main_bin",
  "env_TEMP": "C:_TC_temp_buildTmp",
  "env_TEAMCITY_BUILDCONF_NAME": "xxx xxx Generation of BSW and SWC",
  "env_BUILD_VCS_NUMBER_xxx_xxxxt_xxxx_xxx": "xxxxxxxx",
  "env_CANoe_InstallDir64": "C:_Program Files_Vector CANoe 16_Exec64_",
  "env_COMPUTERNAME": "EC2AMAZ-VTNR0T7",
  "env_JDK_11": "C:_Program Files_Amazon Corretto_jdk11.0.16_9",
  "env_SystemDrive": "C:",
  "env_TEAMCITY_PROJECT_NAME": "Pull_Request",
  "env_TMPDIR": "C:_TC_temp_buildTmp",
  "env_CommonProgramFiles": "C:_Program Files_Common Files",
  "env_WRAPPER_BITS": "32",
  "env_AWS_EXECUTION_ENV": "EC2",
  "env_TEAMCITY_VERSION": "2022.10.4 _build 117134_",
  "env_JDK_11_0": "C:_Program Files_Amazon Corretto_jdk11.0.16_9",
  "env_TSK_OPTIONS_FILE_SW160800v6_3r1": "C:_TASKING_TriCore_v6.3r1_etc_licopt.txt",
  "env_DS_HELP_LOG_DIR": "C:_ProgramData_xxxx_38033EBA-3AB8-4483-B347-FFAF55D0B520",
  "env_CommonProgramFiles(x86)": "C:_Program Files _x86__Common Files",
  "env_LOCALAPPDATA": "C:_Users_Administrator_AppData_Local",
  "env_OS": "Windows_NT",
  "env_TMP": "C:_TC_temp_buildTmp",
  "env_BUILD_VCS_NUMBER": "fxxxxdd8",
  "env_BRS_COMPILER_PATH": "C:_TASKING_TriCore_v6.3r1_ctc",
  "template_id": "velocity_v1"
}

[Super8film87]

is it possible to create a one-line json out of it and remove all \n \r from sending? Is it possible that i've somewhere \n which is sended? In logstash I see everytime \n behind the value

[netwolfuk]

You could make it all one line, but I would argue that logstash should be able to consume the data even with \n in them. Having new lines in JSON is valid.

[netwolfuk]

You could strip out the \n in an input filter in logstash if I remember correctly.

[Super8film87]

like that?

#set( $newline="
")

 #set($cleanValue = ${paramEntry.value.replace('"',"")})
        #set($cleanValue2 = $cleanValue.replaceAll("$newline",""))

it looks like logstash creates trouble - and I was asked to do anything in online. I agree with you - but I sit between the chairs 🗡️

[netwolfuk]

I don't think that would work on velocity. There is some magic in velocity for whitespace reduction.

However I don't think I've seen it work in the tcWebHooks templates. Maybe I need to add some extra configuration when I start the velocity engine to render the payload.

I'll have to take a look into it.

[netwolfuk]

Apparently you can try adding ## to the end of each line. This turns the rest of the line into a comment and so velocity ignores the \n at the end.

You could try that whilst I look for a better solution.

[Super8film87]

Apparently you can try adding ## to the end of each line. -> I already did this. This a good best practice :D

-> The problem occurs in env. variables - my guess is that I've a \n somewhere inside the parameters. But I cannot see when I use your proposal for debugging.

[netwolfuk]

Oh. You could try $paramEntry.value.trim() and it should remove leading and trailing whitespace.

[Super8film87]

ok I solved the problem and it was simple. The number of keys can be limited on elasticsearch. So once you send all parameters for deps;env etc. to the elastic like this you will reach this limit very simple.

[netwolfuk]

Oh wow. Well done. Great to have resolution, but certainly not what I expected.