search

jenkinsci / ghprb-plugin

github pull requests builder plugin for Jenkins
https://plugins.jenkins.io/ghprb/
MIT License
506 stars 612 forks source link

GHPRB overwriting test results from a previous job #718

Closed deweistarsky closed 6 years ago

deweistarsky commented 6 years ago

I have the following setup:

job1 - triggers from GHPRB with context A job2 - triggers from GHPRB with context B

When a PR is made, I expect job1 and job2 to both run, which they do. However, when the test results come back the job that finishes last will overwrite the results from the other job. So if job1 passed and job2 finished last but failed, I see both test results on Github to be failed. Am I setting something incorrectly here?

bjoernhaeuser commented 6 years ago

You need to have different commit status context setup per job. This will make sure that both jobs set a different test result.

deweistarsky commented 6 years ago

@bjoernhaeuser I enabled the setting "Set GitHub commit status with custom context and message" and am using different context names that I manually define. However, the results are still overwritten. 2 separate contexts are created on Github, however, the test results are the same. Is there another config that I am missing here?

bjoernhaeuser commented 6 years ago

If this is the case we would need your job configuration. We are using this setup on a day to day basis and it works for us.

deweistarsky commented 6 years ago

Thanks for the fast response.

Here is job 1:

<?xml version='1.1' encoding='UTF-8'?>
<project>
  <actions/>
  <description>Job triggered when a pull request is made in starskybot</description>
  <keepDependencies>false</keepDependencies>
  <properties>
    <com.coravy.hudson.plugins.github.GithubProjectProperty plugin="github@1.29.3">
      <projectUrl>https://github.com/StarskyRobotics/starskybot/</projectUrl>
      <displayName></displayName>
    </com.coravy.hudson.plugins.github.GithubProjectProperty>
    <hudson.model.ParametersDefinitionProperty>
      <parameterDefinitions>
        <hudson.model.StringParameterDefinition>
          <name>sha1</name>
          <description>Commit id (in sha1 format) to build if you want to trigger this job manually instead of waiting for a PR from Github</description>
          <defaultValue>2695b1c077c0f47b158e68a821dc72d32df1b4c9</defaultValue>
          <trim>true</trim>
        </hudson.model.StringParameterDefinition>
      </parameterDefinitions>
    </hudson.model.ParametersDefinitionProperty>
  </properties>
  <scm class="hudson.plugins.git.GitSCM" plugin="git@3.9.1">
    <configVersion>2</configVersion>
    <userRemoteConfigs>
      <hudson.plugins.git.UserRemoteConfig>
        <name>origin</name>
        <refspec>+refs/pull/*:refs/remotes/origin/pr/*</refspec>
        <url>git@github.com:/StarskyRobotics/devops.git</url>
        <credentialsId>c232086e-4840-401e-9534-2f59f547fb92</credentialsId>
      </hudson.plugins.git.UserRemoteConfig>
    </userRemoteConfigs>
    <branches>
      <hudson.plugins.git.BranchSpec>
        <name>master</name>
      </hudson.plugins.git.BranchSpec>
    </branches>
    <doGenerateSubmoduleConfigurations>false</doGenerateSubmoduleConfigurations>
    <browser class="hudson.plugins.git.browser.GithubWeb">
      <url></url>
    </browser>
    <submoduleCfg class="list"/>
    <extensions>
      <hudson.plugins.git.extensions.impl.BuildChooserSetting>
        <buildChooser class="hudson.plugins.git.util.AncestryBuildChooser">
          <maximumAgeInDays>30</maximumAgeInDays>
          <ancestorCommitSha1></ancestorCommitSha1>
        </buildChooser>
      </hudson.plugins.git.extensions.impl.BuildChooserSetting>
    </extensions>
  </scm>
  <assignedNode>jenkins-jnlp-slave</assignedNode>
  <canRoam>false</canRoam>
  <disabled>false</disabled>
  <blockBuildWhenDownstreamBuilding>false</blockBuildWhenDownstreamBuilding>
  <blockBuildWhenUpstreamBuilding>false</blockBuildWhenUpstreamBuilding>
  <triggers>
    <org.jenkinsci.plugins.ghprb.GhprbTrigger plugin="ghprb@1.42.0">
      <spec>H/5 * * * *</spec>
      <configVersion>3</configVersion>
      <adminlist>achrungoo
aishanou
apurvtwr
davrsky
dfeldstarsky
DivyaThomas1
johnstarsky
kartiktwr
KevinKeogh
mikestarsky
rebfb
deweistarsky</adminlist>
      <allowMembersOfWhitelistedOrgsAsAdmin>false</allowMembersOfWhitelistedOrgsAsAdmin>
      <orgslist></orgslist>
      <cron>H/5 * * * *</cron>
      <buildDescTemplate></buildDescTemplate>
      <onlyTriggerPhrase>false</onlyTriggerPhrase>
      <useGitHubHooks>false</useGitHubHooks>
      <permitAll>false</permitAll>
      <whitelist>deweistarsky starsky-buck leoore starsky-devops SchoonerStarsky mattcreme jev63 NBotta davidbailey</whitelist>
      <autoCloseFailedPullRequests>false</autoCloseFailedPullRequests>
      <displayBuildErrorsOnDownstreamBuilds>false</displayBuildErrorsOnDownstreamBuilds>
      <whiteListTargetBranches>
        <org.jenkinsci.plugins.ghprb.GhprbBranch>
          <branch></branch>
        </org.jenkinsci.plugins.ghprb.GhprbBranch>
      </whiteListTargetBranches>
      <blackListTargetBranches>
        <org.jenkinsci.plugins.ghprb.GhprbBranch>
          <branch></branch>
        </org.jenkinsci.plugins.ghprb.GhprbBranch>
      </blackListTargetBranches>
      <gitHubAuthId>f72c0182-8422-4064-8b12-74c71d5dab1a</gitHubAuthId>
      <triggerPhrase></triggerPhrase>
      <skipBuildPhrase>.*\[skip\W+ci\].*</skipBuildPhrase>
      <blackListCommitAuthor></blackListCommitAuthor>
      <blackListLabels></blackListLabels>
      <whiteListLabels></whiteListLabels>
      <includedRegions></includedRegions>
      <excludedRegions></excludedRegions>
      <extensions>
        <org.jenkinsci.plugins.ghprb.extensions.status.GhprbSimpleStatus>
          <commitStatusContext>jenkins-ecs</commitStatusContext>
          <triggeredStatus></triggeredStatus>
          <startedStatus></startedStatus>
          <statusUrl></statusUrl>
          <addTestResults>true</addTestResults>
        </org.jenkinsci.plugins.ghprb.extensions.status.GhprbSimpleStatus>
        <org.jenkinsci.plugins.ghprb.extensions.build.GhprbCancelBuildsOnUpdate>
          <overrideGlobal>false</overrideGlobal>
        </org.jenkinsci.plugins.ghprb.extensions.build.GhprbCancelBuildsOnUpdate>
      </extensions>
    </org.jenkinsci.plugins.ghprb.GhprbTrigger>
  </triggers>
  <concurrentBuild>false</concurrentBuild>
  <builders>
    <com.cloudbees.jenkins.GitHubSetCommitStatusBuilder plugin="github@1.29.3">
      <statusMessage>
        <content></content>
      </statusMessage>
      <contextSource class="org.jenkinsci.plugins.github.status.sources.ManuallyEnteredCommitContextSource">
        <context>jenkins-ecs</context>
      </contextSource>
    </com.cloudbees.jenkins.GitHubSetCommitStatusBuilder>
    <hudson.tasks.Shell>
      <command>cd test/starskybot-unit-tests

if [ -n &quot;$ghprbActualCommit&quot; ]; then
    ./jenkins.sh $ghprbActualCommit False
else
    ./jenkins.sh $sha1 False
fi</command>
    </hudson.tasks.Shell>
    <org.jenkinsci.plugins.dockerbuildstep.DockerBuilder plugin="docker-build-step@2.0">
      <dockerCmd class="org.jenkinsci.plugins.dockerbuildstep.cmd.RemoveImageCommand">
        <imageName>starskybot-unit-test:$BUILD_TAG</imageName>
        <imageId></imageId>
        <ignoreIfNotFound>true</ignoreIfNotFound>
      </dockerCmd>
    </org.jenkinsci.plugins.dockerbuildstep.DockerBuilder>
  </builders>
  <publishers>
    <hudson.tasks.ArtifactArchiver>
      <artifacts>**/rostest*.png, **/*.log, **/*.xml</artifacts>
      <allowEmptyArchive>false</allowEmptyArchive>
      <onlyIfSuccessful>false</onlyIfSuccessful>
      <fingerprint>false</fingerprint>
      <defaultExcludes>true</defaultExcludes>
      <caseSensitive>true</caseSensitive>
    </hudson.tasks.ArtifactArchiver>
    <hudson.tasks.junit.JUnitResultArchiver plugin="junit@1.24">
      <testResults>**/gtest-*.xml, **/rostest-*.xml, **/rosunit-*.xml, **/nosetests-*.py.xml</testResults>
      <keepLongStdio>false</keepLongStdio>
      <healthScaleFactor>1.0</healthScaleFactor>
      <allowEmptyResults>false</allowEmptyResults>
    </hudson.tasks.junit.JUnitResultArchiver>
    <org.jenkinsci.plugins.github.status.GitHubCommitStatusSetter plugin="github@1.29.3">
      <commitShaSource class="org.jenkinsci.plugins.github.status.sources.BuildDataRevisionShaSource"/>
      <reposSource class="org.jenkinsci.plugins.github.status.sources.AnyDefinedRepositorySource"/>
      <contextSource class="org.jenkinsci.plugins.github.status.sources.ManuallyEnteredCommitContextSource">
        <context>jenkins-ecs</context>
      </contextSource>
      <statusResultSource class="org.jenkinsci.plugins.github.status.sources.DefaultStatusResultSource"/>
      <statusBackrefSource class="org.jenkinsci.plugins.github.status.sources.BuildRefBackrefSource"/>
      <errorHandlers/>
    </org.jenkinsci.plugins.github.status.GitHubCommitStatusSetter>
  </publishers>
  <buildWrappers>
    <org.jenkinsci.plugins.credentialsbinding.impl.SecretBuildWrapper plugin="credentials-binding@1.16">
      <bindings>
        <org.jenkinsci.plugins.credentialsbinding.impl.SSHUserPrivateKeyBinding>
          <credentialsId>c232086e-4840-401e-9534-2f59f547fb91</credentialsId>
          <keyFileVariable>DEVOPS_DEPLOY_KEY</keyFileVariable>
          <usernameVariable></usernameVariable>
          <passphraseVariable></passphraseVariable>
        </org.jenkinsci.plugins.credentialsbinding.impl.SSHUserPrivateKeyBinding>
      </bindings>
    </org.jenkinsci.plugins.credentialsbinding.impl.SecretBuildWrapper>
    <hudson.plugins.build__timeout.BuildTimeoutWrapper plugin="build-timeout@1.19">
      <strategy class="hudson.plugins.build_timeout.impl.AbsoluteTimeOutStrategy">
        <timeoutMinutes>15</timeoutMinutes>
      </strategy>
      <operationList/>
    </hudson.plugins.build__timeout.BuildTimeoutWrapper>
    <org.jenkinsci.plugins.ghprb.upstream.GhprbUpstreamStatus plugin="ghprb@1.42.0">
      <showMatrixStatus>false</showMatrixStatus>
      <commitStatusContext>jenkins-ecs</commitStatusContext>
      <triggeredStatus></triggeredStatus>
      <startedStatus></startedStatus>
      <statusUrl></statusUrl>
      <addTestResults>true</addTestResults>
    </org.jenkinsci.plugins.ghprb.upstream.GhprbUpstreamStatus>
  </buildWrappers>
</project>

and job2:

<?xml version='1.1' encoding='UTF-8'?>
<project>
  <actions/>
  <description>Job triggered when a pull request is made in starskybot (GPU only tests)</description>
  <keepDependencies>false</keepDependencies>
  <properties>
    <com.coravy.hudson.plugins.github.GithubProjectProperty plugin="github@1.29.3">
      <projectUrl>https://github.com/StarskyRobotics/starskybot/</projectUrl>
      <displayName></displayName>
    </com.coravy.hudson.plugins.github.GithubProjectProperty>
    <hudson.model.ParametersDefinitionProperty>
      <parameterDefinitions>
        <hudson.model.StringParameterDefinition>
          <name>sha1</name>
          <description>Commit id (in sha1 format) to build if you want to trigger this job manually instead of waiting for a PR from Github</description>
          <defaultValue>2695b1c077c0f47b158e68a821dc72d32df1b4c9</defaultValue>
          <trim>true</trim>
        </hudson.model.StringParameterDefinition>
      </parameterDefinitions>
    </hudson.model.ParametersDefinitionProperty>
  </properties>
  <scm class="hudson.plugins.git.GitSCM" plugin="git@3.9.1">
    <configVersion>2</configVersion>
    <userRemoteConfigs>
      <hudson.plugins.git.UserRemoteConfig>
        <name>origin</name>
        <refspec>+refs/pull/*:refs/remotes/origin/pr/*</refspec>
        <url>git@github.com:/StarskyRobotics/devops.git</url>
        <credentialsId>c232086e-4840-401e-9534-2f59f547fb91</credentialsId>
      </hudson.plugins.git.UserRemoteConfig>
    </userRemoteConfigs>
    <branches>
      <hudson.plugins.git.BranchSpec>
        <name>master</name>
      </hudson.plugins.git.BranchSpec>
    </branches>
    <doGenerateSubmoduleConfigurations>false</doGenerateSubmoduleConfigurations>
    <browser class="hudson.plugins.git.browser.GithubWeb">
      <url></url>
    </browser>
    <submoduleCfg class="list"/>
    <extensions>
      <hudson.plugins.git.extensions.impl.BuildChooserSetting>
        <buildChooser class="hudson.plugins.git.util.AncestryBuildChooser">
          <maximumAgeInDays>30</maximumAgeInDays>
          <ancestorCommitSha1></ancestorCommitSha1>
        </buildChooser>
      </hudson.plugins.git.extensions.impl.BuildChooserSetting>
    </extensions>
  </scm>
  <assignedNode>jenkins-jnlp-slave-gpu</assignedNode>
  <canRoam>false</canRoam>
  <disabled>false</disabled>
  <blockBuildWhenDownstreamBuilding>false</blockBuildWhenDownstreamBuilding>
  <blockBuildWhenUpstreamBuilding>false</blockBuildWhenUpstreamBuilding>
  <triggers>
    <org.jenkinsci.plugins.ghprb.GhprbTrigger plugin="ghprb@1.42.0">
      <spec>H/5 * * * *</spec>
      <configVersion>3</configVersion>
      <adminlist>achrungoo
aishanou
apurvtwr
davrsky
dfeldstarsky
DivyaThomas1
johnstarsky
kartiktwr
KevinKeogh
mikestarsky
rebfb
deweistarsky</adminlist>
      <allowMembersOfWhitelistedOrgsAsAdmin>false</allowMembersOfWhitelistedOrgsAsAdmin>
      <orgslist></orgslist>
      <cron>H/5 * * * *</cron>
      <buildDescTemplate></buildDescTemplate>
      <onlyTriggerPhrase>false</onlyTriggerPhrase>
      <useGitHubHooks>false</useGitHubHooks>
      <permitAll>false</permitAll>
      <whitelist>deweistarsky starsky-buck leoore starsky-devops SchoonerStarsky mattcreme jev63 NBotta davidbailey</whitelist>
      <autoCloseFailedPullRequests>false</autoCloseFailedPullRequests>
      <displayBuildErrorsOnDownstreamBuilds>false</displayBuildErrorsOnDownstreamBuilds>
      <whiteListTargetBranches>
        <org.jenkinsci.plugins.ghprb.GhprbBranch>
          <branch></branch>
        </org.jenkinsci.plugins.ghprb.GhprbBranch>
      </whiteListTargetBranches>
      <blackListTargetBranches>
        <org.jenkinsci.plugins.ghprb.GhprbBranch>
          <branch></branch>
        </org.jenkinsci.plugins.ghprb.GhprbBranch>
      </blackListTargetBranches>
      <gitHubAuthId>f72c0182-8422-4064-8b12-74c71d5dab2b</gitHubAuthId>
      <triggerPhrase></triggerPhrase>
      <skipBuildPhrase>.*\[skip\W+ci\].*</skipBuildPhrase>
      <blackListCommitAuthor></blackListCommitAuthor>
      <blackListLabels></blackListLabels>
      <whiteListLabels></whiteListLabels>
      <includedRegions></includedRegions>
      <excludedRegions></excludedRegions>
      <extensions>
        <org.jenkinsci.plugins.ghprb.extensions.status.GhprbSimpleStatus>
          <commitStatusContext>jenkins-ecs</commitStatusContext>
          <triggeredStatus></triggeredStatus>
          <startedStatus></startedStatus>
          <statusUrl></statusUrl>
          <addTestResults>true</addTestResults>
        </org.jenkinsci.plugins.ghprb.extensions.status.GhprbSimpleStatus>
        <org.jenkinsci.plugins.ghprb.extensions.build.GhprbCancelBuildsOnUpdate>
          <overrideGlobal>false</overrideGlobal>
        </org.jenkinsci.plugins.ghprb.extensions.build.GhprbCancelBuildsOnUpdate>
      </extensions>
    </org.jenkinsci.plugins.ghprb.GhprbTrigger>
  </triggers>
  <concurrentBuild>false</concurrentBuild>
  <builders>
    <com.cloudbees.jenkins.GitHubSetCommitStatusBuilder plugin="github@1.29.3">
      <statusMessage>
        <content></content>
      </statusMessage>
      <contextSource class="org.jenkinsci.plugins.github.status.sources.ManuallyEnteredCommitContextSource">
        <context>jenkins-ecs-gpu</context>
      </contextSource>
    </com.cloudbees.jenkins.GitHubSetCommitStatusBuilder>
    <hudson.tasks.Shell>
      <command>cd test/starskybot-unit-tests

if [ -n &quot;$ghprbActualCommit&quot; ]; then
    ./jenkins.sh $ghprbActualCommit True
else
    ./jenkins.sh $sha1 True
fi</command>
    </hudson.tasks.Shell>
    <org.jenkinsci.plugins.dockerbuildstep.DockerBuilder plugin="docker-build-step@2.0">
      <dockerCmd class="org.jenkinsci.plugins.dockerbuildstep.cmd.RemoveImageCommand">
        <imageName>starskybot-unit-test:$BUILD_TAG</imageName>
        <imageId></imageId>
        <ignoreIfNotFound>true</ignoreIfNotFound>
      </dockerCmd>
    </org.jenkinsci.plugins.dockerbuildstep.DockerBuilder>
  </builders>
  <publishers>
    <hudson.tasks.ArtifactArchiver>
      <artifacts>**/rostest*.png, **/*.log, **/*.xml</artifacts>
      <allowEmptyArchive>false</allowEmptyArchive>
      <onlyIfSuccessful>false</onlyIfSuccessful>
      <fingerprint>false</fingerprint>
      <defaultExcludes>true</defaultExcludes>
      <caseSensitive>true</caseSensitive>
    </hudson.tasks.ArtifactArchiver>
    <hudson.tasks.junit.JUnitResultArchiver plugin="junit@1.24">
      <testResults>**/gtest-*.xml, **/rostest-*.xml, **/rosunit-*.xml, **/nosetests-*.py.xml</testResults>
      <keepLongStdio>false</keepLongStdio>
      <healthScaleFactor>1.0</healthScaleFactor>
      <allowEmptyResults>false</allowEmptyResults>
    </hudson.tasks.junit.JUnitResultArchiver>
    <org.jenkinsci.plugins.github.status.GitHubCommitStatusSetter plugin="github@1.29.3">
      <commitShaSource class="org.jenkinsci.plugins.github.status.sources.BuildDataRevisionShaSource"/>
      <reposSource class="org.jenkinsci.plugins.github.status.sources.ManuallyEnteredRepositorySource">
        <url>https://github.com/StarskyRobotics/starskybot/</url>
      </reposSource>
      <contextSource class="org.jenkinsci.plugins.github.status.sources.ManuallyEnteredCommitContextSource">
        <context>jenkins-ecs-gpu</context>
      </contextSource>
      <statusResultSource class="org.jenkinsci.plugins.github.status.sources.DefaultStatusResultSource"/>
      <statusBackrefSource class="org.jenkinsci.plugins.github.status.sources.BuildRefBackrefSource"/>
      <errorHandlers/>
    </org.jenkinsci.plugins.github.status.GitHubCommitStatusSetter>
  </publishers>
  <buildWrappers>
    <org.jenkinsci.plugins.credentialsbinding.impl.SecretBuildWrapper plugin="credentials-binding@1.16">
      <bindings>
        <org.jenkinsci.plugins.credentialsbinding.impl.SSHUserPrivateKeyBinding>
          <credentialsId>c232086e-4840-401e-9534-2f59f547fb22</credentialsId>
          <keyFileVariable>DEVOPS_DEPLOY_KEY</keyFileVariable>
          <usernameVariable></usernameVariable>
          <passphraseVariable></passphraseVariable>
        </org.jenkinsci.plugins.credentialsbinding.impl.SSHUserPrivateKeyBinding>
      </bindings>
    </org.jenkinsci.plugins.credentialsbinding.impl.SecretBuildWrapper>
    <hudson.plugins.build__timeout.BuildTimeoutWrapper plugin="build-timeout@1.19">
      <strategy class="hudson.plugins.build_timeout.impl.AbsoluteTimeOutStrategy">
        <timeoutMinutes>15</timeoutMinutes>
      </strategy>
      <operationList/>
    </hudson.plugins.build__timeout.BuildTimeoutWrapper>
    <hudson.plugins.timestamper.TimestamperBuildWrapper plugin="timestamper@1.8.10"/>
    <org.jenkinsci.plugins.ghprb.upstream.GhprbUpstreamStatus plugin="ghprb@1.42.0">
      <showMatrixStatus>false</showMatrixStatus>
      <commitStatusContext>jenkins-ecs-gpu</commitStatusContext>
      <triggeredStatus></triggeredStatus>
      <startedStatus></startedStatus>
      <statusUrl></statusUrl>
      <addTestResults>true</addTestResults>
    </org.jenkinsci.plugins.ghprb.upstream.GhprbUpstreamStatus>
  </buildWrappers>
</project>
deweistarsky commented 6 years ago

This is what on github looks like during testing when both jobs are being run:

screenshot from 2018-10-21 11-49-06

After results are posted for the first test: screenshot from 2018-10-21 11-50-56

After the second test is run, see that both tests now fail because the second test failed: screenshot from 2018-10-21 11-59-33

bjoernhaeuser commented 6 years ago

As suspected, your second job is using jenkins-ecs in the org.jenkinsci.plugins.ghprb.extensions.status.GhprbSimpleStatus context, see your xml files.

It using the other context (jenkins-ecs-gpu) only in the org.jenkinsci.plugins.ghprb.upstream.GhprbUpstreamStatus.

bjoernhaeuser commented 6 years ago

You should remove the upstream status from the second job and configure the trigger context instead.

deweistarsky commented 6 years ago

Thanks, that was indeed the issue. The "Trigger setup" was not set up to the correct context.