The CI moves all CI done in our repositories to use the yaml documentation. After adding all distribution metadata to the yaml file in #1017, we are in the position of moving all the -ci-pr_any- generated jobs to use the yaml configuration.
Until now the -ci-pr_any-ubuntu-auto jobs register all supported branches and dispatch the CI work. The new setup creates jobs based on the ubuntu distribution, so now the names are -ci-pr_any-ubuntu-${distro}. Each job register the supported branches (i.e: -ci-pr_any-ubuntu-jammy supports Garden and Harmonic branches).
Some details:
The current -ci-pr_any-ubuntu-auto jobs can not be renamed to the new -ci-pr_any-ubuntu-${distro}. This PR set a transition step of returning always true from the old jobs before removing the completely. This should help people not to focus on the existing failing ones. After some weeks, the jobs will be removed. See 7473a18f44cb9c43b2e108f9ba2f41553db9cf56
We should cover new main branches jobs using the __upcoming__ distribution defined in the yaml file.
The PR still does not generate the branches testing jobs -ci-${branch}- from the yaml file. This is for a new PR to reduce the scope of the testing. fbe24f3ccc74765e3d930e8d3188cbb70003c2ef
The -abichecker- jobs will follow after we check that ci-pr-any work as expected.
Testing done:
I have been using the XML diff to be sure that the configuration match the expectations from both, CI system and local checks. Here is what I have expected and results:
A) All the ci-pr_any-ubuntu-auto jobs should change to be just fake jobs that returns only true.
A1) Testing with the log file from the CI system in release-tools to see if all auto files differ:
/tmp ❯ grep auto log
Files /tmp/current_xml_configuration/ignition_cmake-ci-pr_any-ubuntu_auto-amd64.xml and /tmp/pr_xml_configuration/ignition_cmake-ci-pr_any-ubuntu_auto-amd64.xml differFiles /tmp/current_xml_configuration/ignition_common-ci-pr_any-ubuntu_auto-amd64.xml and /tmp/pr_xml_configuration/ignition_common-ci-pr_any-ubuntu_auto-amd64.xml differ
Files /tmp/current_xml_configuration/ignition_fuel-tools-ci-pr_any-ubuntu_auto-amd64.xml and /tmp/pr_xml_configuration/ignition_fuel-tools-ci-pr_any-ubuntu_auto-amd64.xml differ
Files /tmp/current_xml_configuration/ignition_gazebo-ci-pr_any-ubuntu_auto-amd64.xml and /tmp/pr_xml_configuration/ignition_gazebo-ci-pr_any-ubuntu_auto-amd64.xml differ
Files /tmp/current_xml_configuration/ignition_gui-ci-pr_any-ubuntu_auto-amd64.xml and /tmp/pr_xml_configuration/ignition_gui-ci-pr_any-ubuntu_auto-amd64.xml differ
Files /tmp/current_xml_configuration/ignition_launch-ci-pr_any-ubuntu_auto-amd64.xml and /tmp/pr_xml_configuration/ignition_launch-ci-pr_any-ubuntu_auto-amd64.xml differ
Files /tmp/current_xml_configuration/ignition_math-ci-pr_any-ubuntu_auto-amd64.xml and /tmp/pr_xml_configuration/ignition_math-ci-pr_any-ubuntu_auto-amd64.xml differ
Files /tmp/current_xml_configuration/ignition_msgs-ci-pr_any-ubuntu_auto-amd64.xml and /tmp/pr_xml_configuration/ignition_msgs-ci-pr_any-ubuntu_auto-amd64.xml differ
Files /tmp/current_xml_configuration/ignition_physics-ci-pr_any-ubuntu_auto-amd64.xml and /tmp/pr_xml_configuration/ignition_physics-ci-pr_any-ubuntu_auto-amd64.xml differ
Files /tmp/current_xml_configuration/ignition_plugin-ci-pr_any-ubuntu_auto-amd64.xml and /tmp/pr_xml_configuration/ignition_plugin-ci-pr_any-ubuntu_auto-amd64.xml differ
Files /tmp/current_xml_configuration/ignition_rendering-ci-pr_any-ubuntu_auto-amd64.xml and /tmp/pr_xml_configuration/ignition_rendering-ci-pr_any-ubuntu_auto-amd64.xml differ
Files /tmp/current_xml_configuration/ignition_sensors-ci-pr_any-ubuntu_auto-amd64.xml and /tmp/pr_xml_configuration/ignition_sensors-ci-pr_any-ubuntu_auto-amd64.xml differ
Files /tmp/current_xml_configuration/ignition_tools-ci-pr_any-ubuntu_auto-amd64.xml and /tmp/pr_xml_configuration/ignition_tools-ci-pr_any-ubuntu_auto-amd64.xml differ
Files /tmp/current_xml_configuration/ignition_transport-ci-pr_any-ubuntu_auto-amd64.xml and /tmp/pr_xml_configuration/ignition_transport-ci-pr_any-ubuntu_auto-amd64.xml differ
Files /tmp/current_xml_configuration/ignition_utils-ci-pr_any-ubuntu_auto-amd64.xml and /tmp/pr_xml_configuration/ignition_utils-ci-pr_any-ubuntu_auto-amd64.xml differ
A2) Check a random job to see the returning true and a minimal configuration file:
<project>
<actions></actions>
<description>Automatic generated job by DSL jenkins. Stub job for migration, not doing any check</description>
<keepDependencies>false</keepDependencies>
<properties>
<hudson.model.ParametersDefinitionProperty>
<parameterDefinitions>
<hudson.model.StringParameterDefinition>
<name>sha1</name>
<defaultValue>main</defaultValue>
<description>commit or refname to build. To manually use a branch: origin/$branch_name</description>
</hudson.model.StringParameterDefinition>
</parameterDefinitions>
</hudson.model.ParametersDefinitionProperty>
<com.coravy.hudson.plugins.github.GithubProjectProperty>
<projectUrl>https://github.com/gazebosim/gz-cmake/</projectUrl>
</com.coravy.hudson.plugins.github.GithubProjectProperty>
</properties>
<canRoam>true</canRoam>
<disabled>false</disabled>
<blockBuildWhenDownstreamBuilding>false</blockBuildWhenDownstreamBuilding>
<blockBuildWhenUpstreamBuilding>false</blockBuildWhenUpstreamBuilding>
<triggers>
<org.jenkinsci.plugins.ghprb.GhprbTrigger>
<adminlist>osrf-jenkins j-rivero</adminlist>
<orgslist>osrf gazebosim</orgslist>
<allowMembersOfWhitelistedOrgsAsAdmin>true</allowMembersOfWhitelistedOrgsAsAdmin>
<useGitHubHooks>true</useGitHubHooks>
<onlyTriggerPhrase>false</onlyTriggerPhrase>
<permitAll>true</permitAll>
<spec></spec>
<cron></cron>
<whiteListTargetBranches></whiteListTargetBranches>
<triggerPhrase>.*(re)?run test(s).*</triggerPhrase>
<extensions>
<org.jenkinsci.plugins.ghprb.extensions.status.GhprbSimpleStatus>
<commitStatusContext>${JOB_NAME}</commitStatusContext>
<triggeredStatus>starting deployment to build.osrfoundation.org</triggeredStatus>
<startedStatus>deploying to build.osrfoundation.org</startedStatus>
<addTestResults>true</addTestResults>
</org.jenkinsci.plugins.ghprb.extensions.status.GhprbSimpleStatus>
<org.jenkinsci.plugins.ghprb.extensions.build.GhprbCancelBuildsOnUpdate>
<overrideGlobal>false</overrideGlobal>
</org.jenkinsci.plugins.ghprb.extensions.build.GhprbCancelBuildsOnUpdate>
</extensions>
</org.jenkinsci.plugins.ghprb.GhprbTrigger>
</triggers>
<concurrentBuild>false</concurrentBuild>
<builders></builders>
<publishers></publishers>
<buildWrappers></buildWrappers>
<scm class='hudson.plugins.git.GitSCM'>
<userRemoteConfigs>
<hudson.plugins.git.UserRemoteConfig>
<refspec>+refs/pull/*:refs/remotes/origin/pr/* +refs/heads/*:refs/remotes/origin/*</refspec>
<url>https://github.com/gazebosim/gz-cmake.git</url>
</hudson.plugins.git.UserRemoteConfig>
</userRemoteConfigs>
<branches>
<hudson.plugins.git.BranchSpec>
<name>${sha1}</name>
</hudson.plugins.git.BranchSpec>
</branches>
<configVersion>2</configVersion>
<doGenerateSubmoduleConfigurations>false</doGenerateSubmoduleConfigurations>
<gitTool>Default</gitTool>
<extensions>
<hudson.plugins.git.extensions.impl.RelativeTargetDirectory>
<relativeTargetDir>ign-cmake</relativeTargetDir>
</hudson.plugins.git.extensions.impl.RelativeTargetDirectory>
</extensions>
<browser class='hudson.plugins.git.browser.GithubWeb'>
<url>https://github.com/gazebosim/gz-cmake/</url>
</browser>
</scm>
</project>
C1) There should be focal testing (for Fortress and Garden)
tmp ❯ grep -v differ log | grep focal
Only in /tmp/pr_xml_configuration: gz_cmake-ci-pr_any-focal-amd64.xml
Only in /tmp/pr_xml_configuration: gz_common-ci-pr_any-focal-amd64.xml
Only in /tmp/pr_xml_configuration: gz_fuel_tools-ci-pr_any-focal-amd64.xml
Only in /tmp/pr_xml_configuration: gz_gui-ci-pr_any-focal-amd64.xml
Only in /tmp/pr_xml_configuration: gz_launch-ci-pr_any-focal-amd64.xml
Only in /tmp/pr_xml_configuration: gz_math-ci-pr_any-focal-amd64.xml
Only in /tmp/pr_xml_configuration: gz_msgs-ci-pr_any-focal-amd64.xml
Only in /tmp/pr_xml_configuration: gz_physics-ci-pr_any-focal-amd64.xml
Only in /tmp/pr_xml_configuration: gz_plugin-ci-pr_any-focal-amd64.xml
Only in /tmp/pr_xml_configuration: gz_rendering-ci-pr_any-focal-amd64.xml
Only in /tmp/pr_xml_configuration: gz_sensors-ci-pr_any-focal-amd64.xml
Only in /tmp/pr_xml_configuration: gz_sim-ci-pr_any-focal-amd64.xml
Only in /tmp/pr_xml_configuration: gz_tools-ci-pr_any-focal-amd64.xml
Only in /tmp/pr_xml_configuration: gz_transport-ci-pr_any-focal-amd64.xml
Only in /tmp/pr_xml_configuration: gz_utils-ci-pr_any-focal-amd64.xml
Only in /tmp/pr_xml_configuration: sdformat-ci-pr_any-focal-amd64.xml
C2) Check that branches are the expected. All focal jobs should have two since Garden bumped everything from Fortress:
C3) Bionic testing (for Citadel) Still uses the ign name
Only in /tmp/pr_xml_configuration: ign_cmake-ci-pr_any-bionic-amd64.xml ---- branch2
Only in /tmp/pr_xml_configuration: ign_common-ci-pr_any-bionic-amd64.xml
Only in /tmp/pr_xml_configuration: ign_fuel_tools-ci-pr_any-bionic-amd64.xml
Only in /tmp/pr_xml_configuration: ign_gazebo-ci-pr_any-bionic-amd64.xml
Only in /tmp/pr_xml_configuration: ign_gui-ci-pr_any-bionic-amd64.xml
Only in /tmp/pr_xml_configuration: ign_launch-ci-pr_any-bionic-amd64.xml
Only in /tmp/pr_xml_configuration: ign_math-ci-pr_any-bionic-amd64.xml
Only in /tmp/pr_xml_configuration: ign_msgs-ci-pr_any-bionic-amd64.xml
Only in /tmp/pr_xml_configuration: ign_physics-ci-pr_any-bionic-amd64.xml
Only in /tmp/pr_xml_configuration: ign_plugin-ci-pr_any-bionic-amd64.xml
Only in /tmp/pr_xml_configuration: ign_rendering-ci-pr_any-bionic-amd64.xml
Only in /tmp/pr_xml_configuration: ign_sensors-ci-pr_any-bionic-amd64.xml
Only in /tmp/pr_xml_configuration: ign_tools-ci-pr_any-bionic-amd64.xml
Only in /tmp/pr_xml_configuration: ign_transport-ci-pr_any-bionic-amd64.xml
Only in /tmp/pr_xml_configuration: sdformat-ci-pr_any-bionic-amd64.xml
release-tools2/jenkins-scripts/dsl on jrivero/ci-pr_any-new [?] ❯ grep -3R '<white' ign_*.xml | grep branch
ign_cmake-ci-pr_any-bionic-amd64.xml- <branch>ign-cmake2</branch>
ign_common-ci-pr_any-bionic-amd64.xml- <branch>ign-common3</branch>
ign_fuel_tools-ci-pr_any-bionic-amd64.xml- <branch>ign-fuel-tools4</branch>
ign_gazebo-ci-pr_any-bionic-amd64.xml- <branch>ign-gazebo3</branch>
ign_gui-ci-pr_any-bionic-amd64.xml- <branch>ign-gui3</branch>
ign_launch-ci-pr_any-bionic-amd64.xml- <branch>ign-launch2</branch>
ign_math-ci-pr_any-bionic-amd64.xml- <branch>ign-math6</branch>
ign_msgs-ci-pr_any-bionic-amd64.xml- <branch>ign-msgs5</branch>
ign_physics-ci-pr_any-bionic-amd64.xml- <branch>ign-physics2</branch>
ign_plugin-ci-pr_any-bionic-amd64.xml- <branch>ign-plugin1</branch>
ign_rendering-ci-pr_any-bionic-amd64.xml- <branch>ign-rendering3</branch>
ign_sensors-ci-pr_any-bionic-amd64.xml- <branch>ign-sensors3</branch>
ign_tools-ci-pr_any-bionic-amd64.xml- <branch>ign-tools1</branch>
ign_transport-ci-pr_any-bionic-amd64.xml- <branch>ign-transport8</branch>
The CI moves all CI done in our repositories to use the yaml documentation. After adding all distribution metadata to the yaml file in #1017, we are in the position of moving all the
-ci-pr_any-generated jobs to use the yaml configuration.Until now the
-ci-pr_any-ubuntu-autojobs register all supported branches and dispatch the CI work. The new setup creates jobs based on the ubuntu distribution, so now the names are-ci-pr_any-ubuntu-${distro}. Each job register the supported branches (i.e:-ci-pr_any-ubuntu-jammysupports Garden and Harmonic branches).Some details:
-ci-pr_any-ubuntu-autojobs can not be renamed to the new-ci-pr_any-ubuntu-${distro}. This PR set a transition step of returning alwaystruefrom the old jobs before removing the completely. This should help people not to focus on the existing failing ones. After some weeks, the jobs will be removed. See 7473a18f44cb9c43b2e108f9ba2f41553db9cf56mainbranches jobs using the__upcoming__distribution defined in the yaml file.-ci-${branch}-from the yaml file. This is for a new PR to reduce the scope of the testing. fbe24f3ccc74765e3d930e8d3188cbb70003c2efTesting done:
I have been using the XML diff to be sure that the configuration match the expectations from both, CI system and local checks. Here is what I have expected and results:
A) All the
ci-pr_any-ubuntu-autojobs should change to be just fake jobs that returns only true.A1) Testing with the log file from the CI system in release-tools to see if all auto files differ:
A2) Check a random job to see the returning true and a minimal configuration file:
A3) Test in Jenkins that new fake jobs works returning just true: https://build.osrfoundation.org/job/_test_migration_ci_pr_any/2/
B) Existing
ci-pr_any-ubuntu-jammyjobs created for Harmonic and in use would need to support the branches in upcoming:B1) XML log diff should report only jammy jobs if we don't include the -auto jobs mentioned in A).
B2) Changes in the content of these files should show the
mainbranch be added:B3) Check all new white listed branches to see if they have sense. We expect main and latest released branches:
C) New jobs added in this PR:
C1) There should be focal testing (for Fortress and Garden)
C2) Check that branches are the expected. All focal jobs should have two since Garden bumped everything from Fortress:
C3) Bionic testing (for Citadel) Still uses the ign name