search

ros-infrastructure / ros_buildfarm

ROS buildfarm based on Docker
Apache License 2.0
81 stars 96 forks source link

jobs from a single trigger doubly queued #475

Closed tfoote closed 7 years ago

tfoote commented 7 years ago

I noticed a couple instances of mavros being doubly queued after the latest release.

With the blocking on upstream the first build should not have started while there were still upstream packages building. We have Block build when upstream project is building enabled. I believe this is some sort of race condition in the queueing/checking for blocking that possibly should be reported upstream.

I've seen this before but never quite as short a trigger queue, nor as repeatable as two arches on the same release.

Details

Build 45 was in the queue, while build 44 was building.

However they were both ultimately triggered by 45539, but it allowed 44 to start despite the fact that upstream jobs were either pending or underway as per the trigger chain for 45.

45 trigger logic

Started by upstream project Kbin_uxv8_uXv8__mavros_msgs__ubuntu_xenial_arm64__binary build number 18
originally caused by:

Started by upstream project Ksrc_uX__mavros_msgs__ubuntu_xenial__source build number 17
originally caused by:
Started by upstream project Krel_trigger-jobs build number 45539
originally caused by:
Started by timer

44 trigger logic

Started by upstream project Ksrc_uX__mavros__ubuntu_xenial__source build number 17
originally caused by:

Started by upstream project Krel_trigger-jobs build number 45539
originally caused by:
Started by timer
Started by upstream project Kbin_uxv8_uXv8__libmavconn__ubuntu_xenial_arm64__binary build number 25
originally caused by:

Started by upstream project Ksrc_uX__libmavconn__ubuntu_xenial__source build number 17
originally caused by:
Started by upstream project Krel_trigger-jobs build number 45539
originally caused by:
Started by timer

Dependency listings:

Upstream Projects

SuccessKbin_uxv8_uXv8__angles__ubuntu_xenial_arm64__binary
SuccessKbin_uxv8_uXv8__catkin__ubuntu_xenial_arm64__binary
SuccessKbin_uxv8_uXv8__cmake_modules__ubuntu_xenial_arm64__binary
SuccessKbin_uxv8_uXv8__diagnostic_msgs__ubuntu_xenial_arm64__binary
SuccessKbin_uxv8_uXv8__diagnostic_updater__ubuntu_xenial_arm64__binary
SuccessKbin_uxv8_uXv8__eigen_conversions__ubuntu_xenial_arm64__binary
SuccessKbin_uxv8_uXv8__geographic_msgs__ubuntu_xenial_arm64__binary
SuccessKbin_uxv8_uXv8__geometry_msgs__ubuntu_xenial_arm64__binary
SuccessKbin_uxv8_uXv8__libmavconn__ubuntu_xenial_arm64__binary
SuccessKbin_uxv8_uXv8__mavlink__ubuntu_xenial_arm64__binary
SuccessKbin_uxv8_uXv8__mavros_msgs__ubuntu_xenial_arm64__binary
SuccessKbin_uxv8_uXv8__message_runtime__ubuntu_xenial_arm64__binary
SuccessKbin_uxv8_uXv8__nav_msgs__ubuntu_xenial_arm64__binary
SuccessKbin_uxv8_uXv8__pluginlib__ubuntu_xenial_arm64__binary
SuccessKbin_uxv8_uXv8__rosconsole_bridge__ubuntu_xenial_arm64__binary
SuccessKbin_uxv8_uXv8__roscpp__ubuntu_xenial_arm64__binary
SuccessKbin_uxv8_uXv8__rospy__ubuntu_xenial_arm64__binary
SuccessKbin_uxv8_uXv8__sensor_msgs__ubuntu_xenial_arm64__binary
SuccessKbin_uxv8_uXv8__std_msgs__ubuntu_xenial_arm64__binary
SuccessKbin_uxv8_uXv8__std_srvs__ubuntu_xenial_arm64__binary
SuccessKbin_uxv8_uXv8__tf2_ros__ubuntu_xenial_arm64__binary
SuccessKsrc_uX__mavros__ubuntu_xenial__source
Downstream Projects

SuccessKbin_uxv8_uXv8__mavros_extras__ubuntu_xenial_arm64__binary
SuccessKbin_uxv8_uXv8__test_mavros__ubuntu_xenial_arm64__binary
SuccessKrel_sync-packages-to-testing_xenial_arm64

I saw the same behavior on the armhf from the same release: http://build.ros.org/view/Queue/job/Kbin_uxhf_uXhf__mavros__ubuntu_xenial_armhf__binary/

67 triggers: 

Started by upstream project Kbin_uxhf_uXhf__mavros_msgs__ubuntu_xenial_armhf__binary build number 29
originally caused by:

Started by upstream project Ksrc_uX__mavros_msgs__ubuntu_xenial__source build number 17
originally caused by:
Started by upstream project Krel_trigger-jobs build number 45539
originally caused by:
Started by timer

66 triggers:

Started by upstream project Ksrc_uX__mavros__ubuntu_xenial__source build number 17
originally caused by:

Started by upstream project Krel_trigger-jobs build number 45539
originally caused by:
Started by timer
Started by upstream project Kbin_uxhf_uXhf__libmavconn__ubuntu_xenial_armhf__binary build number 44
originally caused by:

Started by upstream project Ksrc_uX__libmavconn__ubuntu_xenial__source build number 17
originally caused by:
Started by upstream project Krel_trigger-jobs build number 45539
originally caused by:
Started by timer
dirk-thomas commented 7 years ago

Since the problem is within Jenkins I don't see a reason to track this in the issue tracker of this repository. So I am going to close this.

I think the relevant upstream ticket is https://issues.jenkins-ci.org/browse/JENKINS-5150 (which I have already commented on 1.5 yrs ago).

tfoote commented 7 years ago

Sounds good. If anyone else runs into this please upvote the upstream jenkins issue and maybe it will get addressed eventually.