Closed Crola1702 closed 1 year ago
Linux GPU Agents make build fail early because of unattended-upgrades.
I'm assuming that we prefer to have unattended-upgrades
running on all the packages that trying to pin/freeze the affected packages so we don't have the problem until we decide to update the image for a good reason.
- catch this error in script and send a sigil to the Jenkins post-build runner which
I've being using the "Failure Cause Management" plugin (configuration is stored in chef) for a while to notify a message in the main build page when a regexp is being found but a part of this I don't think that we can do much more with that plugin.
- marks the Jenkins host offline
- re-queues the job
The ros_buidlfarm repo hosts groovy scripts that can serve you to create a custom solution for this. For example this one checks for a regexp in the log or this one disables a node and log the problem.
@j-rivero I'm blocked with this task. Here has been my approaches so far:
Would appreciate any insight you might have on how to do :
Fixed by #915
Sometimes, Linux GPU Agents make build fail early because of unattended-upgrades.
Reference build: https://build.osrfoundation.org/job/ignition_gazebo-ci-gz-sim7-focal-amd64/91/console
Log output:
The solution for now is rebooting the agent each time it happens (check this StackOverflow answer), but we should automate the response to this problem via: