`nvml error: driver/library version mismatch` in Linux GPU agents

[Crola1702]

Sometimes, Linux GPU Agents make build fail early because of unattended-upgrades.

Reference build: https://build.osrfoundation.org/job/ignition_gazebo-ci-gz-sim7-focal-amd64/91/console

Log output:

docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #1: error running hook: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy'
nvidia-container-cli: initialization error: nvml error: driver/library version mismatch: unknown.
Build step 'Execute shell' marked build as failure

The solution for now is rebooting the agent each time it happens (check this StackOverflow answer), but we should automate the response to this problem via:

1. catch this error in script and send a sigil to the Jenkins post-build runner which
2. marks the Jenkins host offline
3. re-queues the job
4. forces a reboot of the Jenkins agent

[j-rivero]

Linux GPU Agents make build fail early because of unattended-upgrades.

I'm assuming that we prefer to have unattended-upgrades running on all the packages that trying to pin/freeze the affected packages so we don't have the problem until we decide to update the image for a good reason.

1. catch this error in script and send a sigil to the Jenkins post-build runner which

I've being using the "Failure Cause Management" plugin (configuration is stored in chef) for a while to notify a message in the main build page when a regexp is being found but a part of this I don't think that we can do much more with that plugin.

2. marks the Jenkins host offline
3. re-queues the job

The ros_buidlfarm repo hosts groovy scripts that can serve you to create a custom solution for this. For example this one checks for a regexp in the log or this one disables a node and log the problem.

[claraberendsen]

@j-rivero I'm blocked with this task. Here has been my approaches so far:

• First approach, run everything in a post build groovy script. Blocker here is restarting the agent (running a bash or shell script on agent from the system groovy script)
• Second approach, split the steps and run it as postbuildSteps. Having some errors on the compilation of dsl. (Current approach working on making it work 😅)

Would appreciate any insight you might have on how to do :

• Run a command from the groovy script that is in the Jenkins controller on the agent.
• Or if you have time check what might be causing the compilation error.

[j-rivero]

Fixed by #915