Update healthchecks to handle ldd failures

adfoster-r7 commented 9 months ago

Description

Relates to https://github.com/chef/omnibus/pull/1056 which added more rigorous checks to the ldd healthchecks

The scenario is - I've just upgraded to a newer version of Omnibus, and it looks like the ldd command here can break halfway through with this current pattern. It took a while to track down things as the status code here is getting ignored, so the error was being swallowed.

Example, reading 3 files and the 2nd ELF file causes an error - shows that the third file is never evaluated - leading to incorrect health checks:

[9] pry(#<Omnibus::HealthCheck>)> puts ldd_output = shellout(ldd_command, input: "/opt/project/file1\n/opt/project/file2\n/opt/project/file3").stdout
/opt/project/file1:
    not a dynamic executable
/opt/project/file2:
=> nil

We get a partial result in the current implementation, but if you extract the status code you can see it has failed an exitstatus of 135:

[11] pry(#<Omnibus::HealthCheck>)> puts ldd_output = shellout(ldd_command, input: "/opt/project/file1\n/opt/project/file2\n/opt/project/file3").result
NoMethodError: undefined method `result' for <Mixlib::ShellOut#1230: command: 'xargs ldd' process_status: #<Process::Status: pid 46708 exit 123> stdout: '/opt/project/file1:
    not a dynamic executable
/opt/project/file2:' stderr: 'ldd: exited with unknown exit code (135)' child_pid: 46708 environment: {} timeout: 7200 user:  group:  working_dir:  >:Mixlib::ShellOut
from (pry):11:in `block in read_shared_libs'

For my current setup, this new code path skips multiple healthchecks as a result of exiting earlier than expect

Intended solution

For the happy path, the ldd command is run against all libraries as before. If the exit code is non-zero, we try the failed files indivdually.

Maintainers

Please ensure that you check for:

[ ] If this change impacts git cache validity, it bumps the git cache serial number
[ ] If this change impacts compatibility with omnibus-software, the corresponding change is reviewed and there is a release plan
[ ] If this change impacts compatibility with the omnibus cookbook, the corresponding change is reviewed and there is a release plan