check-threads-count.rb and metrics-processes-threads-count.rb fail with a `nil:NilClass` when comparing versions.

[rockandska]

Hi,

Is there anyone else who face this issue too with check-threads-count.rb ?

bash-4.3# ruby --version
ruby 2.4.2p198 (2017-09-14 revision 59899) [x86_64-linux-musl]
bash-4.3# bundle --version
Bundler version 1.16.0
bash-4.3# gem list sys-proctable sensu-plugins-process-checks

*** LOCAL GEMS ***

sys-proctable (0.9.8 universal-linux)

*** LOCAL GEMS ***

sensu-plugins-process-checks (2.4.0)

bash-4.3# check-threads-count.rb -c 1
Check failed to run: undefined method `version' for nil:NilClass, ["/usr/local/bundle/gems/sensu-plugins-process-checks-2.4.0/bin/check-threads-count.rb:55:in `check_proctable_version'", "/usr/local/bundle/gems/sensu-plugins-process-checks-2.4.0/bin/check-threads-count.rb:89:in `run'", "/usr/local/bundle/gems/sensu-plugin-1.3.0/lib/sensu-plugin/cli.rb:57:in `block in <class:CLI>'"]
bash-4.3# 

If I remove th check on this version; everything is ok :

bash-4.3# sed -i "s/Gem.loaded_specs\['sys-proctable'\].*/true/" /usr/local/bundle/gems/sensu-plugins-process-checks-*/bin/*.rb
bash-4.3# check-threads-count.rb -c 1
ThreadsCount CRITICAL: 588 threads running, over threshold 1

Same behavior with metrics-processes-threads-count.rb if we use -t option

[majormoses]

Sorry for the late response, taking a look at the code now it certainly seems a bit fishy. Can you please confirm that you have installed sys-proctable?

One thing I noticed is that you are not using an embedded ruby version from sensu client which is not recommended. Can you test if you see the same issue on ruby 2.4.1 which is the closest version we support.

[majormoses]

This logic makes me sad: https://github.com/sensu-plugins/sensu-plugins-process-checks/blob/2.5.0/bin/check-threads-count.rb#L89-L91 I can put in a PR to make this more sense but I am not sure if that is actually your problem.

[majormoses]

I am not able to replicate your nil error with the gem installed:

$ ./bin/check-threads-count.rb -c 1
ThreadsCount CRITICAL: 2584 threads running, over threshold 1

One thing I do see is that the dependency is potentially a bit strict: https://github.com/sensu-plugins/sensu-plugins-process-checks/blob/2.5.0/sensu-plugins-process-checks.gemspec#L36

Can you please show the output of:

$ gem list sys-proctable

[majormoses]

I am still sorta confused why you are seeing that, the information requested above should probably shed some light on the matter. I have proposed a PR bumping the min dependency and removing the check as it no longer serves a purpose.

[rockandska]

Sorry for my late answer.

Can you please show the output of: $ gem list sys-proctable

Already do on my 1st message :)

bash-4.3# gem list sys-proctable sensu-plugins-process-checks

*** LOCAL GEMS ***

sys-proctable (0.9.8 universal-linux)

*** LOCAL GEMS ***

sensu-plugins-process-checks (2.4.0)

One thing I noticed is that you are not using an embedded ruby version from sensu client which is not recommended. Can you test if you see the same issue on ruby 2.4.1 which is the closest version we support.

Indeed, i will try to test it with 2.4.1 and also will give a try to provide you a way to reproduce this behavior (PS: running it on docker under alpine)

[rockandska]

Same error with ruby 2.4.1 under Alpine

bash-4.3# ruby --version
ruby 2.4.1p111 (2017-03-22 revision 58053) [x86_64-linux-musl]

bash-4.3# gem list sys-proctable sensu-plugins-process-checks

*** LOCAL GEMS ***

sys-proctable (0.9.8 universal-linux)

*** LOCAL GEMS ***

sensu-plugins-process-checks (2.4.0)

bash-4.3# check-threads-count.rb -c 1
Check failed to run: undefined method `version' for nil:NilClass, ["/usr/local/bundle/gems/sensu-plugins-process-checks-2.4.0/bin/check-threads-count.rb:55:in `check_proctable_version'", "/usr/local/bundle/gems/sensu-plugins-process-checks-2.4.0/bin/check-threads-count.rb:89:in `run'", "/usr/local/bundle/gems/sensu-plugin-1.4.5/lib/sensu-plugin/cli.rb:58:in `block in <class:CLI>'"]

bash-4.3# sed -i "s/Gem.loaded_specs\['sys-proctable'\].*/true/" /usr/local/bundle/gems/sensu-plugins-process-checks-*/bin/*.rb

bash-4.3# check-threads-count.rb -c 1
ThreadsCount CRITICAL: 725 threads running, over threshold 1

I will try to provide you a complete test case soon

[majormoses]

sorry must have missed that, can you confirm that my PR fixes your issue?

[rockandska]

I can't reproduce the bug myself from a clean environnement.... what a shame... It seems than my bug is relate to the way i built the gems inside my docker image but can't figure why because the logic is the same. I only use "builder" to install gems from Gemfile instead directly with gem.

$ cat Gemfile
source 'https://rubygems.org'

gem 'sensu', '1.1.0'
gem 'sensu-plugins-cpu-checks',         '1.1.2'
gem 'sensu-plugins-docker',             '1.5.0'
gem 'sensu-plugins-disk-checks',        '2.5.0'
gem 'sensu-plugins-io-checks',          '1.0.1'
gem 'sensu-plugins-load-checks',        '3.0.0'
gem 'sensu-plugins-memory-checks',      '3.0.3'
gem 'sensu-plugins-network-checks',     '2.1.0'
gem 'sensu-plugins-process-checks',     '2.4.0'
gem 'vmstat',                           '2.3.0'

I need to understand which action inside my Dockerfile is responsible to hit this bug. I'm sorry, i should test from clean a clean environnement from the start....

Clean environnement with ruby 2.4.1 and sensu-plugins/sensu-plugins-process-checks 2.4.0

$ docker run -it --rm --name test-check --net host  --privileged  --pid host --ipc host 
-v /run:/run:ro 
-v /etc/localtime:/etc/localtime:ro 
-v /sys/fs/cgroup:/sys/fs/cgroup:ro 
-v /:/host:ro 
-v /var/log:/var/log 
--entrypoint /bin/sh  ruby:2.4.1-alpine3.4
/ # apk add -qU build-base curl curl-dev git libffi-dev ruby-dev ruby-io-console
/ # gem list

*** LOCAL GEMS ***

bigdecimal (default: 1.3.0)
bundler (1.15.4)
did_you_mean (1.1.0)
io-console (default: 0.4.6)
json (default: 2.0.2)
minitest (5.10.1)
net-telnet (0.1.1)
openssl (default: 2.0.3)
power_assert (0.4.1)
psych (default: 2.2.2)
rake (12.0.0)
rdoc (default: 5.0.0)
rubygems-update (2.6.13)
test-unit (3.2.3)
xmlrpc (0.2.1)
/ # gem install sensu-plugins-process-checks:2.4.0
Fetching: language-0.6.0.gem (100%)
Successfully installed language-0.6.0
Fetching: english-0.6.3.gem (100%)
Successfully installed english-0.6.3
Fetching: json-1.8.6.gem (100%)
Building native extensions.  This could take a while...
Successfully installed json-1.8.6
Fetching: mixlib-cli-1.7.0.gem (100%)
Successfully installed mixlib-cli-1.7.0
Fetching: sensu-plugin-1.4.5.gem (100%)
Successfully installed sensu-plugin-1.4.5
Fetching: sys-proctable-0.9.8-universal-linux.gem (100%)
Successfully installed sys-proctable-0.9.8-universal-linux
Fetching: sensu-plugins-process-checks-2.4.0.gem (100%)
You can use the embedded Ruby by setting EMBEDDED_RUBY=true in /etc/default/sensu
Successfully installed sensu-plugins-process-checks-2.4.0
7 gems installed
/ # gem list

*** LOCAL GEMS ***

bigdecimal (default: 1.3.0)
bundler (1.15.4)
did_you_mean (1.1.0)
english (0.6.3)
io-console (default: 0.4.6)
json (default: 2.0.2, 1.8.6)
language (0.6.0)
minitest (5.10.1)
mixlib-cli (1.7.0)
net-telnet (0.1.1)
openssl (default: 2.0.3)
power_assert (0.4.1)
psych (default: 2.2.2)
rake (12.0.0)
rdoc (default: 5.0.0)
rubygems-update (2.6.13)
sensu-plugin (1.4.5)
sensu-plugins-process-checks (2.4.0)
sys-proctable (0.9.8 universal-linux)
test-unit (3.2.3)
xmlrpc (0.2.1)

/ # time check-threads-count.rb -c 1
ThreadsCount CRITICAL: 630 threads running, over threshold 1
Command exited with non-zero status 2
real    0m 0.58s
user    0m 0.44s
sys 0m 0.14s

:warning: Your PR (#56 ) works too, but be aware than the update of sys-proctable seems to introduce this bug

Look a the time took for the check with your PR :

$ docker run -it --rm --name test-check --net host  --privileged  --pid host --ipc host 
-v /run:/run:ro 
-v /etc/localtime:/etc/localtime:ro 
-v /sys/fs/cgroup:/sys/fs/cgroup:ro 
-v /:/host:ro 
-v /var/log:/var/log 
--entrypoint /bin/sh  ruby:2.4.1-alpine3.4
/ # apk add -qU build-base curl curl-dev git libffi-dev ruby-dev ruby-io-console
/ # git clone https://github.com/sensu-plugins/sensu-plugins-process-checks.git /tmp/sensu-plugins-process-checks.git
Cloning into '/tmp/sensu-plugins-process-checks.git'...
remote: Counting objects: 593, done.
remote: Compressing objects: 100% (10/10), done.
remote: Total 593 (delta 2), reused 2 (delta 0), pack-reused 583
Receiving objects: 100% (593/593), 125.17 KiB | 0 bytes/s, done.
Resolving deltas: 100% (368/368), done.
Checking connectivity... done.
/ # cd /tmp/sensu-plugins-process-checks.git
/tmp/sensu-plugins-process-checks.git # git checkout feature/cleanup-sys-proctable
Branch feature/cleanup-sys-proctable set up to track remote branch feature/cleanup-sys-proctable from origin.
Switched to a new branch 'feature/cleanup-sys-proctable'
/tmp/sensu-plugins-process-checks.git # gem build sensu-plugins-process-checks.gemspec 
  Successfully built RubyGem
  Name: sensu-plugins-process-checks
  Version: 2.5.0
  File: sensu-plugins-process-checks-2.5.0.gem
/tmp/sensu-plugins-process-checks.git # gem install sensu-plugins-process-checks-2.5.0.gem 
Fetching: sys-proctable-1.1.5-universal-linux.gem (100%)
Successfully installed sys-proctable-1.1.5-universal-linux
Fetching: json-1.8.6.gem (100%)
Building native extensions.  This could take a while...
Successfully installed json-1.8.6
Fetching: mixlib-cli-1.7.0.gem (100%)
Successfully installed mixlib-cli-1.7.0
Fetching: sensu-plugin-1.4.5.gem (100%)
Successfully installed sensu-plugin-1.4.5
Fetching: language-0.6.0.gem (100%)
Successfully installed language-0.6.0
Fetching: english-0.6.3.gem (100%)
Successfully installed english-0.6.3
You can use the embedded Ruby by setting EMBEDDED_RUBY=true in /etc/default/sensu
Successfully installed sensu-plugins-process-checks-2.5.0
7 gems installed
/tmp/sensu-plugins-process-checks.git # gem list

*** LOCAL GEMS ***

bigdecimal (default: 1.3.0)
bundler (1.15.4)
did_you_mean (1.1.0)
english (0.6.3)
io-console (default: 0.4.6)
json (default: 2.0.2, 1.8.6)
language (0.6.0)
minitest (5.10.1)
mixlib-cli (1.7.0)
net-telnet (0.1.1)
openssl (default: 2.0.3)
power_assert (0.4.1)
psych (default: 2.2.2)
rake (12.0.0)
rdoc (default: 5.0.0)
rubygems-update (2.6.13)
sensu-plugin (1.4.5)
sensu-plugins-process-checks (2.5.0)
sys-proctable (1.1.5 universal-linux)
test-unit (3.2.3)
xmlrpc (0.2.1)

/tmp/sensu-plugins-process-checks.git # time check-threads-count.rb -c 1
ThreadsCount CRITICAL: 646 threads running, over threshold 1
Command exited with non-zero status 2
real    0m 11.30s
user    0m 10.43s
sys 0m 0.83s

0.58s VS 11.30s :astonished:

[majormoses]

oh boy that looks like fun. I moved your comment over to the PR regarding performance issues. I am not sure if we should close this and open an new issue to discuss the performance stuff with newer versions of the gem. Since you can't replicate your issues on a clean instance I'd say we close this out. I will see if I can spend some time looking at that and seeing if we can improve the performance.

[rockandska]

I'm closing it due to the fact i can't reproduce it myself with a clean environment. For the performance issue with the new version of sys-proctable, i think someone else should confirm the behavior. The best fix from my vision will be to let the previous version of gemspec who fix the version of sys-proctable to use that seems to not have this bug