collectd-cloudwatch stopped, After I updated collectd.

[sasagawayou]

Hi,

Thank you for making great plugins. But I have a problem right now. The collectd-cloudwatch stopped working, After I updated collectd. Please tell me how to complete this properly.

First I installed yum install -y collectd, And I run to setup.py. It's worked. But I tried to upgrade collectd because I wanted to get disk data. When I implemented it, I got an error like the one below.

[ec2-user@10-0-0-5 ~]]$ sudo ./setup.py 
Installing dependencies ... OK
Installing python dependencies ... OK
Downloading plugin ... OK
Extracting plugin ... OK
Moving to collectd plugins directory ... OK
Copying CloudWatch plugin include file ... OK

Choose AWS region for published metrics:
  1. Automatic [xxxxxxxx]
  2. Custom
Enter choice [1]: 

Choose hostname for published metrics:
  1. EC2 instance id [xxxxxxxx]
  2. Custom
Enter choice [1]: 

Choose authentication method:
  1. IAM Role [xxxxxxxx]
  2. IAM User
Enter choice [1]: 

Enter proxy server name:
  1. None
  2. Custom
Enter choice [1]: 

Enter proxy server port:
  1. None
  2. Custom
Enter choice [1]: 

Choose how to install CloudWatch plugin in collectd:
  1. Do not modify existing collectd configuration
  2. Add plugin to the existing configuration
  3. Use CloudWatch recommended configuration (4 metrics)
Enter choice [3]: 
Plugin configuration written successfully.
Creating backup of the original configuration ... OK
Replacing collectd configuration ... OK
Replacing whitelist configuration ... OK
Stopping collectd process ... NOT OK
Starting collectd process ... NOT OK
Installation cancelled due to an error.
Executed command: '/usr/sbin/collectd'.
Error output: '[2017-04-03 16:45:53] plugin_load: plugin "logfile" successfully loaded.
Error: Reading the config file failed!
Read the logs for details.'.

So I read collectd.log

[2017-04-03 16:45:53] [error] python plugin: Error importing module "cloudwatch_writer".
[2017-04-03 16:45:53] [error] Unhandled python exception in importing module: ImportError: No module named requests
[2017-04-03 16:45:53] [error] Traceback (most recent call last):
[2017-04-03 16:45:53] [error]   File "/opt/collectd-plugins/cloudwatch_writer.py", line 10, in <module>
    from cloudwatch.modules.configuration.confighelper import ConfigHelper
[2017-04-03 16:45:53] [error]   File "/opt/collectd-plugins/cloudwatch/modules/configuration/confighelper.py", line 4, in <module>
    from metadatareader import MetadataReader
[2017-04-03 16:45:53] [error]   File "/opt/collectd-plugins/cloudwatch/modules/configuration/metadatareader.py", line 2, in <module>
    from requests import Session, codes
[2017-04-03 16:45:53] [error] ImportError: No module named requests

I thought whether the requests library does not exist. But it seems to exist.

[ec2-user@10-0-0-5 ~]$ pip freeze
aws-cfn-bootstrap==1.4
awscli==1.11.29
Babel==0.9.4
backports.ssl-match-hostname==3.4.0.2
boto==2.42.0
botocore==1.4.86
chardet==2.0.1
cloud-init==0.7.6
colorama==0.2.5
configobj==4.7.2
docutils==0.11
ecdsa==0.11
futures==3.0.3
iniparse==0.3.1
Jinja2==2.7.2
jmespath==0.9.0
jsonpatch==1.2
jsonpointer==1.0
kitchen==1.1.1
lockfile==0.8
MarkupSafe==0.11
paramiko==1.15.1
PIL==1.1.6
ply==3.4
pyasn1==0.1.7
pycrypto==2.6.1
pycurl==7.19.0
pygpgme==0.3
pyliblzma==0.5.3
pystache==0.5.3
python-daemon==1.5.2
python-dateutil==2.1
pyxattr==0.5.0
PyYAML==3.10
requests==2.13.0
rsa==3.4.1
simplejson==3.6.5
six==1.8.0
urlgrabber==3.9.1
urllib3==1.8.2
virtualenv==12.0.7
yum-metadata-parser==1.1.4

[ec2-user@10-0-0-5 ~]$ ls -la /usr/local/lib/python2.7/site-packages/
total 28
drwxr-xr-x 5 root root 4096 Apr  3 16:45 .
drwxr-xr-x 3 root root 4096 Jan 21 08:25 ..
-rw-r--r-- 1 root root  119 Sep  2  2016 README
-rw-r--r-- 1 root root  207 Mar 28 18:52 easy-install.pth
drwxr-xr-x 4 root root 4096 Mar 28 18:52 pip-9.0.1-py2.7.egg
drwxr-xr-x 3 root root 4096 Apr  3 16:45 requests
drwxr-xr-x 2 root root 4096 Apr  3 16:45 requests-2.13.0.dist-info

I would like to know how to solve this problem. Incidentally, the collectd currently installed is as follows. (i tried version 5.5/5.6/5.7, all version happen same problem.)

[ec2-user@10-0-0-5 ~]$ collectd -h
Usage: collectd [OPTIONS]

Available options:
  General:
    -C <file>       Configuration file.
                    Default: /etc/collectd.conf
    -t              Test config and exit.
    -T              Test plugin read and exit.
    -P <file>       PID-file.
                    Default: /var/run/collectd.pid
    -f              Don't fork to the background.
    -h              Display help (this message)

Builtin defaults:
  Config file       /etc/collectd.conf
  PID file          /var/run/collectd.pid
  Plugin directory  /usr/lib64/collectd
  Data directory    /var/lib/collectd

collectd 5.5.3.1.g4f9adee, http://collectd.org/
by Florian octo Forster <octo@collectd.org>
for contributions see `AUTHORS'

[ec2-user@10-0-0-5 ~]$ ls -la | grep -e "collectd*"
-rw-rw-r--   1 ec2-user ec2-user    525176 Nov 29 20:19 collectd-5.5.3.1.g4f9adee-1.el6.x86_64.rpm
-rw-rw-r--   1 ec2-user ec2-user    533672 Mar  6 03:27 collectd-5.6.2.26.g294aa61-4.el6.x86_64.rpm
-rw-rw-r--   1 ec2-user ec2-user    566148 Mar 11 00:22 collectd-5.7.1.34.g3f710d0-3.el6.x86_64.rpm
-rw-rw-r--   1 ec2-user ec2-user    641152 Mar 11 00:17 collectd-5.7.1.34.g3f710d0-3.el7.centos.x86_64.rpm
-rw-rw-r--   1 ec2-user ec2-user     15564 Nov 29 20:19 collectd-disk-5.5.3.1.g4f9adee-1.el6.x86_64.rpm
-rw-rw-r--   1 ec2-user ec2-user     16160 Mar  6 03:27 collectd-disk-5.6.2.26.g294aa61-4.el6.x86_64.rpm
-rw-rw-r--   1 ec2-user ec2-user     16364 Mar 11 00:22 collectd-disk-5.7.1.34.g3f710d0-3.el6.x86_64.rpm
-rw-rw-r--   1 ec2-user ec2-user     43004 Nov 29 20:19 collectd-python-5.5.3.1.g4f9adee-1.el6.x86_64.rpm
-rw-rw-r--   1 ec2-user ec2-user     43928 Mar  6 03:27 collectd-python-5.6.2.26.g294aa61-4.el6.x86_64.rpm
-rw-rw-r--   1 ec2-user ec2-user     44200 Mar 11 00:23 collectd-python-5.7.1.34.g3f710d0-3.el6.x86_64.rpm

Thank you for your helpful. Best regards. Yo

[yimuniao]

Could you check if there are multi versions of collectd in your system? For example: /sbin/collectd, /opt/collectd/....

[sasagawayou]

Thank you fo replay! I checked it, But I found collectd is only /usr/sbin/collectd ...

[ec2-user@10-0-0-5 ~]$ sudo find / -name "collectd"
/usr/include/collectd           #directory
/usr/lib64/collectd             #directory
/usr/share/collectd             #directory
/usr/sbin/collectd              #use command
/etc/rc.d/init.d/collectd       #sysV init program
/var/lib/collectd               #directory

[sasagawayou]

And share for my OS version.

[ec2-user@10-0-0-5 src]$ uname -a
Linux ip-10-0-0-5 4.4.44-39.55.amzn1.x86_64 #1 SMP Mon Jan 30 18:15:53 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

[yimuniao]

I suggest you reinstall the collectd, and check if it works without this plugin

[yimuniao]

Looks like the dependent library has problem. could you run "easy_install -U --quiet requests" to check the result?

[sasagawayou]

Hi, I resolved it, I checked python version. Before run setup.py.

[ec2-user@10-0-0-5 src]$ whereis python
python: /usr/bin/python /usr/bin/python2.7 /usr/bin/python2.7-config /usr/lib/python2.7 /usr/lib64/python2.7 /usr/local/lib/python2.7 /usr/include/python2.7 /usr/share/man/man1/python.1.gz

After run it.

[ec2-user@10-0-0-5 src]$ whereis python
python: /usr/bin/python /usr/bin/python2.7 /usr/bin/python2.7-config /usr/bin/python2.6 /usr/lib/python2.7 /usr/lib/python2.6 /usr/lib64/python2.7 /usr/lib64/python2.6 /usr/local/lib/python2.7 /usr/local/lib/python2.6 /usr/include/python2.7 /usr/include/python2.6 /usr/share/man/man1/python.1.gz

I thought python 2.6 was used in setup.py. So I used python-pip to install requests into the 2.6 library.

[ec2-user@10-0-0-5 src]$ pip -V
pip 6.1.1 from /usr/lib/python2.7/dist-packages (python 2.7)
[ec2-user@10-0-0-5 src]$ python-pip -V
pip 6.1.1 from /usr/lib/python2.6/dist-packages (python 2.6)
[ec2-user@10-0-0-5 src]$ ls -la /usr/local/lib/python2.6/site-packages/
total 20
drwxr-xr-x 4 root root 4096 Apr  9 22:36 .
drwxr-xr-x 3 root root 4096 Apr  9 22:14 ..
-rw-r--r-- 1 root root  119 Sep  2  2016 README

[ec2-user@10-0-0-5 src]$ sudo python-pip install requests
You are using pip version 6.1.1, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Collecting requests
  Using cached requests-2.13.0-py2.py3-none-any.whl
Installing collected packages: requests
Successfully installed requests-2.13.0

After that I successfully terminated setup.py and confirmed that collectd worked.

Do you know why setup.py uses python 2.6? Please tell me if you knew.

Thank you for your helpful. Best regards. Yo

[yimuniao]

In setup.py, it just uses the easy_install to install the dependency. I guess python2.6 is your default python interpretor.

[SARUNAR]

I have the same issue, [2019-04-01 20:32:16] [error] python plugin: Error importing module "cloudwatch_writer". [2019-04-01 20:32:16] [error] Unhandled python exception in importing module: ImportError: No module named cloudwatch_writer After, changing the ownership to collectd ls -al /opt/collectd-plugins/ total 56 drw-r--r--. 3 collectd collectd 116 Apr 1 20:28 . drwxr-xr-x. 7 root root 109 Apr 1 20:11 .. drwxrwxrwx. 4 collectd collectd 128 Apr 1 20:28 cloudwatch -rwxrwxrwx. 1 collectd collectd 1120 Apr 1 20:28 cloudwatch_writer.py -rwxrwxrwx. 1 collectd collectd 1611 Apr 1 20:28 cloudwatch_writer.pyc -rwxrwxrwx. 1 collectd collectd 1 Apr 1 20:28 init.py -rwxrwxrwx. 1 collectd collectd 42620 Apr 1 20:28 setup.py I am facing this, error on collectd user If i Change it to root it working as expected!