Resource Usage in dashboard saying something went wrong with version v4.2.2

thenucleophile commented 10 years ago

Resource Usage in dashboard saying something went wrong with version v4.2.2 while ceilometer resource-list is giving output as below

root@controller:~# ceilometer +------------------------------------- | Resource ID +------------------------------------- | 50a9a09e-a71a-477e-816a-3ac8a8a1b055 | 6c794811-73fa-4f4c-bef4-ce5fea20d9fc | 8813ff3d-6b65-45c5-b0ca-f104bab41f7c | 8f81eefb-5f28-4b5c-b283-58911710adcd | 99344452-5fee-4607-acb7-74b257bd948a | 9d16cb95-fd6d-4132-8a5c-ec06ebbb7319 | aff4e933-8e83-4aa0-83c0-c9a559dacdaa | b1206217-cc37-4780-a882-6013b885cc79 | b912ab7f-380a-49dd-abe6-347635660429 | cbc0c113-45e2-427b-8ec4-8ddf3a3c6bf5 | cbf18e9f-33ed-439b-89e7-a8f106fdc537 | cc3f8efa-ed18-49b1-b696-2fa3a58115a8 | d33f8089-b751-456f-9054-98de74560feb | d85e264d-8bef-4679-b17f-65575192f558 | d919a729-90f3-46d7-994d-86371c47b37b | e9e0e8ba-4420-4b05-9759-1c68c626356c | f2f30735-196f-4db8-90d7-9b9eb6eca732 | nova-instance-instance-00000001-fa163eb56aa0 | | nova-instance-instance-00000003-fa163e9a105d | | nova-instance-instance-00000005-fa163e7b328f | | nova-instance-instance-0000000b-fa163efdd281 | | nova-instance-instance-0000001b-fa163e2ed3ee | +------------------------------------- resource-list ---------+--------+----------------------------------+----------------------------------+ | Source | User ID | Project ID | ---------+--------+----------------------------------+----------------------------------+ | | 5aeff724eaa24320b800c1805fb6c438 | 9ab8946b9b7c4e81af03e8b0d359f1cf | | | 5aeff724eaa24320b800c1805fb6c438 | 5b3463337426443fbc85633051c2e972 | | | 5aeff724eaa24320b800c1805fb6c438 | 5b3463337426443fbc85633051c2e972 | | | 5aeff724eaa24320b800c1805fb6c438 | 9ab8946b9b7c4e81af03e8b0d359f1cf | | | None | 5b3463337426443fbc85633051c2e972 | | | None | 5b3463337426443fbc85633051c2e972 | | | None | 9ab8946b9b7c4e81af03e8b0d359f1cf | | | None | 9ab8946b9b7c4e81af03e8b0d359f1cf | | | 5aeff724eaa24320b800c1805fb6c438 | 5b3463337426443fbc85633051c2e972 | | | 5aeff724eaa24320b800c1805fb6c438 | 9ab8946b9b7c4e81af03e8b0d359f1cf | | | 5aeff724eaa24320b800c1805fb6c438 | 9ab8946b9b7c4e81af03e8b0d359f1cf | | | None | 5b3463337426443fbc85633051c2e972 | | | 5aeff724eaa24320b800c1805fb6c438 | 9ab8946b9b7c4e81af03e8b0d359f1cf | | | None | 9ab8946b9b7c4e81af03e8b0d359f1cf | | | None | 9ab8946b9b7c4e81af03e8b0d359f1cf | | | None | 9ab8946b9b7c4e81af03e8b0d359f1cf | | | 5aeff724eaa24320b800c1805fb6c438 | 9ab8946b9b7c4e81af03e8b0d359f1cf | | 5aeff724eaa24320b800c1805fb6c438 | 5b3463337426443fbc85633051c2e972 | | 5aeff724eaa24320b800c1805fb6c438 | 9ab8946b9b7c4e81af03e8b0d359f1cf | | 5aeff724eaa24320b800c1805fb6c438 | 5b3463337426443fbc85633051c2e972 | | 5aeff724eaa24320b800c1805fb6c438 | 9ab8946b9b7c4e81af03e8b0d359f1cf | | 5aeff724eaa24320b800c1805fb6c438 | 9ab8946b9b7c4e81af03e8b0d359f1cf | ---------+--------+----------------------------------+----------------------------------+

root@controller:/var/log/ceilometer# cat ceilometer-api.log 2014-04-25 11:02:48.869 7826 ERROR wsme.api [-] Server-side error: "metaquery not implemented". Detail: Traceback (most recent call last):

File "/usr/lib/python2.7/dist-packages/wsmeext/pecan.py", line 72, in callfunction result = f(self, _args, *_kwargs)

File "/usr/lib/python2.7/dist-packages/ceilometer/api/controllers/v2.py", line 968, in get_all for r in pecan.request.storage_conn.get_resources(**kwargs)]

File "/usr/lib/python2.7/dist-packages/ceilometer/storage/impl_sqlalchemy.py", line 310, in getresources raise NotImplementedError(('metaquery not implemented'))

NotImplementedError: metaquery not implemented

Output of cat ceilometer-agent-compute.log

2014-04-25 11:29:13.811 29155 TRACE ceilometer.openstack.common.rpc.common return self.connection.drain_events(timeout=timeout) 2014-04-25 11:29:13.811 29155 TRACE ceilometer.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 281, in drain_events 2014-04-25 11:29:13.811 29155 TRACE ceilometer.openstack.common.rpc.common return self.transport.drain_events(self.connection, kwargs) 2014-04-25 11:29:13.811 29155 TRACE ceilometer.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 91, in drain_events 2014-04-25 11:29:13.811 29155 TRACE ceilometer.openstack.common.rpc.common return connection.drain_events(kwargs) 2014-04-25 11:29:13.811 29155 TRACE ceilometer.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 266, in drain_events 2014-04-25 11:29:13.811 29155 TRACE ceilometer.openstack.common.rpc.common chanmap, None, timeout=timeout, 2014-04-25 11:29:13.811 29155 TRACE ceilometer.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 328, in _wait_multiple 2014-04-25 11:29:13.811 29155 TRACE ceilometer.openstack.common.rpc.common channel, method_sig, args, content = read_timeout(timeout) 2014-04-25 11:29:13.811 29155 TRACE ceilometer.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 292, in read_timeout 2014-04-25 11:29:13.811 29155 TRACE ceilometer.openstack.common.rpc.common return self.method_reader.read_method() 2014-04-25 11:29:13.811 29155 TRACE ceilometer.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 187, in read_method 2014-04-25 11:29:13.811 29155 TRACE ceilometer.openstack.common.rpc.common raise m 2014-04-25 11:29:13.811 29155 TRACE ceilometer.openstack.common.rpc.common error: [Errno 110] Connection timed out 2014-04-25 11:29:13.811 29155 TRACE ceilometer.openstack.common.rpc.common 2014-04-25 11:29:13.813 29155 INFO ceilometer.openstack.common.rpc.common [-] Reconnecting to AMQP server on 10.20.0.175:5672 2014-04-25 11:29:13.833 29155 INFO ceilometer.openstack.common.rpc.common [-] Connected to AMQP server on 10.20.0.175:5672

Please help!

cloudnull commented 10 years ago

Couple of questions:

what os are you running?
have the packages been upgraded outside of the cookbooks and the installation?
when was the installation done?
Is this an HA setup?
what is the version of the packages you have installed (dpkg -l | grep ceilometer || rpm -qa | grep ceilometer)

Get back to us with all of the information you can about your environment and we'll see what needs to be done to get it back online.

thenucleophile commented 10 years ago

Ubuntu 12.04 upgraded packages after cookbook installation initially it was 4.2.0 installation was done 1 month back It is not HA setup. One of each chef server, controller node (chef-client) & compute node (chef-client)

Controller Node: root@controller:~# dpkg -l | grep ceilometer || rpm -qa | grep ceilometer ii ceilometer-agent-central 2013.2.2-0ubuntu1~cloud0 ceilometer central agent rc ceilometer-agent-compute 2013.2.2-0ubuntu1~cloud0 ceilometer compute agent ii ceilometer-api 2013.2.2-0ubuntu1~cloud0 ceilometer api service ii ceilometer-collector 2013.2.2-0ubuntu1~cloud0 ceilometer collector service ii ceilometer-common 2013.2.2-0ubuntu1~cloud0 ceilometer common files ii python-ceilometer 2013.2.2-0ubuntu1~cloud0 ceilometer python libraries ii python-ceilometerclient 1.0.5-0ubuntu1~cloud0 Client library for Openstack ceilometer server.

Compute Node: root@compute:~# dpkg -l | grep ceilometer || rpm -qa | grep ceilometer ii ceilometer-agent-compute 2013.2.2-0ubuntu1~cloud0 ceilometer compute agent ii ceilometer-common 2013.2.2-0ubuntu1~cloud0 ceilometer common files ii python-ceilometer 2013.2.2-0ubuntu1~cloud0 ceilometer python libraries ii python-ceilometerclient 1.0.5-0ubuntu1~cloud0 Client library for Openstack ceilometer server.

It is showing something went wrong in dashboard in resource usage tab

root@controller:/var/log/ceilometer# cat ceilometer-api.log

2014-04-28 11:29:39.235 1207 ERROR wsme.api [-] Server-side error: "metaquery not implemented". Detail: Traceback (most recent call last):

File "/usr/lib/python2.7/dist-packages/wsmeext/pecan.py", line 72, in callfunction result = f(self, _args, *_kwargs)

File "/usr/lib/python2.7/dist-packages/ceilometer/api/controllers/v2.py", line 968, in get_all for r in pecan.request.storage_conn.get_resources(**kwargs)]

File "/usr/lib/python2.7/dist-packages/ceilometer/storage/impl_sqlalchemy.py", line 310, in getresources raise NotImplementedError(('metaquery not implemented'))

NotImplementedError: metaquery not implemented

As per below link https://github.com/rcbops/chef-cookbooks/issues/799

this problem should not come as cookbook version is upgraded to 4.2.2

Please help! Thanks.

thenucleophile commented 10 years ago

i installed ceilometer-agent-compute on controller by mistake..it is already there on compute node...it can be removed as well..but resource usage tab doesn't appear..

Also all services are running fine.

root@controller:/etc/nova# nova-manage service list Binary Host Zone Status State Updated_At nova-conductor controller internal enabled :-) 2014-04-28 12:12:57 nova-cert controller internal enabled :-) 2014-04-28 12:12:58 nova-consoleauth controller internal enabled :-) 2014-04-28 12:12:58 nova-scheduler controller internal enabled :-) 2014-04-28 12:12:58 nova-network compute internal enabled :-) 2014-04-28 12:13:04 nova-compute compute nova enabled :-) 2014-04-28 12:12:08

thenucleophile commented 10 years ago

Do not why but nova-network is going down after sometime after rebooting compute node.

root@controller:/etc# nova-manage service list Binary Host Zone Status State Updated_At nova-conductor controller internal enabled :-) 2014-04-29 07:02:52 nova-cert controller internal enabled :-) 2014-04-29 07:03:01 nova-consoleauth controller internal enabled :-) 2014-04-29 07:03:00 nova-scheduler controller internal enabled :-) 2014-04-29 07:02:58 nova-network compute internal enabled XXX 2014-04-29 06:25:37 nova-compute compute nova enabled :-) 2014-04-29 07:02:52

breu commented 10 years ago

Ubuntu and RedHat packages were updated that caused our hand patch of ceilometer to fail.

RedHat
openstack-ceilometer-collector-2013.2.3-1.el6.noarch
python-ceilometerclient-1.0.8-1.el6.noarch
python-ceilometer-2013.2.3-1.el6.noarch
openstack-ceilometer-api-2013.2.3-1.el6.noarch
openstack-ceilometer-common-2013.2.3-1.el6.noarch
openstack-ceilometer-central-2013.2.3-1.el6.noarch

Ubuntu
ii ceilometer-agent-central 2013.2.2-0ubuntu1~cloud0 ceilometer central agent
rc ceilometer-agent-compute 2013.2.2-0ubuntu1~cloud0 ceilometer compute agent
ii ceilometer-api 2013.2.2-0ubuntu1~cloud0 ceilometer api service
ii ceilometer-collector 2013.2.2-0ubuntu1~cloud0 ceilometer collector service
ii ceilometer-common 2013.2.2-0ubuntu1~cloud0 ceilometer common files
ii python-ceilometer 2013.2.2-0ubuntu1~cloud0 ceilometer python libraries
ii python-ceilometerclient 1.0.5-0ubuntu1~cloud0 Client library for Openstack ceilometer server.

We will need to patch the patcher again..

breu commented 10 years ago

@thenucleophile Your nova-network problem is something separate from ceilometer. Check the nova-network log file but don't lump this into the same github issue.

thenucleophile commented 10 years ago

@rackerjoe Thanks!! Please update once the patcher is patched..

sstent commented 10 years ago

I'm seeing this on Rhel 6.5 as well

runningman84 commented 10 years ago

Any news here? This problem still exists in Ubuntu 12.04 LTS

claco commented 10 years ago

@runningman84 Yes! There's a patch merged into master and the v4.2.3rc branches of ceilometer to fix this:

https://github.com/rcbops-cookbooks/ceilometer/pull/48 https://github.com/rcbops-cookbooks/ceilometer/tree/v4.2.3rc

rcbops / chef-cookbooks

Resource Usage in dashboard saying something went wrong with version v4.2.2 #940