search

Tendrl / ceph-integration

Extracts all data from a Ceph cluster for consumption by Tendrl
GNU Lesser General Public License v2.1
3 stars 11 forks source link

cluster creation failed #245

Closed mkudlej closed 7 years ago

mkudlej commented 7 years ago

I've tried to create cluster from UI and it failed. Please see screenshot. I was not able to find ceph-installer job or tendrl job with ID d483fb26-82a8-4e04-9d78-aa6f55624fe3. I cannot find the root of failure.

bug_create

I was able to find by job ID in logs just:

server journal: 2017-05-23 09:18:59.156428+00:00 - node_agent - /usr/lib/python2.7/site-packages/tendrl/commons/flows/create_cluster/__init__.py:75 - run - INFO - 21893e4f-3a9b-45dc-b7b8-23db3c33e9f5:SSH setup completed for all nodes in cluster d483fb26-82a8-4e04-9d78-aa6f55624fe3
server journal: 2017-05-23 09:18:59.173822+00:00 - node_agent - /usr/lib/python2.7/site-packages/tendrl/commons/flows/create_cluster/__init__.py:95 - run - INFO - 21893e4f-3a9b-45dc-b7b8-23db3c33e9f5:Starting SDS install and config d483fb26-82a8-4e04-9d78-aa6f55624fe3
server journal: 2017-05-23 09:18:59.206823+00:00 - node_agent - /usr/lib/python2.7/site-packages/tendrl/commons/flows/create_cluster/__init__.py:108 - run - INFO - 21893e4f-3a9b-45dc-b7b8-23db3c33e9f5:Creating Ceph Storage Cluster d483fb26-82a8-4e04-9d78-aa6f55624fe3
server journal: 2017-05-23 09:18:59.223544+00:00 - node_agent - /usr/lib/python2.7/site-packages/tendrl/commons/flows/create_cluster/ceph_help.py:19 - create_ceph - INFO - 21893e4f-3a9b-45dc-b7b8-23db3c33e9f5:Installing Ceph Packages d483fb26-82a8-4e04-9d78-aa6f55624fe3
server journal: 2017-05-23 09:29:45.680007+00:00 - node_agent - /usr/lib/python2.7/site-packages/tendrl/commons/flows/create_cluster/ceph_help.py:32 - create_ceph - INFO - 21893e4f-3a9b-45dc-b7b8-23db3c33e9f5:Creating Ceph Monitors d483fb26-82a8-4e04-9d78-aa6f55624fe3
server tendrl-node-agent: 2017-05-23 09:18:59.156428+00:00 - node_agent - /usr/lib/python2.7/site-packages/tendrl/commons/flows/create_cluster/__init__.py:75 - run - INFO - 21893e4f-3a9b-45dc-b7b8-23db3c33e9f5:SSH setup completed for all nodes in cluster d483fb26-82a8-4e04-9d78-aa6f55624fe3
server tendrl-node-agent: 2017-05-23 09:18:59.173822+00:00 - node_agent - /usr/lib/python2.7/site-packages/tendrl/commons/flows/create_cluster/__init__.py:95 - run - INFO - 21893e4f-3a9b-45dc-b7b8-23db3c33e9f5:Starting SDS install and config d483fb26-82a8-4e04-9d78-aa6f55624fe3
server tendrl-node-agent: 2017-05-23 09:18:59.206823+00:00 - node_agent - /usr/lib/python2.7/site-packages/tendrl/commons/flows/create_cluster/__init__.py:108 - run - INFO - 21893e4f-3a9b-45dc-b7b8-23db3c33e9f5:Creating Ceph Storage Cluster d483fb26-82a8-4e04-9d78-aa6f55624fe3
server tendrl-node-agent: 2017-05-23 09:18:59.223544+00:00 - node_agent - /usr/lib/python2.7/site-packages/tendrl/commons/flows/create_cluster/ceph_help.py:19 - create_ceph - INFO - 21893e4f-3a9b-45dc-b7b8-23db3c33e9f5:Installing Ceph Packages d483fb26-82a8-4e04-9d78-aa6f55624fe3
server tendrl-node-agent: 2017-05-23 09:29:45.680007+00:00 - node_agent - /usr/lib/python2.7/site-packages/tendrl/commons/flows/create_cluster/ceph_help.py:32 - create_ceph - INFO - 21893e4f-3a9b-45dc-b7b8-23db3c33e9f5:Creating Ceph Monitors d483fb26-82a8-4e04-9d78-aa6f55624fe3

But something is there:

May 23 09:29:45 mkudlej-usm2-server journal: 2017-05-23 09:29:45.680007+00:00 - node_agent - /usr/lib/python2.7/site-packages/tendrl/commons/flows/create_cluster/ceph_help.py:32 - create_ceph - INFO - 21893e4f-3a9b-45dc-b7b8-23db3c33e9f5:Creating Ceph Monitors d483fb26-82a8-4e04-9d78-aa6f55624fe3
journal: Resetting dropped connection: localhost
ceph-installer-gunicorn: 2017-05-23 09:29:45,703 ERROR [ceph_installer.controllers.errors.schema][MainThread] -> fsid -> None did not pass validation against callable: string (not of type string)
journal: 2017-05-23 09:29:45.712454+00:00 - node_agent - /usr/lib/python2.7/site-packages/tendrl/commons/jobs/__init__.py:225 - _run - ERROR - Failure in Job 21893e4f-3a9b-45dc-b7b8-23db3c33e9f5 Flow 2f94a48a-05d7-408c-b400-e27827f4eacd with error: ceph-installer task configure_mon {mon: 10.70.16.148, fsid: None} failed, response:  - Exception: [{u'function': u'run', u'line': 327, u'file': u'/usr/lib64/python2.7/site-packages/gevent/greenlet.py', u'statement': u'result = self._run(*self.args, **self.kwargs)'}, {u'function': u'_run', u'line': 198, u'file': u'/usr/lib/python2.7/site-packages/tendrl/commons/jobs/__init__.py', u'statement': u'the_flow.run()'}, {u'function': u'run', u'line': 114, u'file': u'/usr/lib/python2.7/site-packages/tendrl/commons/flows/create_cluster/__init__.py', u'statement': u'ceph_help.create_ceph(self.parameters)'}, {u'function': u'create_ceph', u'line': 37, u'file': u'/usr/lib/python2.7/site-packages/tendrl/commons/flows/create_cluster/ceph_help.py', u'statement': u'created_mons = create_mons(parameters, mon_ips)'}, {u'function': u'create_mons', u'line': 110, u'file': u'/usr/lib/python2.7/site-packages/tendrl/commons/flows/create_cluster/ceph_help.py', u'statement': u'created_mons'}, {u'function': u'configure_mon', u'line': 176, u'file': u'/usr/lib/python2.7/site-packages/tendrl/node_agent/provisioner/ceph/plugins/ceph_installer.py', u'statement': u'raise Exception(_msg)'}]
journal: 2017-05-23 09:29:46.297194+00:00 - node_agent - /usr/lib/python2.7/site-packages/tendrl/commons/jobs/__init__.py:235 - _run - ERROR - 21893e4f-3a9b-45dc-b7b8-23db3c33e9f5:Job failed ceph-installer task configure_mon {mon: 10.70.16.148, fsid: None} failed, response: : ceph-installer task configure_mon {mon: 10.70.16.148, fsid: None} failed, response:
mkudlej commented 7 years ago

Packages:

centos-release-ceph-jewel-1.0-1.el7.centos.noarch
ceph-ansible-2.2.6-1.el7scon.noarch
ceph-base-11.2.0-0.el7.x86_64
ceph-common-11.2.0-0.el7.x86_64
ceph-installer-1.3.0-1.el7scon.noarch
ceph-mon-11.2.0-0.el7.x86_64
ceph-osd-11.2.0-0.el7.x86_64
ceph-selinux-11.2.0-0.el7.x86_64
etcd-3.1.3-1.el7.x86_64
libcephfs2-11.2.0-0.el7.x86_64
python-cephfs-11.2.0-0.el7.x86_64
python-etcd-0.4.5-1.noarch
rubygem-etcd-0.3.0-1.el7.centos.noarch
tendrl-api-1.3.0-05_22_2017_01_51_04.noarch
tendrl-api-doc-1.3.0-05_22_2017_01_51_04.noarch
tendrl-api-httpd-1.3.0-05_22_2017_01_51_04.noarch
tendrl-commons-1.3.0-05_22_2017_20_33_47.noarch
tendrl-dashboard-1.3.0-05_22_2017_19_15_06.noarch
tendrl-node-agent-1.3.0-05_22_2017_19_20_55.noarch
tendrl-node-monitoring-1.3.0-05_22_2017_04_25_03.noarch
tendrl-performance-monitoring-1.3.0-05_22_2017_03_02_07.noarch
nthomas-redhat commented 7 years ago

@mkudlej , This issue(fsid->None) is already fixed with latest builds. please test with latest builds and update here.