search

ceph / ceph-iscsi-cli

NOTICE: moved to https://github.com/ceph/ceph-iscsi
GNU General Public License v3.0
25 stars 25 forks source link

ceph mimic iscsi issue #144

Open fabianlv opened 5 years ago

fabianlv commented 5 years ago

my environment: os:CentOS Linux release 7.6.1810 (Core) kernel:Linux ceph-all 4.19.10 #1 SMP Mon Jan 14 21:47:11 CST 2019 x86_64 x86_64 x86_64 GNU/Linux ceph version:ceph -v ceph version 13.2.4 (b10be4d44915a4d78a8e06aa31919e74927b142e) mimic (stable)

rpm packages version: [root@ceph-all log]# rpm -qa|grep rtslib python-rtslib-2.1.67-1.noarch [root@ceph-all log]# rpm -qa|grep targetcli targetcli-2.1.fb47-0.1.20170815.git5bf3517.el7cp.noarch [root@ceph-all log]# rpm -qa|grep targetcli targetcli-2.1.fb47-0.1.20170815.git5bf3517.el7cp.noarch [root@ceph-all log]# rpm -qa|grep ceph-iscsi ceph-iscsi-config-2.6-2.6.el7.noarch ceph-iscsi-tools-2.1-3.el7cp.noarch ceph-iscsi-cli-2.7-2.7.el7.noarch

ceph -s : [root@ceph-all log]# ceph -s cluster: id: bdd73746-31ff-4155-96d5-a2b7e1d933b2 health: HEALTH_WARN 2/3 objects misplaced (66.667%) Reduced data availability: 160 pgs inactive Degraded data redundancy: 243 pgs undersized

services: mon: 1 daemons, quorum ceph-all mgr: mgr1(active) osd: 6 osds: 6 up, 6 in; 96 remapped pgs

data: pools: 1 pools, 256 pgs objects: 1 objects, 167 B usage: 107 MiB used, 5.9 GiB / 6.0 GiB avail pgs: 62.500% pgs not active 2/3 objects misplaced (66.667%) 160 undersized+peered 83 active+undersized+remapped 13 active+clean+remapped

[root@ceph-all log]# ceph osd lspools 5 rbd [root@ceph-all log]# ceph osd pool stats pool rbd id 5 2/3 objects misplaced (66.667%)

my problem: rbd-target-api service can not start

log: Jan 15 00:17:03 ceph-all polkitd[9498]: Registered Authentication Agent for unix-process:13912:126693 (system bus name :1.64 [/usr/bin/pkttyagent --notify-fd 5 --fallback], object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8) Jan 15 00:17:03 ceph-all systemd[1]: Started Ceph iscsi target configuration API. Jan 15 00:17:03 ceph-all polkitd[9498]: Unregistered Authentication Agent for unix-process:13912:126693 (system bus name :1.64, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8) (disconnected from bus) Jan 15 00:17:03 ceph-all rbd-target-api[13918]: Started the configuration object watcher Jan 15 00:17:03 ceph-all rbd-target-api[13918]: Checking for config object changes every 1s Jan 15 00:17:03 ceph-all rbd-target-api[13918]: Secure API requested but the crt/key files missing/incompatible? Jan 15 00:17:03 ceph-all rbd-target-api[13918]: Unable to start Jan 15 00:17:03 ceph-all systemd[1]: rbd-target-api.service: main process exited, code=exited, status=16/n/a Jan 15 00:17:03 ceph-all systemd[1]: Unit rbd-target-api.service entered failed state. Jan 15 00:17:03 ceph-all systemd[1]: rbd-target-api.service failed. Jan 15 00:17:03 ceph-all systemd[1]: rbd-target-api.service holdoff time over, scheduling restart. Jan 15 00:17:03 ceph-all systemd[1]: Stopped Ceph iscsi target configuration API. Jan 15 00:17:03 ceph-all systemd[1]: Started Ceph iscsi target configuration API. Jan 15 00:17:03 ceph-all rbd-target-api[13961]: Started the configuration object watcher Jan 15 00:17:03 ceph-all rbd-target-api[13961]: Secure API requested but the crt/key files missing/incompatible? Jan 15 00:17:03 ceph-all rbd-target-api[13961]: Unable to start Jan 15 00:17:03 ceph-all rbd-target-api[13961]: Checking for config object changes every 1s Jan 15 00:17:03 ceph-all rbd-target-api[13961]: Exception in thread Thread-1: Jan 15 00:17:03 ceph-all rbd-target-api[13961]: Traceback (most recent call last): Jan 15 00:17:03 ceph-all rbd-target-api[13961]: File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner Jan 15 00:17:03 ceph-all rbd-target-api[13961]: self.run() Jan 15 00:17:03 ceph-all rbd-target-api[13961]: File "/usr/bin/rbd-target-api", line 1816, in run Jan 15 00:17:03 ceph-all rbd-target-api[13961]: cluster = rados.Rados(conffile=settings.config.cephconf) Jan 15 00:17:03 ceph-all rbd-target-api[13961]: AttributeError: 'NoneType' object has no attribute 'Rados' Jan 15 00:17:03 ceph-all systemd[1]: rbd-target-api.service: main process exited, code=exited, status=16/n/a Jan 15 00:17:03 ceph-all systemd[1]: Unit rbd-target-api.service entered failed state. Jan 15 00:17:03 ceph-all systemd[1]: rbd-target-api.service failed. Jan 15 00:17:04 ceph-all systemd[1]: rbd-target-api.service holdoff time over, scheduling restart. Jan 15 00:17:04 ceph-all systemd[1]: Stopped Ceph iscsi target configuration API. Jan 15 00:17:04 ceph-all systemd[1]: Started Ceph iscsi target configuration API. Jan 15 00:17:04 ceph-all rbd-target-api[13990]: Started the configuration object watcher Jan 15 00:17:04 ceph-all rbd-target-api[13990]: Secure API requested but the crt/key files missing/incompatible? Jan 15 00:17:04 ceph-all rbd-target-api[13990]: Checking for config object changes every 1s Jan 15 00:17:04 ceph-all rbd-target-api[13990]: Unable to start Jan 15 00:17:04 ceph-all kernel: ms_dispatch[14025]: segfault at 7f3922695378 ip 00007f39248e06cf sp 00007f391a26b8d0 error 4 in libceph-common.so.0[7f39243ca000+7d9000] Jan 15 00:17:04 ceph-all kernel: Code: 1f 84 00 00 00 00 00 0f 1f 40 00 41 57 41 56 41 55 41 54 4c 8d 6f 08 55 53 48 83 ec 08 48 8b 6f 10 48 85 ed 0f 84 ad 00 00 00 <4c> 8b 36 4d 89 ec 4d 8b 7e e8 eb 21 0f 1f 44 00 00 4c 89 f6 e8 28 Jan 15 00:17:04 ceph-all systemd[1]: rbd-target-api.service: main process exited, code=killed, status=11/SEGV Jan 15 00:17:04 ceph-all systemd[1]: Unit rbd-target-api.service entered failed state. Jan 15 00:17:04 ceph-all systemd[1]: rbd-target-api.service failed. Jan 15 00:17:04 ceph-all systemd[1]: rbd-target-api.service holdoff time over, scheduling restart. Jan 15 00:17:04 ceph-all systemd[1]: Stopped Ceph iscsi target configuration API. Jan 15 00:17:04 ceph-all systemd[1]: start request repeated too quickly for rbd-target-api.service Jan 15 00:17:04 ceph-all systemd[1]: Failed to start Ceph iscsi target configuration API. Jan 15 00:17:04 ceph-all systemd[1]: Unit rbd-target-api.service entered failed state. Jan 15 00:17:04 ceph-all systemd[1]: rbd-target-api.service failed.

gwcli can not execute [root@ceph-all log]# gwcli -d Adding ceph cluster 'ceph' to the UI Fetching ceph osd information Querying ceph for state information Unable to access the configuration object : REST API failure, code : 500 Traceback (most recent call last): File "/usr/bin/gwcli", line 191, in main() File "/usr/bin/gwcli", line 103, in main root_node.refresh() File "/usr/lib/python2.7/site-packages/gwcli/gateway.py", line 87, in refresh raise GatewayError gwcli.utils.GatewayError

can anyone give me advise? thanks.

dillaman commented 5 years ago

It looks like you don't have python-rados installed (properly) given:

Jan 15 00:17:03 ceph-all rbd-target-api[13961]: File "/usr/bin/rbd-target-api", line 1816, in run
Jan 15 00:17:03 ceph-all rbd-target-api[13961]: cluster = rados.Rados(conffile=settings.config.cephconf)
Jan 15 00:17:03 ceph-all rbd-target-api[13961]: AttributeError: 'NoneType' object has no attribute 'Rados'

I would try to re-install librados2, librbd1, python-rados, and python-rbd (and make sure the RPM versions match).

fabianlv commented 5 years ago

It looks like you don't have python-rados installed (properly) given:

Jan 15 00:17:03 ceph-all rbd-target-api[13961]: File "/usr/bin/rbd-target-api", line 1816, in run
Jan 15 00:17:03 ceph-all rbd-target-api[13961]: cluster = rados.Rados(conffile=settings.config.cephconf)
Jan 15 00:17:03 ceph-all rbd-target-api[13961]: AttributeError: 'NoneType' object has no attribute 'Rados'

I would try to re-install librados2, librbd1, python-rados, and python-rbd (and make sure the RPM versions match).

[root@ceph-all ~]# yum list |grep python-rados python-rados.x86_64 2:13.2.4-0.el7 @ceph-mimic-13.2.4 [root@ceph-all ~]# yum list |grep python-rbd python-rbd.x86_64 2:13.2.4-0.el7 @ceph-mimic-13.2.4 [root@ceph-all ~]# yum list |grep librbd1 librbd1.x86_64 2:13.2.4-0.el7 @ceph-mimic-13.2.4 librbd1.i686 1:10.2.5-4.el7 os
librbd1-devel.i686 1:10.2.5-4.el7 os
librbd1-devel.x86_64 1:10.2.5-4.el7 os [root@ceph-all ~]# yum list |grep librados2 librados2.x86_64 2:13.2.4-0.el7 @ceph-mimic-13.2.4 librados2.i686 1:10.2.5-4.el7 os
librados2-devel.i686 1:10.2.5-4.el7 os
librados2-devel.x86_64 1:10.2.5-4.el7 os
[root@ceph-all ~]# yum list |grep rados librados2.x86_64 2:13.2.4-0.el7 @ceph-mimic-13.2.4 libradosstriper1.x86_64 2:13.2.4-0.el7 @ceph-mimic-13.2.4 python-rados.x86_64 2:13.2.4-0.el7 @ceph-mimic-13.2.4 ceph-radosgw.x86_64 2:13.2.2-0.el7 ceph
librados-devel.x86_64 2:13.2.2-0.el7 ceph
librados2.i686 1:10.2.5-4.el7 os
librados2-devel.i686 1:10.2.5-4.el7 os
librados2-devel.x86_64 1:10.2.5-4.el7 os
libradosstriper-devel.x86_64 2:13.2.2-0.el7 ceph
python34-rados.x86_64 2:13.2.2-0.el7 ceph
rados-objclass-devel.x86_64 2:13.2.2-0.el7 ceph

fabianlv commented 5 years ago

i confused how to set trusted_ip_list in iscsi-gateway.cfg if i disabled trusted_ip_list in iscsi-gateway.cfg,i can execute gwcli if i set trusted_ip_list in iscsi-gateway.cfg,i can't execute gwcli my ip environment: 192.168.80.11 ceph1 192.168.80.12 ceph2 192.168.80.13 ceph3 192.168.80.180 centos1810 192.168.10.11 ceph1-iscsi 192.168.10.12 ceph2-iscsi 192.168.10.13 ceph3-iscsi

ceph.conf [global] fsid = 9a3d7e76-8ce1-46e2-940f-083be11712d3 public_network = 192.168.80.0/24 cluster_network = 192.168.10.0/24 mon_initial_members = ceph1,ceph2,ceph3 mon_host = 192.168.80.11,192.168.80.12,192.168.80.13 auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx mon_allow_pool_delete=true

dumped monmap epoch 1 epoch 1 fsid 9a3d7e76-8ce1-46e2-940f-083be11712d3 last_changed 2019-01-15 15:54:48.676260 created 2019-01-15 15:54:48.676260 0: 192.168.80.11:6789/0 mon.ceph1 1: 192.168.80.12:6789/0 mon.ceph2 2: 192.168.80.13:6789/0 mon.ceph3

whatever i can set 80.x/10.x/1.x, i can't execute only disable it i can execute gwcli

dillaman commented 5 years ago

@fabianlv

You appear to have both "python34-rados.x86_64 2:13.2.2-0.el7 ceph" and "python-rados.x86_64 2:13.2.4-0.el7 @ceph-mimic-13.2.4", which are different versions.

As for the trusted_ip_list, without providing more information, I really cannot help. The IP list is just additional IP addresses to allow to access the API, so populating it shouldn't ever restrict access -- it should only expand it.