Cannot initialize driver

[carmelotoner]

I've installed the driver on my netwon cluster but i encounter the error bellow:

2017-09-13 15:51:01.568 57228 ERROR cinder.volume.manager [req-95e9c02f-0190-4225-b0cb-3bb665cae594 - - - - -] Failed to initialize driver. 2017-09-13 15:51:01.568 57228 ERROR cinder.volume.manager Traceback (most recent call last): 2017-09-13 15:51:01.568 57228 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 414, in init_host 2017-09-13 15:51:01.568 57228 ERROR cinder.volume.manager self.driver.do_setup(ctxt) 2017-09-13 15:51:01.568 57228 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/dell_emc/unity/driver.py", line 78, i n do_setup 2017-09-13 15:51:01.568 57228 ERROR cinder.volume.manager self.adapter.do_setup(self, self.configuration) 2017-09-13 15:51:01.568 57228 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/dell_emc/unity/adapter.py", line 78, in do_setup 2017-09-13 15:51:01.568 57228 ERROR cinder.volume.manager if utils.is_before_4_1(sys_version): 2017-09-13 15:51:01.568 57228 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/dell_emc/unity/utils.py", line 291, i n is_before_4_1 2017-09-13 15:51:01.568 57228 ERROR cinder.volume.manager return version.LooseVersion(ver) < version.LooseVersion('4.1') 2017-09-13 15:51:01.568 57228 ERROR cinder.volume.manager File "/usr/lib64/python2.7/distutils/version.py", line 296, in cmp 2017-09-13 15:51:01.568 57228 ERROR cinder.volume.manager return cmp(self.version, other.version) 2017-09-13 15:51:01.568 57228 ERROR cinder.volume.manager AttributeError: LooseVersion instance has no attribute 'version' 2017-09-13 15:51:01.568 57228 ERROR cinder.volume.manager 2017-09-13 15:51:01.660 57228 INFO cinder.volume.manager [req-95e9c02f-0190-4225-b0cb-3bb665cae594 - - - - -] Initializing RPC dependent components of volume driver UnityDriver (00.04.03) 2017-09-13 15:51:01.661 57228 ERROR cinder.utils [req-95e9c02f-0190-4225-b0cb-3bb665cae594 - - - - -] Volume driver UnityDriver not initialized 2017-09-13 15:51:01.663 57228 ERROR cinder.volume.manager [req-95e9c02f-0190-4225-b0cb-3bb665cae594 - - - - -] Cannot complete RPC initialization because driv er isn't initialized properly.

How to resolve it?

Thank you for your good works!!

[denny-zhao]

What's your array? VNXe3200 or Unity? Looks like you are using a Unity driver (per log), but opened an issue for vnxe driver. So not sure.

[carmelotoner]

Hello, I was in the wrong section sorry. I use the VNXe3200 array. My openstack version is Newton, but the latest vnxedriver is for mitaka. Is it compatible or not for newton? But unity is no compatible for VNXe?

[carmelotoner]

I tried to use the vnxe driver even if it's for mitaka :)

i have the error bellow:

2017-09-14 09:14:01.751 57160 ERROR cinder.cmd.volume Traceback (most recent call last): 2017-09-14 09:14:01.751 57160 ERROR cinder.cmd.volume File "/usr/lib/python2.7/site-packages/cinder/cmd/volume.py", line 99, in main 2017-09-14 09:14:01.751 57160 ERROR cinder.cmd.volume cluster=cluster) 2017-09-14 09:14:01.751 57160 ERROR cinder.cmd.volume File "/usr/lib/python2.7/site-packages/cinder/service.py", line 382, in create 2017-09-14 09:14:01.751 57160 ERROR cinder.cmd.volume cluster=cluster) 2017-09-14 09:14:01.751 57160 ERROR cinder.cmd.volume File "/usr/lib/python2.7/site-packages/cinder/service.py", line 202, in init 2017-09-14 09:14:01.751 57160 ERROR cinder.cmd.volume *args, kwargs) 2017-09-14 09:14:01.751 57160 ERROR cinder.cmd.volume File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 242, in init 2017-09-14 09:14:01.751 57160 ERROR cinder.cmd.volume active_backend_id=curr_active_backend_id) 2017-09-14 09:14:01.751 57160 ERROR cinder.cmd.volume File "/usr/lib/python2.7/site-packages/oslo_utils/importutils.py", line 44, in import_object 2017-09-14 09:14:01.751 57160 ERROR cinder.cmd.volume return import_class(import_str)(*args, *kwargs) 2017-09-14 09:14:01.751 57160 ERROR cinder.cmd.volume File "/usr/lib/python2.7/site-packages/oslo_utils/importutils.py", line 30, in import_class 2017-09-14 09:14:01.751 57160 ERROR cinder.cmd.volume import(mod_str) 2017-09-14 09:14:01.751 57160 ERROR cinder.cmd.volume File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/emc/emc_vnxe.py", line 64, in 2017-09-14 09:14:01.751 57160 ERROR cinder.cmd.volume CONF.register_opts(loc_opts) 2017-09-14 09:14:01.751 57160 ERROR cinder.cmd.volume File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 2193, in __inner 2017-09-14 09:14:01.751 57160 ERROR cinder.cmd.volume result = f(self, args, kwargs) 2017-09-14 09:14:01.751 57160 ERROR cinder.cmd.volume File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 2362, in register_opts 2017-09-14 09:14:01.751 57160 ERROR cinder.cmd.volume self.register_opt(opt, group, clear_cache=False) 2017-09-14 09:14:01.751 57160 ERROR cinder.cmd.volume File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 2197, in __inner 2017-09-14 09:14:01.751 57160 ERROR cinder.cmd.volume return f(self, *args, **kwargs) 2017-09-14 09:14:01.751 57160 ERROR cinder.cmd.volume File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 2351, in register_opt 2017-09-14 09:14:01.751 57160 ERROR cinder.cmd.volume if _is_opt_registered(self._opts, opt): 2017-09-14 09:14:01.751 57160 ERROR cinder.cmd.volume File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 662, in _is_opt_registered 2017-09-14 09:14:01.751 57160 ERROR cinder.cmd.volume raise DuplicateOptError(opt.name) 2017-09-14 09:14:01.751 57160 ERROR cinder.cmd.volume DuplicateOptError: duplicate option: storage_protocol 2017-09-14 09:14:01.751 57160 ERROR cinder.cmd.volume

[denny-zhao]

2017-09-14 09:14:01.751 57160 ERROR cinder.cmd.volume DuplicateOptError: duplicate option: storage_protocol

There could be some error in driver configuration. Can you paste your cinder.conf?

[carmelotoner]

hi Denny,

in attached you can find the cinder.conf

cinder.txt

[denny-zhao]

storage_pool_names is missing from [vnxe] section. Can you add and try again?

[carmelotoner]

i added the storage_pool_names but the same error:

2017-09-15 13:32:25.705 97547 ERROR cinder.cmd.volume DuplicateOptError: duplicate option: storage_protocol

[Murray-LIANG]

Maybe the option storage_protocol was moved up to the framework level in Newton instead down in the driver level. We will check that.

[Murray-LIANG]

@carmelotoner We modified the codes and pushed them into a new branch newton. And tested the vnxe driver came up in Newton. Could you have a try? Thanks!

[carmelotoner]

i have a diffrenet result. the error is now like this:

ERROR cinder.cmd.volume URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)>

[Murray-LIANG]

Oops, this issue is same as https://review.openstack.org/#/c/449408/

From python 2.7.9 on, the ssl verification is enabled by default,

There are two solutions:

1. Downgrade the python.
2. Waiting for our fix. This need another couple of days.

[carmelotoner]

Thank for your help Sir. For now i cannot downgrade the python because it will broke my entire cluster :)

I will wait for the fix. Thank you for your hard work.

[Murray-LIANG]

The fix is pushed to the newton branch. I tested it on the version 2.7.14. Please have a try.

[carmelotoner]

No more error. it work really good!!! Thank you very much guys!!

[carmelotoner]

One more thing: when attaching the volume to an instance, it fails with this message:

2017-09-19 11:58:42.925 127055 WARNING cinder.zonemanager.utils [req-87d84d1a-6762-465b-b246-721296d9b859 e8d51bbd50124fd7bfe74ac1671c8b79 9f27d0ccaeaa4bb789da0505463cec27 - default default] Driver didn't return connection info from terminate_connection call.

is it related to the driver?

[Murray-LIANG]

The failure may be not related to the log you pasted.

Could you find any other errors in your cinder log? You can try to find follow entries:

        LOG.debug('ArrangeHostTask.execute %s', self.connector)
        LOG.debug('ExposeLUNTask.execute %(vol)s %(host)s'
                  % {'vol': self.lun_data,
                     'host': host_id})
        LOG.debug('GetConnectionInfoTask.execute %(vol)s %(conn)s %(host)s',
                  {'vol': self.lun_data, 'conn': self.connector,
                   'host': host_id})

[carmelotoner]

do i need to activate debug mode for the cinder services? Because from now i don't have these logs. (api, volume,scheduler)

[Murray-LIANG]

Yes, you should enable DEBUG level.

[carmelotoner]

Okay. I will do this and be back. thanks

[carmelotoner]

I have these log on cinder volume actually: REQ: curl -i -X GET -H "X-emc-rest-client: true" -H "Visibility: Enduser" -H "User-agent: EMC-OpenStack" -H "Accept: application/json" -H "Accept_language: en_US" -H "Content-type: application/json" https://10.249.20.86:443/api/types/hostInitiator/instances?filter=initiatorId%20eq%20%22iqn.1994-05.com.redhat%3Ad9fbf96a3340%22 _http_log_req /usr/lib/python2.7/site-packages/cinder/volume/drivers/emc/emc_vnxe.py:177 2017-09-20 14:45:41.802 35348 DEBUG cinder.volume.drivers.emc.emc_vnxe [req-b80aeb8d-65be-4dfe-bf3a-2c954845b2c5 e8d51bbd50124fd7bfe74ac1671c8b79 9f27d0ccaeaa4bb789da05\nContent-Type: application/json;version=1.0;charset=UTF-8p 2017 11:45:41 GMT RESP BODY: {"@base":"https://10.249.20.86/api/types/hostInitiator/instances?filter=initiatorId eq \"iqn.1994-05.com.redhat:d9fbf96a3340\"&per_page=2000","updated":"2017-09-20T11:45:42.082Z","links":[{"rel":"self","href":"&page=1"}],"entryCount":1,"entries":[{"@base":"https://10.249.20.86/api/instances/hostInitiator","updated":"2017-09-20T11:45:42.082Z","links":[{"rel":"self","href":"/HostInitiator_8"}],"content":{"type":2,"id":"HostInitiator_8","paths":[],"isIgnored":false,"isChapSecretEnabled":false,"parentHost":{"id":"Host_5"},"initiatorId":"iqn.1994-05.com.redhat:d9fbf96a3340","health":{"descriptionIds":["ALRT_INITIATOR_NO_LOGGED_IN_PATH"],"descriptions":["The initiator does not have any logged in initiator paths. Check the connection between the initiator and the storage system."],"resolutionIds":["initiator_fix_connection"],"resolutions":["vxeuni_t_initiator_has_no_logged_in_initiator_path.html"],"value":10},"operationalStatus":[32768]}}]} _http_log_resp /usr/lib/python2.7/site-packages/cinder/volume/drivers/emc/emc_vnxe.py:200 2017-09-20 14:45:41.804 35348 DEBUG cinder.volume.drivers.emc.emc_vnxe [req-b80aeb8d-65be-4dfe-bf3a-2c954845b2c5 e8d51bbd50124fd7bfe74ac1671c8b79 9f27d0ccaeaa4bb789da0505463cec27 - default default] REQ: curl -i -X GET -H "X-emc-rest-client: true" -H "Visibility: Enduser" -H "User-agent: EMC-OpenStack" -H "Accept: application/json" -H "Accept_language: en_US" -H "Content-type: application/json" https://10.249.20.86:443/api/instances/lun/sv_13?fields=id,type,name,currentNode,hostAccess,pool _http_log_req /usr/lib/python2.7/site-packages/cinder/volume/drivers/emc/emc_vnxe.py:177 2017-09-20 14:45:41.987 35348 DEBUG cinder.volume.drivers.emc.emc_vnxe [req-b80aeb8d-65be-4dfe-bf3a-2c954845b2c5 e8d51bbd50124fd7bfe74ac1671c8b79 9f27d0ccaeaa4bb789da05\nContent-Type: application/json;version=1.0;charset=UTF-8p 2017 11:45:42 GMT RESP BODY: {"@base":"https://10.249.20.86/api/instances/lun","updated":"2017-09-20T11:45:42.266Z","links":[{"rel":"self","href":"/sv_13"}],"content":{"type":2,"id":"sv_13","hostAccess":[{"host":{"id":"Host_5"},"accessMask":1}],"pool":{"id":"pool_4"},"currentNode":1,"name":"volume-4428c1a0-93d7-4103-b155-cccd17b6f2f1"}} _http_log_resp /usr/lib/python2.7/site-packages/cinder/volume/drivers/emc/emc_vnxe.py:200 2017-09-20 14:45:41.988 35348 DEBUG cinder.volume.drivers.emc.emc_vnxe [req-b80aeb8d-65be-4dfe-bf3a-2c954845b2c5 e8d51bbd50124fd7bfe74ac1671c8b79 9f27d0ccaeaa4bb789da0505463cec27 - default default] REQ: curl -i -X POST -H "X-emc-rest-client: true" -H "Visibility: Enduser" -H "User-agent: EMC-OpenStack" -H "Accept: application/json" -H "Accept_language: en_US" -H "Content-type: application/json" -d '{"lunParameters": {"hostAccess": [{"host": {"id": "Host_5"}, "accessMask": 0}]}}' https://10.249.20.86:443/api/instances/storageResource/sv_13/action/modifyLun _http_log_req /usr/lib/python2.7/site-packages/cinder/volume/drivers/emc/emc_vnxe.py:177 2017-09-20 14:45:43.606 35348 DEBUG cinder.volume.drivers.emc.emc_vnxe [req-b80aeb8d-65be-4dfe-bf3a-2c954845b2c5 e8d51bbd50124fd7bfe74ac1671c8b79 9f27d0ccaeaa4bb789da05\nContent-Type: text/plain; charset=UTF-8 Date: Wed, 20 Sep 2017 11:45:42 GMT RESP BODY: _http_log_resp /usr/lib/python2.7/site-packages/cinder/volume/drivers/emc/emc_vnxe.py:200 2017-09-20 14:45:43.609 35348 WARNING cinder.zonemanager.utils [req-b80aeb8d-65be-4dfe-bf3a-2c954845b2c5 e8d51bbd50124fd7bfe74ac1671c8b79 9f27d0ccaeaa4bb789da0505463cec27 - default default] Driver didn't return connection info from terminate_connection call. 2017-09-20 14:45:43.610 35348 INFO cinder.volume.manager [req-b80aeb8d-65be-4dfe-bf3a-2c954845b2c5 e8d51bbd50124fd7bfe74ac1671c8b79 9f27d0ccaeaa4bb789da0505463cec27 - default default] Terminate volume connection completed successfully.

maybe this is the trick: "The initiator does not have any logged in initiator paths. Check the connection between the initiator and the storage system" ?

I followed this link for the VNX manual initiator connexion configuration. https://docs.openstack.org/mitaka/config-reference/block-storage/drivers/emc-vnx-driver.html#emc-vnx-conf

[carmelotoner]

volume.zip

[Murray-LIANG]

You could meet the LUNZ issue: https://docs.openstack.org/cinder/latest/configuration/block-storage/drivers/dell-emc-unity-driver.html#solution-for-lunz-device

The solution is:

1. Go to the Unisphere of your VNXe.
2. Create a LUN.
3. Attach the LUN to the host to occupy the LUN 0.
4. Try again from Openstack to attach a volume.

[carmelotoner]

I followed the procedure but no effect. i notice this error also on compute node: 2017-09-22 10:16:59.295 12534 ERROR oslo_messaging.rpc.server libvirtError: internal error: unable to execute QEMU command '__com.redhat_drive_add': Device 'drive-virtio-disk1' could not be initialized

[Murray-LIANG]

We never met this libvert issue. Maybe you could contact with a nova expert for help.

[carmelotoner]

okey, thank you guys!!