william-gr
commented
3 years ago Are you using the master version of this repo?
Open marc5feb opened 3 years ago
william-gr
commented
3 years ago Are you using the master version of this repo?
marc5feb
commented
3 years ago Hy,
The one downloaded and installed with
git clone --depth=1 https://github.com/iXsystems/cinder cp -R ./cinder/driver/ixsystems/ /usr/lib/python3/dist-packages/cinder/volume/drivers/
Greetings,
Marc
Are you using the master version of this repo?
william-gr
commented
3 years ago Doesn't ring a bell, I have not been using it myself. Looks like the API reply was empty or invalid.
Would need happen from somebody to debug it
dmcgrandle
commented
3 years ago Looks like the error is happening when attempting to get volume stats. William, if you'll recall when I updated this driver to use the new v2 API we had an issue where the volume stats were only available using the older 1.0 API, so that one portion of this driver still relies on the older API. Is it possible that with truenas 12 support for the older API has been taken out and so that call is failing? I haven't installed 12 yet myself for testing ...
william-gr
commented
3 years ago Ah yes. 12.0 removes support for v1.0 API
marc5feb
commented
3 years ago thanks for the confirmation. will there be a new version which adresses this issue?
dmcgrandle
commented
3 years ago I am guessing that will have to wait until I have time to tackle it in the new year. I should have time in January to install a new test environment with 12 and update the code. If anyone else wants to tackle this in the meantime I have some documentation I generated for the 1.0 -> 2.0 upgrade I can share that will help.
adasauce
commented
3 years ago @dmcgrandle is would be swell if you could share the 1.0 -> 2.0 upgrade notes.
adasauce
commented
3 years ago this commit https://github.com/shosti/cinder/commit/b4aabdfce05b67322bd105e3c36e3f4e3d983572 by @shosti appears to enable the 2.0 API for volume stats, and as well has another commit in their repo to handle API keys instead of using root username/password.
shosti
commented
3 years ago Oh yeah I think I got it working with API v2 on https://github.com/shosti/cinder/tree/api-key . Never got it up and running to the extent that I could verify that it works end-to-end which is why I didn’t open a PR, but I’m happy opening a PR if someone wants to test it.
adasauce
commented
3 years ago I'm in the process of testing out your branch right now, I'll let you know how the integration goes -- at least as an additional data point for whether or not it works as expected.
adasauce
commented
3 years ago @shosti at the most basic level, it's working 100% so far. stats work, volume create/delete works.
I'll be running it through it's paces a little more over the next few days.
dmcgrandle
commented
3 years ago Good work, and this is very interesting. A couple of comments:
_update_volume_stats() and another for the use of API keys.With these points in mind, the updated code for update_volume_stats() would look like this:
def _update_volume_stats(self):
"""Retrieve stats info from volume group
REST API: $ GET /pools/mypool "size":95,"allocated":85,
"""
request_urn = ('%s/id/%s') % (FreeNASServer.REST_API_VOLUME,
urllib.parse.quote_plus(self.configuration.ixsystems_dataset_path))
LOG.debug('_update_volume_stats request_urn : %s', request_urn)
ret = self.handle.invoke_command(FreeNASServer.SELECT_COMMAND,
request_urn, None)
LOG.debug("_update_volume_stats response : %s", json.dumps(ret))
data = {}
data["volume_backend_name"] = self.backend_name
data["vendor_name"] = self.vendor_name
data["driver_version"] = self.VERSION
data["storage_protocol"] = self.storage_protocol
data['total_capacity_gb'] = ix_utils.get_size_in_gb(json.loads(ret['response'])['available']['parsed'] \
+ json.loads(ret['response'])['used']['parsed'])
data['free_capacity_gb'] = ix_utils.get_size_in_gb(json.loads(ret['response'])['available']['parsed'])
data['reserved_percentage'] = \
self.configuration.ixsystems_reserved_percentage
data['reserved_percentage'] = 0
data['QoS_support'] = False
self.stats = data
return self.statsI think it’s best if someone forks my code and opens up a new PR—I’m not really comfortable opening up a PR that I have no way of testing.
marc5feb
commented
3 years ago if needed i can build a test setup later to test things out.
adasauce
commented
3 years ago We've been using this since January and have only come across one issue where if something gets wonky, volumes can't be deleted through cinder if they're in error state but never made it to the backend.
Feel free to include it if this ever goes forward in a new PR.
def _delete_volume(self, name):
"""Deletes specified volume
"""
request_urn = ('%s/id/%s%s') % (FreeNASServer.REST_API_VOLUME,
urllib.parse.quote_plus(self.configuration.ixsystems_dataset_path + '/'), name)
LOG.debug('_delete_volume urn : %s', request_urn)
+
+ check_ret = self.handle.invoke_command(
+ FreeNASServer.SELECT_COMMAND,
+ request_urn, None)
+
+ if check_ret['status'] == FreeNASServer.STATUS_ERROR \
+ and '404:Not Found' in check_ret['response']:
+ return None
+
clone = self._dependent_clone(name) # add check for dependent clone, if exists will delete
ret = self.handle.invoke_command(FreeNASServer.DELETE_COMMAND,
xeonzen
commented
3 years ago Hello,
has there been any movement on this issue? I am using TrueNAS Core 12 with Kolla-ansible Openstack, Ussuri build. I tried using https://github.com/shosti/cinder/tree/api-key fork but get the error below in my cinder-volumes.log file. Any insight or direction would be appriciated=)
2021-04-21 13:27:43.361 210 INFO cinder.volume.manager [req-90ccc720-3611-4263-855b-d3574087b76a - - - - -] Starting volume driver FreeNASISCSIDriver (2.0.0) 2021-04-21 13:27:43.362 210 INFO cinder.volume.drivers.ixsystems.iscsi [req-90ccc720-3611-4263-855b-d3574087b76a - - - - -] iXsystems Do Setup 2021-04-21 13:27:43.362 210 INFO cinder.volume.drivers.ixsystems.iscsi [req-90ccc720-3611-4263-855b-d3574087b76a - - - - -] iXSystems: Check For Setup Error 2021-04-21 13:27:43.363 210 INFO cinder.volume.drivers.ixsystems.iscsi [req-90ccc720-3611-4263-855b-d3574087b76a - - - - -] iXSystems: Check For Setup Error 2021-04-21 13:27:43.363 210 INFO cinder.volume.drivers.ixsystems.iscsi [req-90ccc720-3611-4263-855b-d3574087b76a - - - - -] iXsystems Get Volume Status 2021-04-21 13:27:43.366 210 ERROR oslo_service.service [req-90ccc720-3611-4263-855b-d3574087b76a - - - - -] Error starting thread.: simplejson.errors.JSONDecodeError: Extra data: line 1 column 4 - line 1 column 17 (char 3 - 16) 2021-04-21 13:27:43.366 210 ERROR oslo_service.service Traceback (most recent call last): 2021-04-21 13:27:43.366 210 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_service/service.py", line 810, in run_service 2021-04-21 13:27:43.366 210 ERROR oslo_service.service service.start() 2021-04-21 13:27:43.366 210 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinder/service.py", line 229, in start 2021-04-21 13:27:43.366 210 ERROR oslo_service.service service_id=Service.service_id) 2021-04-21 13:27:43.366 210 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinder/volume/manager.py", line 443, in init_host 2021-04-21 13:27:43.366 210 ERROR oslo_service.service self._init_host(added_to_cluster, **kwargs) 2021-04-21 13:27:43.366 210 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinder/volume/manager.py", line 477, in _init_host 2021-04-21 13:27:43.366 210 ERROR oslo_service.service self.driver.init_capabilities() 2021-04-21 13:27:43.366 210 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinder/volume/driver.py", line 770, in init_capabilities 2021-04-21 13:27:43.366 210 ERROR oslo_service.service stats = self.get_volume_stats(True) 2021-04-21 13:27:43.366 210 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinder/volume/drivers/ixsystems/iscsi.py", line 199, in get_volume_stats 2021-04-21 13:27:43.366 210 ERROR oslo_service.service self.stats = self.common._update_volume_stats() 2021-04-21 13:27:43.366 210 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.6/site-packages/cinder/volume/drivers/ixsystems/common.py", line 368, in _update_volume_stats 2021-04-21 13:27:43.366 210 ERROR oslo_service.service data['total_capacity_gb'] = ix_utils.get_size_in_gb(json.loads(ret['response'])['avail'] + json.loads(ret['response'])['used']) 2021-04-21 13:27:43.366 210 ERROR oslo_service.service File "/var/lib/kolla/venv/lib64/python3.6/site-packages/simplejson/init.py", line 525, in loads 2021-04-21 13:27:43.366 210 ERROR oslo_service.service return _default_decoder.decode(s) 2021-04-21 13:27:43.366 210 ERROR oslo_service.service File "/var/lib/kolla/venv/lib64/python3.6/site-packages/simplejson/decoder.py", line 373, in decode 2021-04-21 13:27:43.366 210 ERROR oslo_service.service raise JSONDecodeError("Extra data", s, end, len(s)) 2021-04-21 13:27:43.366 210 ERROR oslo_service.service simplejson.errors.JSONDecodeError: Extra data: line 1 column 4 - line 1 column 17 (char 3 - 16) 2021-04-21 13:27:43.366 210 ERROR oslo_service.service 2021-04-21 13:27:43.380 7 INFO oslo_service.service [req-81bf8b35-b9af-4420-b18e-62ecd08f236d - - - - -] Child 210 exited with status 1 2021-04-21 13:27:43.381 7 INFO oslo_service.service [req-81bf8b35-b9af-4420-b18e-62ecd08f236d - - - - -] Forking too fast, sleeping
hemna
commented
3 years ago Hello, I am playing around with trueNas at home in a vm. I am running into this same problem. It seems that the driver is broken at this point.
I am an OpenStack cinder core dev and I am running into the same issue. This is one of the reasons why we encourage projects to try and get their drivers into the cinder tree, which requires a working CI system to ensure the driver actually works for each cinder release.
We have an extensive testing setup to ensure that drivers pass the rigorous testing to ensure that drivers work properly when they are deployed. I would encourage you to join our irc channel and see if it's possible to add the driver to the tree.
We are on irc.freenode.net. Channel #openstack-cinder
hemna
commented
3 years ago The response I'm getting back inside of _update_volume_stats() is straight html for the call to fetch the stats from the API.
Apr 22 07:56:19 devstack-sap-train cinder-volume[16135]: DEBUG cinder.volume.drivers.ixsystems.common [None req-fa6ddf43-b93b-4867-bc31-17a24897de06 None None] _update_volume_stats response : {"status": "ok", "response": "<!doctype html>\n<html lang=\"en\">\n<head>\n <meta charset=\"utf-8\">\n <meta http-equiv=\"Pragma\" content=\"no-cache\">\n <meta http-equiv=\"Expires\" content=\"-1\">\n <meta http-equiv=\"CACHE-CONTROL\" content=\"NO-CACHE\">\n <!--<meta http-equiv=\"X-UA-Compatible\" content=\"IE=edge\" />-->\n <!--<script src=\"generated/ie-polyfills.min.js\"></script>-->\n <script nomodule=\"\" src=\"assets/scripts/ie-support/ie-polyfills.min.js\"></script>\n <title id=\"main-page-title\"></title>\n <base href=\"/ui/\"> <meta name=\"viewport\" content=\"width=device-width, initial-scale=1\">\n <link rel=\"apple-touch-icon\" sizes=\"57x57\" href=\"assets/images/apple-icon-57x57.png\">\n <link rel=\"apple-touch-icon\" sizes=\"60x60\" href=\"assets/images/apple-icon-60x60.png\">\n <link rel=\"apple-touch-icon\" sizes=\"72x72\" href=\"assets/images/apple-icon-72x72.png\">\n <link rel=\"apple-touch-icon\" sizes=\"76x76\" href=\"assets/images/apple-icon-76x76.png\">\n <link rel=\"apple-touch-icon\" sizes=\"114x114\" href=\"assets/images/apple-icon-114x114.png\">\n <link rel=\"apple-touch-icon\" sizes=\"120x120\" href=\"assets/images/apple-icon-120x120.png\">\n <link rel=\"apple-touch-icon\" sizes=\"144x144\" href=\"assets/images/apple-icon-144x144.png\">\n <link rel=\"apple-touch-icon\" sizes=\"152x152\" href=\"assets/images/apple-icon-152x152.png\">\n <link rel=\"apple-touch-icon\" sizes=\"180x180\" href=\"assets/images/apple-icon-180x180.png\">\n <link rel=\"icon\" type=\"image/png\" sizes=\"192x192\" href=\"assets/images/android-icon-192x192.png\">\n <!-- <link rel=\"icon\" type=\"image/png\" sizes=\"32x32\" href=\"assets/images/favicon-32x32.png\">\n <link rel=\"icon\" type=\"image/png\" sizes=\"96x96\" href=\"assets/images/favicon-96x96.png\"> -->\n <!-- <link id='fav16' rel=\"icon\" type=\"image/png\" sizes=\"16x16\" href=\"assets/images/favicon-16x16.png\"> -->\n <link rel=\"manifest\" href=\"assets/manifest.json\">\n <meta name=\"msapplication-TileColor\" content=\"#ffffff\">\n <meta name=\"msapplication-TileImage\" content=\"assets/images/ms-icon-144x144.png\">\n <meta name=\"theme-color\" content=\"#ffffff\">\n <link href=\"assets/iconfont/material-icons.css\" rel=\"stylesheet\" />\n <link href=\"assets/iconfont/mdi/css/materialdesignicons.min.css\" rel=\"stylesheet\" />\n <link href=\"assets/iconfont/primeicons/primeicons.css\" rel=\"stylesheet\" />\n<link rel=\"stylesheet\" href=\"styles.de29a0a0a1c752d9b55c.css\"></head>\n<body class=\"ix-blue\">\n <app-root>\n <div class=\"app-loadr\" style=\"background-color:var(--bg2);height:100vh; --bg1:#171E26;--bg2:#232d35;--fg1:#aaaaaa;--fg2:#cccccc;\">\n <div style=\"position:relative;z-index:2;width:180px;height:180px;top:calc(50% - 90px);left:calc(50% - 90px);\">\n <div style=\"text-align:center; position:relative; top:calc(50% - 75px);\">\n <mat-icon svgIcon=\"truenas_core_logomark\" id=\"brandLogo\" alt=\"\" style=\"width: 100%; height:100%\"></mat-icon>\n <!-- Put a loader in here at some point -->\n </div>\n </div>\n <script src='assets/scripts/product.js'>\n </script>\n <script>\n if (product.trim() === 'TrueNAS') {\n document.getElementById('brandLogo').src = 'assets/images/TrueNAS_Logomark_White.svg';\n }\n </script>\n </app-root>\n<script src=\"runtime.a0f2fab97f979fb9b248.js\" defer></script><script src=\"polyfills-es5.0943c14763a8c44f3563.js\" nomodule defer></script><script src=\"polyfills.8e04786abe2dac2d705d.js\" defer></script><script src=\"scripts.4767b83c9e7380540df0.js\" defer></script><script src=\"main.e1a01bb9dedf2ee03d70.js\" defer></script></body>\n</html>\n"} {{(pid=16737) _update_volume_stats /opt/stack/cinder/cinder/volume/drivers/ixsystems/common.py:362}}
Apr 22 07:56:19 devstack-sap-train cinder-volume[16135]: ERROR oslo_service.service [None req-fa6ddf43-b93b-4867-bc31-17a24897de06 None None] Error starting thread.: simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Apr 22 07:56:19 devstack-sap-train cinder-volume[16135]: ERROR oslo_service.service Traceback (most recent call last):
Apr 22 07:56:19 devstack-sap-train cinder-volume[16135]: ERROR oslo_service.service File "/usr/local/lib/python3.6/dist-packages/oslo_service/service.py", line 810, in run_service
Apr 22 07:56:19 devstack-sap-train cinder-volume[16135]: ERROR oslo_service.service service.start()
Apr 22 07:56:19 devstack-sap-train cinder-volume[16135]: ERROR oslo_service.service File "/opt/stack/cinder/cinder/service.py", line 229, in start
Apr 22 07:56:19 devstack-sap-train cinder-volume[16135]: ERROR oslo_service.service service_id=Service.service_id)
Apr 22 07:56:19 devstack-sap-train cinder-volume[16135]: ERROR oslo_service.service File "/usr/local/lib/python3.6/dist-packages/osprofiler/profiler.py", line 160, in wrapper
Apr 22 07:56:19 devstack-sap-train cinder-volume[16135]: ERROR oslo_service.service result = f(*args, **kwargs)
Apr 22 07:56:19 devstack-sap-train cinder-volume[16135]: ERROR oslo_service.service File "/usr/local/lib/python3.6/dist-packages/osprofiler/profiler.py", line 160, in wrapper
Apr 22 07:56:19 devstack-sap-train cinder-volume[16135]: ERROR oslo_service.service result = f(*args, **kwargs)
Apr 22 07:56:19 devstack-sap-train cinder-volume[16135]: ERROR oslo_service.service File "/opt/stack/cinder/cinder/volume/manager.py", line 449, in init_host
Apr 22 07:56:19 devstack-sap-train cinder-volume[16135]: ERROR oslo_service.service self._init_host(added_to_cluster, **kwargs)
Apr 22 07:56:19 devstack-sap-train cinder-volume[16135]: ERROR oslo_service.service File "/opt/stack/cinder/cinder/volume/manager.py", line 483, in _init_host
Apr 22 07:56:19 devstack-sap-train cinder-volume[16135]: ERROR oslo_service.service self.driver.init_capabilities()
Apr 22 07:56:19 devstack-sap-train cinder-volume[16135]: ERROR oslo_service.service File "/usr/local/lib/python3.6/dist-packages/osprofiler/profiler.py", line 160, in wrapper
Apr 22 07:56:19 devstack-sap-train cinder-volume[16135]: ERROR oslo_service.service result = f(*args, **kwargs)
Apr 22 07:56:19 devstack-sap-train cinder-volume[16135]: ERROR oslo_service.service File "/opt/stack/cinder/cinder/volume/driver.py", line 761, in init_capabilities
Apr 22 07:56:19 devstack-sap-train cinder-volume[16135]: ERROR oslo_service.service stats = self.get_volume_stats(True)
Apr 22 07:56:19 devstack-sap-train cinder-volume[16135]: ERROR oslo_service.service File "/usr/local/lib/python3.6/dist-packages/osprofiler/profiler.py", line 160, in wrapper
Apr 22 07:56:19 devstack-sap-train cinder-volume[16135]: ERROR oslo_service.service result = f(*args, **kwargs)
Apr 22 07:56:19 devstack-sap-train cinder-volume[16135]: ERROR oslo_service.service File "/opt/stack/cinder/cinder/volume/drivers/ixsystems/iscsi.py", line 199, in get_volume_stats
Apr 22 07:56:19 devstack-sap-train cinder-volume[16135]: ERROR oslo_service.service self.stats = self.common._update_volume_stats()
Apr 22 07:56:19 devstack-sap-train cinder-volume[16135]: ERROR oslo_service.service File "/opt/stack/cinder/cinder/volume/drivers/ixsystems/common.py", line 368, in _update_volume_stats
Apr 22 07:56:19 devstack-sap-train cinder-volume[16135]: ERROR oslo_service.service data['total_capacity_gb'] = ix_utils.get_size_in_gb(json.loads(ret['response'])['avail'] + json.loads(ret['response'])['used'])
Apr 22 07:56:19 devstack-sap-train cinder-volume[16135]: ERROR oslo_service.service File "/usr/local/lib/python3.6/dist-packages/simplejson/__init__.py", line 518, in loads
Apr 22 07:56:19 devstack-sap-train cinder-volume[16135]: ERROR oslo_service.service return _default_decoder.decode(s)
Apr 22 07:56:19 devstack-sap-train cinder-volume[16135]: ERROR oslo_service.service File "/usr/local/lib/python3.6/dist-packages/simplejson/decoder.py", line 370, in decode
Apr 22 07:56:19 devstack-sap-train cinder-volume[16135]: ERROR oslo_service.service obj, end = self.raw_decode(s)
Apr 22 07:56:19 devstack-sap-train cinder-volume[16135]: ERROR oslo_service.service File "/usr/local/lib/python3.6/dist-packages/simplejson/decoder.py", line 400, in raw_decode
Apr 22 07:56:19 devstack-sap-train cinder-volume[16135]: ERROR oslo_service.service return self.scan_once(s, idx=_w(s, idx).end())
Apr 22 07:56:19 devstack-sap-train cinder-volume[16135]: ERROR oslo_service.service simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0)@hemna with the patches applied from earlier in the thread the driver has worked for me with very minimal modification in queens, victoria and wallaby.
sounds like it may be a config issue if you're up to date with the latest replies and getting wonky API responses.
hemna
commented
3 years ago I replaced the common._update_volume_stats() from https://github.com/iXsystems/cinder/issues/23#issuecomment-760383665 and that seems to have worked.
I'm noticing inconsistencies with the driver and the cinder manager API expectations, that I've been able to overcome a bit with changes to the cinder.conf driver settings. I'll file a separate issue for that.
But I can't create a volume now.
I have trunas 12.0 in a vm and have create a pool with a dataset in it.
[truenas]
iscsi_helper = tgtadm
vlume_backend_name = truenas
volume_dd_blocksize = 512
volume_driver = cinder.volume.drivers.ixsystems.iscsi.FreeNASISCSIDriver
ixsystems_login = root
ixsystems_password = mypassword
ixsystems_server_hostname = 192.168.0.4
ixsystems_volume_backend_name = truenas
ixsystems_iqn_prefix = iqn.2005-10.org.freenas.ctl
ixsystems_datastore_pool = cinder
ixsystems_dataset_path = cinder/openstack
ixsystems_vendor_name = iXsystems
ixsystems_storage_protocol = iscsiLooking at the c-vol log, it looks like the driver is trying to create an iscsi target during create_volume() time, which is really odd. That shouldn't be happening until initialize_connection() is called. This is incorrect cinder driver behavior.
It looks like the call to common._create_volume() works, and then fails during the incorrect call to common._create_iscsitarget(). This leaves the volume in an error state and unusuable.
│May 04 11:57:38 devstack-sap-train cinder-volume[28980]: ERROR cinder.volume.manager Traceback (most recent call last): │
│May 04 11:57:38 devstack-sap-train cinder-volume[28980]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/ixsystems/freenasapi.py", line 206, in invoke_command │
│May 04 11:57:38 devstack-sap-train cinder-volume[28980]: ERROR cinder.volume.manager response_d = urllib.request.urlopen(request) │
│May 04 11:57:38 devstack-sap-train cinder-volume[28980]: ERROR cinder.volume.manager File "/usr/lib/python3.6/urllib/request.py", line 223, in urlopen │
│May 04 11:57:38 devstack-sap-train cinder-volume[28980]: ERROR cinder.volume.manager return opener.open(url, data, timeout) │
│May 04 11:57:38 devstack-sap-train cinder-volume[28980]: ERROR cinder.volume.manager File "/usr/lib/python3.6/urllib/request.py", line 532, in open │
│May 04 11:57:38 devstack-sap-train cinder-volume[28980]: ERROR cinder.volume.manager response = meth(req, response) │
│May 04 11:57:38 devstack-sap-train cinder-volume[28980]: ERROR cinder.volume.manager File "/usr/lib/python3.6/urllib/request.py", line 642, in http_response │
│May 04 11:57:38 devstack-sap-train cinder-volume[28980]: ERROR cinder.volume.manager 'http', request, response, code, msg, hdrs) │
│May 04 11:57:38 devstack-sap-train cinder-volume[28980]: ERROR cinder.volume.manager File "/usr/lib/python3.6/urllib/request.py", line 570, in error │
│May 04 11:57:38 devstack-sap-train cinder-volume[28980]: ERROR cinder.volume.manager return self._call_chain(*args) │
│May 04 11:57:38 devstack-sap-train cinder-volume[28980]: ERROR cinder.volume.manager File "/usr/lib/python3.6/urllib/request.py", line 504, in _call_chain │
│May 04 11:57:38 devstack-sap-train cinder-volume[28980]: ERROR cinder.volume.manager result = func(*args) │
│May 04 11:57:38 devstack-sap-train cinder-volume[28980]: ERROR cinder.volume.manager File "/usr/lib/python3.6/urllib/request.py", line 650, in http_error_default │
│May 04 11:57:38 devstack-sap-train cinder-volume[28980]: ERROR cinder.volume.manager raise HTTPError(req.full_url, code, msg, hdrs, fp) │
│May 04 11:57:38 devstack-sap-train cinder-volume[28980]: ERROR cinder.volume.manager urllib.error.HTTPError: HTTP Error 422: Unprocessable Entity │
│May 04 11:57:38 devstack-sap-train cinder-volume[28980]: ERROR cinder.volume.manager │
│May 04 11:57:38 devstack-sap-train cinder-volume[28980]: ERROR cinder.volume.manager During handling of the above exception, another exception occurred: │
│May 04 11:57:38 devstack-sap-train cinder-volume[28980]: ERROR cinder.volume.manager │
│May 04 11:57:38 devstack-sap-train cinder-volume[28980]: ERROR cinder.volume.manager Traceback (most recent call last): │
│May 04 11:57:38 devstack-sap-train cinder-volume[28980]: ERROR cinder.volume.manager File "/usr/local/lib/python3.6/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task │
│May 04 11:57:38 devstack-sap-train cinder-volume[28980]: ERROR cinder.volume.manager result = task.execute(**arguments) │
│May 04 11:57:38 devstack-sap-train cinder-volume[28980]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1125, in execute │
│May 04 11:57:38 devstack-sap-train cinder-volume[28980]: ERROR cinder.volume.manager context, volume, **volume_spec) │
│May 04 11:57:38 devstack-sap-train cinder-volume[28980]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1091, in _create_raw_volume │
│May 04 11:57:38 devstack-sap-train cinder-volume[28980]: ERROR cinder.volume.manager exception=ex) │
│May 04 11:57:38 devstack-sap-train cinder-volume[28980]: ERROR cinder.volume.manager File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ │
│May 04 11:57:38 devstack-sap-train cinder-volume[28980]: ERROR cinder.volume.manager self.force_reraise() │
│May 04 11:57:38 devstack-sap-train cinder-volume[28980]: ERROR cinder.volume.manager File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise │
│May 04 11:57:38 devstack-sap-train cinder-volume[28980]: ERROR cinder.volume.manager six.reraise(self.type_, self.value, self.tb) │
│May 04 11:57:38 devstack-sap-train cinder-volume[28980]: ERROR cinder.volume.manager File "/usr/local/lib/python3.6/dist-packages/six.py", line 693, in reraise │
│May 04 11:57:38 devstack-sap-train cinder-volume[28980]: ERROR cinder.volume.manager raise value │
│May 04 11:57:38 devstack-sap-train cinder-volume[28980]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 1083, in _create_raw_volume │
│May 04 11:57:38 devstack-sap-train cinder-volume[28980]: ERROR cinder.volume.manager ret = self.driver.create_volume(volume) │
│May 04 11:57:38 devstack-sap-train cinder-volume[28980]: ERROR cinder.volume.manager File "/usr/local/lib/python3.6/dist-packages/osprofiler/profiler.py", line 160, in wrapper │
│May 04 11:57:38 devstack-sap-train cinder-volume[28980]: ERROR cinder.volume.manager result = f(*args, **kwargs) │
│May 04 11:57:38 devstack-sap-train cinder-volume[28980]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/ixsystems/iscsi.py", line 103, in create_volume │
│May 04 11:57:38 devstack-sap-train cinder-volume[28980]: ERROR cinder.volume.manager freenas_volume['name']) │
│May 04 11:57:38 devstack-sap-train cinder-volume[28980]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/ixsystems/common.py", line 256, in _create_iscsitarget │
│May 04 11:57:38 devstack-sap-train cinder-volume[28980]: ERROR cinder.volume.manager tgt_id = self._create_target(name) │
│May 04 11:57:38 devstack-sap-train cinder-volume[28980]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/ixsystems/common.py", line 150, in _create_target │
│May 04 11:57:38 devstack-sap-train cinder-volume[28980]: ERROR cinder.volume.manager request_urn, jtgt_params) │
│May 04 11:57:38 devstack-sap-train cinder-volume[28980]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/ixsystems/freenasapi.py", line 214, in invoke_command │
│May 04 11:57:38 devstack-sap-train cinder-volume[28980]: ERROR cinder.volume.manager json.loads(e.read().decode('utf8'))['message']) │
│May 04 11:57:38 devstack-sap-train cinder-volume[28980]: ERROR cinder.volume.manager KeyError: 'message'
YiHuangDB
commented
1 year ago This should has been fixed on master branch now.
New implementation for function _update_volume_stats uses v2.0 API has been merged to https://github.com/iXsystems/cinder/blob/master/driver/ixsystems/common.py#L431 This provides driver support for TrueNAS 12/13 and TrueNAS scale 22.0.
Tested on my environments:
TrueNAS-12.0-U8.1 (working) TrueNAS-13.0-U2 (working) TrueNAS-SCALE-22.02.1 (working)
Hy,
i have installed a ussuri openstack with a truenas 12 storage system. The problem i face is that i can't connect with this cinder driver to my storage unit. As a workaround i downgrade to 11.3 update 5 freenas and now it works like a charm. i havent made any changes to my openstack configs for it to work.
this is an output of my cinder volume log. 2020-11-25 12:26:29.193 37975 INFO cinder.volume.manager [req-d55ea4c0-4095-47ad-944a-e8cc4b1cebc8 - - - - -] Starting volume driver FreeNASISCSIDriver (2.0.0) 2020-11-25 12:26:29.193 37975 INFO cinder.volume.drivers.ixsystems.iscsi [req-d55ea4c0-4095-47ad-944a-e8cc4b1cebc8 - - - - -] iXsystems Do Setup 2020-11-25 12:26:29.194 37975 INFO cinder.volume.drivers.ixsystems.iscsi [req-d55ea4c0-4095-47ad-944a-e8cc4b1cebc8 - - - - -] iXSystems: Check For Setup Error 2020-11-25 12:26:29.194 37975 INFO cinder.volume.drivers.ixsystems.iscsi [req-d55ea4c0-4095-47ad-944a-e8cc4b1cebc8 - - - - -] iXSystems: Check For Setup Error 2020-11-25 12:26:29.195 37975 INFO cinder.volume.drivers.ixsystems.iscsi [req-d55ea4c0-4095-47ad-944a-e8cc4b1cebc8 - - - - -] iXsystems Get Volume Status 2020-11-25 12:26:29.199 37975 ERROR oslo_service.service [req-d55ea4c0-4095-47ad-944a-e8cc4b1cebc8 - - - - -] Error starting thread.: simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0) 2020-11-25 12:26:29.199 37975 ERROR oslo_service.service Traceback (most recent call last): 2020-11-25 12:26:29.199 37975 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/oslo_service/service.py", line 810, in run_service 2020-11-25 12:26:29.199 37975 ERROR oslo_service.service service.start() 2020-11-25 12:26:29.199 37975 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/cinder/service.py", line 228, in start 2020-11-25 12:26:29.199 37975 ERROR oslo_service.service self.manager.init_host(added_to_cluster=self.added_to_cluster, 2020-11-25 12:26:29.199 37975 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/cinder/volume/manager.py", line 443, in init_host 2020-11-25 12:26:29.199 37975 ERROR oslo_service.service self._init_host(added_to_cluster, **kwargs) 2020-11-25 12:26:29.199 37975 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/cinder/volume/manager.py", line 477, in _init_host 2020-11-25 12:26:29.199 37975 ERROR oslo_service.service self.driver.init_capabilities() 2020-11-25 12:26:29.199 37975 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/cinder/volume/driver.py", line 770, in init_capabilities 2020-11-25 12:26:29.199 37975 ERROR oslo_service.service stats = self.get_volume_stats(True) 2020-11-25 12:26:29.199 37975 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/cinder/volume/drivers/ixsystems/iscsi.py", line 199, in get_volume_stats 2020-11-25 12:26:29.199 37975 ERROR oslo_service.service self.stats = self.common._update_volume_stats() 2020-11-25 12:26:29.199 37975 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/cinder/volume/drivers/ixsystems/common.py", line 368, in _update_volume_stats 2020-11-25 12:26:29.199 37975 ERROR oslo_service.service data['total_capacity_gb'] = ix_utils.get_size_in_gb(json.loads(ret['response'])['avail'] + json.loads(ret['response'])['used']) 2020-11-25 12:26:29.199 37975 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/simplejson/init.py", line 518, in loads 2020-11-25 12:26:29.199 37975 ERROR oslo_service.service return _default_decoder.decode(s) 2020-11-25 12:26:29.199 37975 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/simplejson/decoder.py", line 370, in decode 2020-11-25 12:26:29.199 37975 ERROR oslo_service.service obj, end = self.raw_decode(s) 2020-11-25 12:26:29.199 37975 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/simplejson/decoder.py", line 400, in raw_decode 2020-11-25 12:26:29.199 37975 ERROR oslo_service.service return self.scan_once(s, idx=_w(s, idx).end()) 2020-11-25 12:26:29.199 37975 ERROR oslo_service.service simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0) 2020-11-25 12:26:29.199 37975 ERROR oslo_service.service 2020-11-25 12:26:29.209 37890 INFO oslo_service.service [req-40937f36-8916-4ef8-a741-45301f10732f - - - - -] Child 37975 exited with status 1 2020-11-25 12:26:29.218 37976 INFO cinder.service [-] Starting cinder-volume node (version 16.2.0)
has someone else tried it and got the same error? i have fully installed my openstack from scratch back then and i got the same error 2 times.
my cinder conf is
[ixsystems-iscsi] iscsi_helper = tgtadm volume_dd_blocksize = 512 volume_driver = cinder.volume.drivers.ixsystems.iscsi.FreeNASISCSIDriver ixsystems_login = someloginname ixsystems_password = somepassword ixsystems_server_hostname = someipadress ixsystems_volume_backend_name = iXsystems_FREENAS_Storage ixsystems_iqn_prefix = iqn.2005-10.org.freenas.ctl ixsystems_datastore_pool = Tank1 ixsystems_dataset_path = Tank1/XS-SH-ISCSI ixsystems_vendor_name = iXsystems ixsystems_storage_protocol = iscsi
volume_backend_name = ixsystems-iscsi