search

LINBIT / linstor-proxmox

Integration pluging bridging LINSTOR to Proxmox VE
31 stars 7 forks source link

Proxmox plugin broken after upgrading to Linstor-server v0.9.9, linstor-client 0.9.4 #17

Closed acidrop closed 5 years ago

acidrop commented 5 years ago

root@pve3:~# dpkg -l | grep linstor ii linstor-client 0.9.4-1 all Linstor client command line tool ii linstor-common 0.9.9-1 all DRBD distributed resource management utility ii linstor-proxmox 3.0.3-1 all DRBD distributed resource management utility ii linstor-satellite 0.9.9-1 all DRBD distributed resource management utility ii python-linstor 0.9.4-1 all Linstor python api library

root@drbd-controller:~# dpkg -l | grep linstor ii linstor-client 0.9.4-1 all Linstor client command line tool ii linstor-common 0.9.9-1 all DRBD distributed resource management uti lity ii linstor-controller 0.9.9-1 all DRBD distributed resource management uti lity ii python-linstor 0.9.4-1 all Linstor python api library

root@pve3:~# pveversion -v proxmox-ve: 5.4-1 (running kernel: 4.15.18-9-pve) pve-manager: 5.4-6 (running version: 5.4-6/aa7856c5) pve-kernel-4.15: 5.2-12 pve-kernel-4.15.18-9-pve: 4.15.18-30 pve-kernel-4.15.18-8-pve: 4.15.18-28 ceph: 12.2.12-pve1 corosync: 2.4.4-pve1 criu: 2.11.1-1~bpo90 glusterfs-client: 3.8.8-1 ksm-control-daemon: 1.2-2 libjs-extjs: 6.0.1-2 libpve-access-control: 5.1-10 libpve-apiclient-perl: 2.0-5 libpve-common-perl: 5.0-52 libpve-guest-common-perl: 2.0-20 libpve-http-server-perl: 2.0-13 libpve-storage-perl: 5.0-43 libqb0: 1.0.3-1~bpo9 lvm2: 2.02.168-pve6 lxc-pve: 3.1.0-3 lxcfs: 3.0.3-pve1 novnc-pve: 1.0.0-3 openvswitch-switch: 2.7.0-3 proxmox-widget-toolkit: 1.0-28 pve-cluster: 5.0-37 pve-container: 2.0-39 pve-docs: 5.4-2 pve-edk2-firmware: 1.20190312-1 pve-firewall: 3.0-21 pve-firmware: 2.0-6 pve-ha-manager: 2.0-9 pve-i18n: 1.1-4 pve-libspice-server1: 0.14.1-2 pve-qemu-kvm: 3.0.1-2 pve-xtermjs: 3.12.0-1 qemu-server: 5.0-51 smartmontools: 6.5+svn4324-1 spiceterm: 3.0-5 vncterm: 1.5-3 zfsutils-linux: 0.7.13-pve1~bpo2

root@pve3:~# qm start 107 Plugin "PVE::Storage::Custom::LINSTORPlugin" is implementing an older storage API, an upgrade is recommended Traceback (most recent call last): File "/usr/bin/linstor", line 24, in linstor_client_main.main() File "/usr/lib/python2.7/dist-packages/linstor_client_main.py", line 627, in main LinStorCLI().run() File "/usr/lib/python2.7/dist-packages/linstor_client_main.py", line 592, in run rc = self.parse_and_execute(sys.argv[1:]) File "/usr/lib/python2.7/dist-packages/linstor_client_main.py", line 307, in parse_and_execute rc = args.func(args) File "/usr/lib/python2.7/dist-packages/linstor_client/commands/rsc_cmds.py", line 485, in list lstmsg = self._linstor.resource_list(filter_by_nodes=args.nodes, filter_by_resources=args.resources) File "/usr/lib/python2.7/dist-packages/linstor/linstorapi.py", line 1264, in resource_list return self.volume_list(filter_by_nodes=filter_by_nodes, filter_by_resources=filter_by_resources) File "/usr/lib/python2.7/dist-packages/linstor/linstorapi.py", line 1291, in volume_list path File "/usr/lib/python2.7/dist-packages/linstor/linstorapi.py", line 198, in _rest_request error_data = json.loads(error_data_raw) File "/usr/lib/python2.7/json/init.py", line 339, in loads return _default_decoder.decode(s) File "/usr/lib/python2.7/json/decoder.py", line 364, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib/python2.7/json/decoder.py", line 382, in raw_decode raise ValueError("No JSON object could be decoded") ValueError: No JSON object could be decoded malformed JSON string, neither tag, array, object, number, string or atom, at character offset 0 (before "(end of string)") at /usr/share/perl5/PVE/Storage/Custom/LINSTORPlugin.pm line 126, <$pipe> chunk 1.

rp- commented 5 years ago

could you paste the output of curl -X GET http://localhost:3370/v1/view/resources the url of course adapted to your controller host.

acidrop commented 5 years ago

report.txt

rp- commented 5 years ago

I would have assumed you would get back a non json output, but this is valid json and what I would expect. I guess also linstor resource list is working? Is it maybe now working?

acidrop commented 5 years ago

Yes, linstor seems to be working fine, it's when trying to start a VM in PVE that it crashes (qm start . Just a note, the HA VM (linstor controller) does not crash. I assume that's because it excluded from linstor, right ? This appears to be a problem in proxmox perl storage plugin and not something related to the linstor itselft. Perhaps a recent change in linstor broke the perl plugin ? I could try to rollback to the previous versions of linstor-server and linstor-client just to verify that this is the case. This is a test environment anyways...

root@pve3:~# linstor r l ╭─────────────────────────────────────────────────╮ ┊ ResourceName ┊ Node ┊ Port ┊ Usage ┊ State ┊ ╞┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄╡ ┊ testvol ┊ pve2 ┊ 7008 ┊ Unused ┊ UpToDate ┊ ┊ testvol ┊ pve3 ┊ 7008 ┊ Unused ┊ UpToDate ┊ ┊ vm-100-disk-1 ┊ pve1 ┊ 7000 ┊ Unused ┊ UpToDate ┊ ┊ vm-100-disk-1 ┊ pve2 ┊ 7000 ┊ Unused ┊ UpToDate ┊ ┊ vm-100-disk-1 ┊ pve3 ┊ 7000 ┊ Unused ┊ UpToDate ┊ ┊ vm-101-disk-1 ┊ pve2 ┊ 7001 ┊ Unused ┊ UpToDate ┊ ┊ vm-101-disk-1 ┊ pve3 ┊ 7001 ┊ Unused ┊ UpToDate ┊ ┊ vm-102-disk-1 ┊ pve1 ┊ 7003 ┊ Unused ┊ UpToDate ┊ ┊ vm-102-disk-1 ┊ pve2 ┊ 7003 ┊ Unused ┊ UpToDate ┊ ┊ vm-102-disk-1 ┊ pve3 ┊ 7003 ┊ Unused ┊ UpToDate ┊ ┊ vm-104-disk-1 ┊ pve2 ┊ 7006 ┊ Unused ┊ UpToDate ┊ ┊ vm-104-disk-1 ┊ pve3 ┊ 7006 ┊ Unused ┊ UpToDate ┊ ┊ vm-105-disk-2 ┊ pve1 ┊ 7002 ┊ Unused ┊ UpToDate ┊ ┊ vm-105-disk-2 ┊ pve2 ┊ 7002 ┊ Unused ┊ UpToDate ┊ ┊ vm-105-disk-2 ┊ pve3 ┊ 7002 ┊ InUse ┊ UpToDate ┊ ┊ vm-106-disk-1 ┊ pve1 ┊ 7005 ┊ Unused ┊ UpToDate ┊ ┊ vm-106-disk-1 ┊ pve2 ┊ 7005 ┊ Unused ┊ UpToDate ┊ ┊ vm-106-disk-1 ┊ pve3 ┊ 7005 ┊ Unused ┊ UpToDate ┊ ┊ vm-107-disk-1 ┊ pve1 ┊ 7004 ┊ Unused ┊ UpToDate ┊ ┊ vm-107-disk-1 ┊ pve2 ┊ 7004 ┊ Unused ┊ UpToDate ┊ ┊ vm-107-disk-1 ┊ pve3 ┊ 7004 ┊ Unused ┊ UpToDate ┊ ┊ vm-108-disk-1 ┊ pve1 ┊ 7007 ┊ Unused ┊ UpToDate ┊ ┊ vm-108-disk-1 ┊ pve2 ┊ 7007 ┊ Unused ┊ UpToDate ┊ ┊ vm-108-disk-1 ┊ pve3 ┊ 7007 ┊ Unused ┊ UpToDate ┊ ╰─────────────────────────────────────────────────╯

rck commented 5 years ago

This is pretty weird, and to me it looks like the problem is in the client. The client fails when called from the plugin, therefore reports back something that is not JSON, therefore the plugin fails.

Your manual "linstor-client" tests talk to the same controller as the plugin does? the host/IP was the same as in storage.cfg?

I pushed a new version of the plugin to master. This one should have better traceback logging. Then we at least know which command exactly failed.

Just override the LINSTORPlugin.pm with the one from current master and paste the trace.

acidrop commented 5 years ago

Yes, all tests are taking place by using the same controller and its ip address is the same as in storage.cfg, no changes there.

Quote from /etc/pve/storage.cfg ...

drbd: drbdstorage content rootdir,images controller 10.10.10.4 controllervm 105 redundancy 3

root@pve1:~# env | grep LS_CONTROLLERS LS_CONTROLLERS=10.10.10.4

Result by using LINSTORPlugin.pm from master ....

root@pve1:/usr/share/perl5/PVE/Storage/Custom# qm start 100
Plugin "PVE::Storage::Custom::LINSTORPlugin" is implementing an older storage API, an upgrade is recommended
Traceback (most recent call last):
  File "/usr/bin/linstor", line 24, in <module>
    linstor_client_main.main()
  File "/usr/lib/python2.7/dist-packages/linstor_client_main.py", line 627, in main
    LinStorCLI().run()
  File "/usr/lib/python2.7/dist-packages/linstor_client_main.py", line 592, in run
    rc = self.parse_and_execute(sys.argv[1:])
  File "/usr/lib/python2.7/dist-packages/linstor_client_main.py", line 307, in parse_and_execute
    rc = args.func(args)
  File "/usr/lib/python2.7/dist-packages/linstor_client/commands/rsc_cmds.py", line 485, in list
    lstmsg = self._linstor.resource_list(filter_by_nodes=args.nodes, filter_by_resources=args.resources)
  File "/usr/lib/python2.7/dist-packages/linstor/linstorapi.py", line 1264, in resource_list
    return self.volume_list(filter_by_nodes=filter_by_nodes, filter_by_resources=filter_by_resources)
  File "/usr/lib/python2.7/dist-packages/linstor/linstorapi.py", line 1291, in volume_list
    path
  File "/usr/lib/python2.7/dist-packages/linstor/linstorapi.py", line 198, in _rest_request
    error_data = json.loads(error_data_raw)
  File "/usr/lib/python2.7/json/__init__.py", line 339, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python2.7/json/decoder.py", line 364, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python2.7/json/decoder.py", line 382, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
malformed JSON string, neither tag, array, object, number, string or atom, at character offset 0 (before "(end of string)") at /usr/share/perl5/PVE/Storage/Custom/LINSTORPlugin.pm line 129, <$pipe> chunk 1.  at /usr/share/perl5/PVE/Storage/Custom/LINSTORPlugin.pm line 131.  PVE::Storage::Custom::LINSTORPlugin::decode_json_from_pipe("/usr/bin/linstor", "--controllers=10.10.10.4", "-m", "resource", "list", "--resources", "vm-100-disk-1", "--nodes", ...) called at /usr/share/perl5/PVE/Storage/Custom/LINSTORPlugin.pm line 139       PVE::Storage::Custom::LINSTORPlugin::drbd_exists_locally(HASH(0x55d7bff9b148), "vm-100-disk-1", "pve1", 0) called at /usr/share/perl5/PVE/Storage/Custom/LINSTORPlugin.pm line 487         PVE::Storage::Custom::LINSTORPlugin::activate_volume("PVE::Storage::Custom::LINSTORPlugin", "drbdstorage", HASH(0x55d7bff9b148), "vm-100-disk-1", undef, HASH(0x55d7bff77df0)) called at /usr/share/perl5/PVE/Storage.pm line 1060         PVE::Storage::activate_volumes(HASH(0x55d7bff88e98), ARRAY(0x55d7bff77b80)) called at /usr/share/perl5/PVE/QemuServer.pm line 5311      PVE::QemuServer::__ANON__() called at /usr/share/perl5/PVE/Tools.pm line 201       eval {...} called at /usr/share/perl5/PVE/Tools.pm line 201     PVE::Tools::lock_file_full("/var/lock/qemu-server/lock-100.conf", 10, 0, CODE(0x55d7bff0d668)) called at /usr/share/perl5/PVE/Tools.pm line 210    PVE::Tools::lock_file("/var/lock/qemu-server/lock-100.conf", 10, CODE(0x55d7bff0d668)) called at /usr/share/perl5/PVE/AbstractConfig.pm line 78   PVE::AbstractConfig::lock_config_full("PVE::QemuConfig", 100, 10, CODE(0x55d7bff0d668)) called at /usr/share/perl5/PVE/AbstractConfig.pm line 117        PVE::AbstractConfig::lock_config("PVE::QemuConfig", 100, CODE(0x55d7bff0d668)) called at /usr/share/perl5/PVE/QemuServer.pm line 5439      PVE::QemuServer::vm_start(HASH(0x55d7bff88e98), 100, undef, undef, undef, undef, undef, undef, ...) called at /usr/share/perl5/PVE/API2/Qemu.pm line 2099  PVE::API2::Qemu::__ANON__("UPID:pve1:00013A77:00102442:5CE7BA60:qmstart:100:root\@pam:") called at /usr/share/perl5/PVE/RESTEnvironment.pm line 610     eval {...} called at /usr/share/perl5/PVE/RESTEnvironment.pm line 601      PVE::RESTEnvironment::fork_worker(PVE::RPCEnvironment=HASH(0x55d7bff2ee50), "qmstart", 100, "root\@pam", CODE(0x55d7bff9c740)) called at /usr/share/perl5/PVE/API2/Qemu.pm line 2105    PVE::API2::Qemu::__ANON__(HASH(0x55d7bff7c1a0)) called at /usr/share/perl5/PVE/RESTHandler.pm line 446     PVE::RESTHandler::handle("PVE::API2::Qemu", HASH(0x55d7bfd0c548), HASH(0x55d7bff7c1a0)) called at /usr/share/perl5/PVE/RESTHandler.pm line 855     eval {...} called at /usr/share/perl5/PVE/RESTHandler.pm line 838       PVE::RESTHandler::cli_handler("PVE::API2::Qemu", "qm start", "vm_start", ARRAY(0x55d7bbe73a00), ARRAY(0x55d7bff9aad0), HASH(0x55d7bff9ab18), CODE(0x55d7bff68cd8), undef) called at /usr/share/perl5/PVE/CLIHandler.pm line 580    PVE::CLIHandler::__ANON__(ARRAY(0x55d7bb19e7c0), undef, CODE(0x55d7bff68cd8)) called at /usr/share/perl5/PVE/CLIHandler.pm line 657        PVE::CLIHandler::run_cli_handler("PVE::CLI::qm") called at /usr/sbin/qm line 8
rck commented 5 years ago

and on the same node: /usr/bin/linstor --controllers=10.10.10.4 -m resource list --resources vm-100-disk-1 --nodes pve1

acidrop commented 5 years ago 
root@pve1:~# /usr/bin/linstor --controllers=10.10.10.4 -m resource list --resources vm-100-disk-1 --nodes pve1
Traceback (most recent call last):
  File "/usr/bin/linstor", line 24, in <module>
    linstor_client_main.main()
  File "/usr/lib/python2.7/dist-packages/linstor_client_main.py", line 627, in main
    LinStorCLI().run()
  File "/usr/lib/python2.7/dist-packages/linstor_client_main.py", line 592, in run
    rc = self.parse_and_execute(sys.argv[1:])
  File "/usr/lib/python2.7/dist-packages/linstor_client_main.py", line 307, in parse_and_execute
    rc = args.func(args)
  File "/usr/lib/python2.7/dist-packages/linstor_client/commands/rsc_cmds.py", line 485, in list
    lstmsg = self._linstor.resource_list(filter_by_nodes=args.nodes, filter_by_resources=args.resources)
  File "/usr/lib/python2.7/dist-packages/linstor/linstorapi.py", line 1264, in resource_list
    return self.volume_list(filter_by_nodes=filter_by_nodes, filter_by_resources=filter_by_resources)
  File "/usr/lib/python2.7/dist-packages/linstor/linstorapi.py", line 1291, in volume_list
    path
  File "/usr/lib/python2.7/dist-packages/linstor/linstorapi.py", line 198, in _rest_request
    error_data = json.loads(error_data_raw)
  File "/usr/lib/python2.7/json/__init__.py", line 339, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python2.7/json/decoder.py", line 364, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python2.7/json/decoder.py", line 382, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
rck commented 5 years ago

Thanks, that confirmed what I expected. @rp- ?

rp- commented 5 years ago

Ok could reproduce now, mixing filters (resource and nodes) doesn't work very well. Working on a fix, thanks for reporting

rp- commented 5 years ago

here is a patch, if you want to continue testing, otherwise we will have a release early next week I guess.

diff --git a/linstor/linstorapi.py b/linstor/linstorapi.py
index f4e21e3..2e7345f 100644
--- a/linstor/linstorapi.py
+++ b/linstor/linstorapi.py
@@ -771,15 +771,15 @@ class Linstor(object):
         :return: A MsgLstStorPool proto message containing all information.
         :rtype: list[RESTMessageResponse]
         """
-        query_params = ""
+        query_params = []
         if filter_by_nodes:
-            query_params += "&".join(["nodes=" + x for x in filter_by_nodes])
+            query_params += ["nodes=" + x for x in filter_by_nodes]
         if filter_by_stor_pools:
-            query_params += "&".join(["storage_pools=" + x for x in filter_by_stor_pools])
+            query_params += ["storage_pools=" + x for x in filter_by_stor_pools]

         path = "/v1/view/storage-pools"
         if query_params:
-            path += "?" + query_params
+            path += "?" + "&".join(query_params)
         storage_pool_res = self._rest_request(
             apiconsts.API_LST_STOR_POOL,
             "GET",
@@ -1275,16 +1275,16 @@ class Linstor(object):
         """
         result = []
         errors = []
-        query_params = ""
+        query_params = []
         if filter_by_nodes:
-            query_params += "&".join(["nodes=" + x for x in filter_by_nodes])
+            query_params += ["nodes=" + x for x in filter_by_nodes]
         if filter_by_stor_pools:
-            query_params += "&".join(["storage_pools=" + x for x in filter_by_stor_pools])
+            query_params += ["storage_pools=" + x for x in filter_by_stor_pools]
         if filter_by_resources:
-            query_params += "&".join(["resources=" + x for x in filter_by_resources])
+            query_params += ["resources=" + x for x in filter_by_resources]
         path = "/v1/view/resources"
         if query_params:
-            path += "?" + query_params
+            path += "?" + "&".join(query_params)
         resource_resp = self._rest_request(
             apiconsts.API_LST_RSC,
             "GET",
acidrop commented 5 years ago

Seems to be working fine after applying the patch, I will do some more testing and report back... thanks!