search

rposudnevskiy / RBDSR

RBDSR - XenServer/XCP-ng Storage Manager plugin for CEPH
GNU Lesser General Public License v2.1
58 stars 23 forks source link

VDI.copy failed #22

Closed zhoubofsy closed 7 years ago

zhoubofsy commented 7 years ago

hi I try to copy vdi , but failed.

command line:

xe vdi-copy sr-uuid=a04f4451-3761-415b-a6c2-2b12656cb13a uuid=6a571d0a-99da-405e-8bfe-76bb1c79c608
The VDI copy action has failed
<extra>: End_of_file

xensource.log say a exception VDI_COPY_FAILED, [End_of_file]. but, SMlog no error.

xensource.log:

Dec 16 16:49:02 xenserver90 xapi: [debug|xenserver90|7401 UNIX /var/lib/xcp/xapi|VDI.db_forget R:badd76e6ee92|xapi] db_forget uuid=159c5306-fb1a-48ed-980b-e517762a5247
Dec 16 16:49:02 xenserver90 xapi: [ info|xenserver90|7366 |sm_exec D:9e52034d6d8d|xapi] Session.destroy trackid=f8b10c5d21747bb7ba9fd1a22b01d90f
Dec 16 16:49:02 xenserver90 xapi: [debug|xenserver90|202 INET :::80|VDI.destroy R:e27331f4fa33|xapi] Unmarking SR after VDI.destroy (task=OpaqueRef:e27331f4-fa33-537e-09b5-4e5d514ee96e)
Dec 16 16:49:02 xenserver90 xapi: [debug|xenserver90|7314 UNIX /var/lib/xcp/xapi|VDI.copy R:dcad2835578f|xmlrpc_client] stunnel pid: 25601 (cached = true) returned stunnel to cache
Dec 16 16:49:02 xenserver90 xapi: [ info|xenserver90|7314 UNIX /var/lib/xcp/xapi|VDI.copy R:dcad2835578f|xapi] Session.destroy trackid=7de92aa3fe91acadf24925dd5dda7b20
Dec 16 16:49:02 xenserver90 xapi: [debug|xenserver90|7314 UNIX /var/lib/xcp/xapi|VDI.copy R:dcad2835578f|taskhelper] the status of R:dcad2835578f is failure; cannot set it to `failure
Dec 16 16:49:02 xenserver90 xapi: [error|xenserver90|7314 UNIX /var/lib/xcp/xapi|dispatch:VDI.copy D:043f2fef673f|backtrace] VDI.copy R:dcad2835578f failed with exception Server_error(VDI_COPY_FAILED, [ End_of_file ])
Dec 16 16:49:02 xenserver90 xapi: [error|xenserver90|7314 UNIX /var/lib/xcp/xapi|dispatch:VDI.copy D:043f2fef673f|backtrace] Raised Server_error(VDI_COPY_FAILED, [ End_of_file ])
Dec 16 16:49:02 xenserver90 xapi: [error|xenserver90|7314 UNIX /var/lib/xcp/xapi|dispatch:VDI.copy D:043f2fef673f|backtrace] 1/14 xapi @ xenserver90 Raised at file client.ml, line 6
Dec 16 16:49:02 xenserver90 xapi: [error|xenserver90|7314 UNIX /var/lib/xcp/xapi|dispatch:VDI.copy D:043f2fef673f|backtrace] 2/14 xapi @ xenserver90 Called from file client.ml, line 18
Dec 16 16:49:02 xenserver90 xapi: [error|xenserver90|7314 UNIX /var/lib/xcp/xapi|dispatch:VDI.copy D:043f2fef673f|backtrace] 3/14 xapi @ xenserver90 Called from file client.ml, line 10623
Dec 16 16:49:02 xenserver90 xapi: [error|xenserver90|7314 UNIX /var/lib/xcp/xapi|dispatch:VDI.copy D:043f2fef673f|backtrace] 4/14 xapi @ xenserver90 Called from file lib/pervasiveext.ml, line 22
Dec 16 16:49:02 xenserver90 xapi: [error|xenserver90|7314 UNIX /var/lib/xcp/xapi|dispatch:VDI.copy D:043f2fef673f|backtrace] 5/14 xapi @ xenserver90 Called from file lib/pervasiveext.ml, line 26
Dec 16 16:49:02 xenserver90 xapi: [error|xenserver90|7314 UNIX /var/lib/xcp/xapi|dispatch:VDI.copy D:043f2fef673f|backtrace] 6/14 xapi @ xenserver90 Called from file message_forwarding.ml, line 106
Dec 16 16:49:02 xenserver90 xapi: [error|xenserver90|7314 UNIX /var/lib/xcp/xapi|dispatch:VDI.copy D:043f2fef673f|backtrace] 7/14 xapi @ xenserver90 Called from file message_forwarding.ml, line 3386
Dec 16 16:49:02 xenserver90 xapi: [error|xenserver90|7314 UNIX /var/lib/xcp/xapi|dispatch:VDI.copy D:043f2fef673f|backtrace] 8/14 xapi @ xenserver90 Called from file lib/pervasiveext.ml, line 22
Dec 16 16:49:02 xenserver90 xapi: [error|xenserver90|7314 UNIX /var/lib/xcp/xapi|dispatch:VDI.copy D:043f2fef673f|backtrace] 9/14 xapi @ xenserver90 Called from file rbac.ml, line 236
Dec 16 16:49:02 xenserver90 xapi: [error|xenserver90|7314 UNIX /var/lib/xcp/xapi|dispatch:VDI.copy D:043f2fef673f|backtrace] 10/14 xapi @ xenserver90 Called from file server_helpers.ml, line 75
Dec 16 16:49:02 xenserver90 xapi: [error|xenserver90|7314 UNIX /var/lib/xcp/xapi||cli] Converting exception VDI_COPY_FAILED: [ End_of_file ] into a CLI response
Dec 16 16:49:02 xenserver90 xapi: [error|xenserver90|7314 UNIX /var/lib/xcp/xapi||backtrace] Raised Server_error(VDI_COPY_FAILED, [ End_of_file ])
Dec 16 16:49:02 xenserver90 xapi: [error|xenserver90|7314 UNIX /var/lib/xcp/xapi||backtrace] 1/1 xapi @ xenserver90 Raised at file (Thread 7314 has no backtrace table. Was with_backtraces called?, line 0
Dec 16 16:49:02 xenserver90 xapi: [error|xenserver90|7314 UNIX /var/lib/xcp/xapi||backtrace]
Dec 16 16:49:03 xenserver90 xapi: [debug|xenserver90|7402 UNIX /var/lib/xcp/xapi||cli] xe sr-list type=rbd username=root password=(omitted)
Dec 16 16:49:03 xenserver90 xapi: [ info|xenserver90|7402 UNIX /var/lib/xcp/xapi|session.login_with_password D:523a350c00e1|xapi] Session.create trackid=f1d2261cb14aa0761c3a7ecd58bf38d4 pool=false uname=root originator=cli is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Dec 16 16:49:03 xenserver90 xapi: [debug|xenserver90|7402 UNIX /var/lib/xcp/xapi|session.login_with_password D:523a350c00e1|mscgen] xapi=>xapi [label="pool.get_all"];
Dec 16 16:49:03 xenserver90 xapi: [debug|xenserver90|7403 UNIX /var/lib/xcp/xapi||dummytaskhelper] task dispatch:pool.get_all D:0116e34d9bd1 created by task D:523a350c00e1
Dec 16 16:49:03 xenserver90 xapi: [ info|xenserver90|7402 UNIX /var/lib/xcp/xapi|session.logout D:bcf38b811408|xapi] Session.destroy trackid=f1d2261cb14aa0761c3a7ecd58bf38d4
Dec 16 16:49:03 xenserver90 xapi: [debug|xenserver90|7404 UNIX /var/lib/xcp/xapi||cli] xe sr-list type=rbd username=root password=(omitted)
Dec 16 16:49:03 xenserver90 xapi: [ info|xenserver90|7404 UNIX /var/lib/xcp/xapi|session.login_with_password D:ff099a47ec19|xapi] Session.create trackid=8623a14e5e9b25d8b48107fda658b14c pool=false uname=root originator=cli is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Dec 16 16:49:03 xenserver90 xapi: [debug|xenserver90|7404 UNIX /var/lib/xcp/xapi|session.login_with_password D:ff099a47ec19|mscgen] xapi=>xapi [label="pool.get_all"];
Dec 16 16:49:03 xenserver90 xapi: [debug|xenserver90|7405 UNIX /var/lib/xcp/xapi||dummytaskhelper] task dispatch:pool.get_all D:554175289bbd created by task D:ff099a47ec19
Dec 16 16:49:03 xenserver90 xapi: [ info|xenserver90|7404 UNIX /var/lib/xcp/xapi|session.logout D:440ec21eefb7|xapi] Session.destroy trackid=8623a14e5e9b25d8b48107fda658b14c

SMlog(no error) :

Dec 16 16:56:45 xenserver90 SM: [28821] ['uuidgen', '-r']
Dec 16 16:56:45 xenserver90 SM: [28821]   pread SUCCESS
Dec 16 16:56:45 xenserver90 SM: [28821] ['ceph', 'df', '--format', 'json', '--name', 'client.admin']
Dec 16 16:56:46 xenserver90 SM: [28821]   pread SUCCESS
Dec 16 16:56:46 xenserver90 SM: [28821] ['rbd', 'ls', '-l', '--format', 'json', '--pool', 'RBD_XenStorage-a04f4451-3761-415b-a6c2-2b12656cb13a', '--name', 'client.admin']
Dec 16 16:56:46 xenserver90 SM: [28821]   pread SUCCESS
Dec 16 16:56:46 xenserver90 SM: [28821] vdi_create {'sr_uuid': 'a04f4451-3761-415b-a6c2-2b12656cb13a', 'subtask_of': 'DummyRef:|b841ac12-2b43-1709-1c08-35475065a4d0|VDI.create', 'vdi_type': 'user', 'args': ['1075838976', 'xen_server_vol_1', '', '', 'false', '19700101T00:00:00Z$, '', 'false'], 'host_ref': 'OpaqueRef:698b2548-bdbe-91ea-35e7-76c7a49c3b4f', 'session_ref': 'OpaqueRef:9be6f546-ea71-a566-f8a7-02bf29df7e79', 'device_config': {'SRmaster': 'true'}, 'command': 'vdi_create', 'sr_ref': 'OpaqueRef:c2bc7247-9c5d-b110-7704-625513d0d50a', 'vdi_sm_co$fig': {}}
Dec 16 16:56:46 xenserver90 SM: [28821] RBDVDI.create for 84168a75-0829-4612-b0cc-a04ceff4b184
Dec 16 16:56:46 xenserver90 SM: [28821] ['rbd', 'create', 'VHD-84168a75-0829-4612-b0cc-a04ceff4b184', '--size', '1028', '--object-size', '2097152', '--image-format', '2', '--pool', 'RBD_XenStorage-a04f4451-3761-415b-a6c2-2b12656cb13a', '--name', 'client.admin']
Dec 16 16:56:46 xenserver90 SM: [28821]   pread SUCCESS
Dec 16 16:56:46 xenserver90 SM: [28821] ['rbd', 'image-meta', 'set', 'VHD-84168a75-0829-4612-b0cc-a04ceff4b184', 'VDI_LABEL', 'xen_server_vol_1', '--pool', 'RBD_XenStorage-a04f4451-3761-415b-a6c2-2b12656cb13a', '--name', 'client.admin']
Dec 16 16:56:46 xenserver90 SM: [28821]   pread SUCCESS
Dec 16 16:56:55 xenserver90 SM: [29065] ['ceph', 'df', '--format', 'json', '--name', 'client.admin']
Dec 16 16:56:56 xenserver90 SM: [29065]   pread SUCCESS
Dec 16 16:56:56 xenserver90 SM: [29065] ['rbd', 'ls', '-l', '--format', 'json', '--pool', 'RBD_XenStorage-a04f4451-3761-415b-a6c2-2b12656cb13a', '--name', 'client.admin']
Dec 16 16:56:56 xenserver90 SM: [29065]   pread SUCCESS
Dec 16 16:56:56 xenserver90 SM: [29065] vdi_delete {'sr_uuid': 'a04f4451-3761-415b-a6c2-2b12656cb13a', 'subtask_of': 'DummyRef:|d71324d1-77d2-078c-2823-99011b7a19b8|VDI.destroy', 'vdi_ref': 'OpaqueRef:3b5c7454-d6d3-f07f-8506-6990d40caf8f', 'vdi_on_boot': 'persist', 'args': [], 'vdi_location': '84168a75-0829-4612-b0cc-a04ceff4b184', 'host_ref': 'OpaqueRef:698b2548-bdbe-91ea-35e7-76c7a49c3b4f', 'session_ref': 'OpaqueRef:57bb1d9e-32a5-7ad0-1ec4-65fd3f2b9cef', 'device_config': {'SRmaster': 'true'}, 'command': 'vdi_delete', 'vdi_allow_caching': 'false', $sr_ref': 'OpaqueRef:c2bc7247-9c5d-b110-7704-625513d0d50a', 'vdi_uuid': '84168a75-0829-4612-b0cc-a04ceff4b184'}
Dec 16 16:56:56 xenserver90 SM: [29065] RBDVDI.delete for 84168a75-0829-4612-b0cc-a04ceff4b184
Dec 16 16:56:56 xenserver90 SM: [29065] RBDVDI.delete vdi_uuid : 84168a75-0829-4612-b0cc-a04ceff4b184
Dec 16 16:56:56 xenserver90 SM: [29065] ['rbd', 'rm', 'VHD-84168a75-0829-4612-b0cc-a04ceff4b184', '--pool', 'RBD_XenStorage-a04f4451-3761-415b-a6c2-2b12656cb13a', '--name', 'client.admin']
Dec 16 16:56:56 xenserver90 SM: [29065]   pread SUCCESS
Dec 16 16:57:04 xenserver90 SM: [29254] lock: opening lock file /var/lock/sm/d5905479-8c07-1d71-3150-61be2d340e91/sr
Dec 16 16:57:04 xenserver90 SM: [29254] lock: acquired /var/lock/sm/d5905479-8c07-1d71-3150-61be2d340e91/sr
Dec 16 16:57:04 xenserver90 SM: [29254] sr_scan {'sr_uuid': 'd5905479-8c07-1d71-3150-61be2d340e91', 'subtask_of': 'DummyRef:|dbf19442-fef9-7c2c-3024-09ae503fc91d|SR.scan', 'args': [], 'host_ref': 'OpaqueRef:698b2548-bdbe-91ea-35e7-76c7a49c3b4f', 'session_ref': 'OpaqueRef:cea38$e4-4c9c-93bb-d799-788950cdf108', 'device_config': {'SRmaster': 'true', 'serverpath': '/export/psXen7', 'server': '192.168.1.14'}, 'command': 'sr_scan', 'sr_ref': 'OpaqueRef:557aaf76-9b3d-9eb2-7252-fc2b47b53f6b'}
Dec 16 16:57:04 xenserver90 SM: [29254] ['/usr/bin/vhd-util', 'scan', '-f', '-c', '-m', '/var/run/sr-mount/d5905479-8c07-1d71-3150-61be2d340e91/*.vhd']
Dec 16 16:57:04 xenserver90 SM: [29254]   pread SUCCESS
Dec 16 16:57:04 xenserver90 SM: [29254] FileVDI.load vdi_uuid : b79785de-d311-4273-8186-304b4699f1d7
Dec 16 16:57:04 xenserver90 SM: [29254] FileVDI.load vdi_uuid : 7ce5d03d-925d-49f5-bea0-323041cc6bfb
Dec 16 16:57:04 xenserver90 SM: [29254] FileVDI.load vdi_uuid : 8dd11cfd-2681-423c-934e-bcb2ba103a3c
Dec 16 16:57:04 xenserver90 SM: [29254] FileVDI.load vdi_uuid : eb3e08e5-563c-469d-997d-09ff15551e2b
Dec 16 16:57:04 xenserver90 SM: [29254] FileVDI.load vdi_uuid : 41fa87b2-9175-4bb3-b052-38b0fe971d02
Dec 16 16:57:04 xenserver90 SM: [29254] FileVDI.load vdi_uuid : 1730ecd0-5f12-453d-8d96-e7121e148419
Dec 16 16:57:04 xenserver90 SM: [29254] FileVDI.load vdi_uuid : dbf23b5f-13c8-4c37-812d-941ee28782ec
Dec 16 16:57:04 xenserver90 SM: [29254] FileVDI.load vdi_uuid : e38c121c-9bd0-4392-b464-6a499fa5f798
Dec 16 16:57:04 xenserver90 SM: [29254] FileVDI.load vdi_uuid : f44a2e66-9b28-4831-a56f-7ff9b2bd66f9
Dec 16 16:57:04 xenserver90 SM: [29254] FileVDI.load vdi_uuid : c29f5c5a-5c80-4bc1-923e-750bb2f24a92
Dec 16 16:57:04 xenserver90 SM: [29254] FileVDI.load vdi_uuid : 4bdef00e-d50e-49e9-8ab4-8659ce35c3c9
scpcom commented 7 years ago

I am getting the same error. Host: XenServer 7.0 (with all patches up to XS70E032) Ceph: Jewel

Full Copy from Local Storage to RBD Storage -> works Fast Clone on RBD Storage -> works Full Copy on RBD Storage -> fails Full Copy from RBD Storage to Local Storage -> fails Moving from RBD Storage to Local Storage -> fails

Here is the full sequence I did on one VM for testing:

  1. Full Copy from Local Storage to RBD Storage
  2. Startup VM, Create Snapshot, shutdown VM (3 times)
  3. Fast Clone (5 times)
  4. Startup all 5 VM clones and shutdown again
  5. Tried to Full Copy a clone back to Local Storage
  6. Tried to Full Copy the VM from step 2 back to Local Storage
  7. Tried to move the VM from step 2 back to Local Storage
rposudnevskiy commented 7 years ago

Hi, Please check the latest version (commit d7b62cf) Thank you for you help with testing. Roman