Closed mhoffmann75 closed 7 years ago
I was able to narrow it down a little further and it seems to me as if HTTP Export via XAPI is broken with RBDSR:
In Xen Orchestra Logs (syslog in my machine) it looks like Xapi produces IO Error :
Aug 16 11:42:25 xo-test xo-server[12387]: [Warn] Possibly unhandled rejection: XapiError: VDI_IO_ERROR(Device I/O errors) Aug 16 11:42:25 xo-test xo-server[12387]: at wrapError (/opt/xo-server/node_modules/xen-api/src/index.js:97:34) Aug 16 11:42:25 xo-test xo-server[12387]: at /source/xapi/index.js:168:51 Aug 16 11:42:25 xo-test xo-server[12387]: at /opt/xo-server/node_modules/lodash/_createBaseFor.js:17:11 Aug 16 11:42:25 xo-test xo-server[12387]: at baseForOwn (/opt/xo-server/node_modules/lodash/_baseForOwn.js:13:20) Aug 16 11:42:25 xo-test xo-server[12387]: at /opt/xo-server/node_modules/lodash/_createBaseEach.js:17:14 Aug 16 11:42:25 xo-test xo-server[12387]: at forEach (/opt/xo-server/node_modules/lodash/forEach.js:38:10) Aug 16 11:42:25 xo-test xo-server[12387]: at Collection.onAddOrUpdate (/source/xapi/index.js:140:14) Aug 16 11:42:25 xo-test xo-server[12387]: at Collection.patchedError as emit Aug 16 11:42:25 xo-test xo-server[12387]: at /opt/xo-server/node_modules/xo-collection/src/collection.js:280:15 Aug 16 11:42:25 xo-test xo-server[12387]: at /opt/xo-server/node_modules/lodash._baseeach/index.js:110:11 Aug 16 11:42:25 xo-test xo-server[12387]: at baseForOwn (/opt/xo-server/node_modules/lodash._baseeach/index.js:52:10) Aug 16 11:42:25 xo-test xo-server[12387]: at /opt/xo-server/node_modules/lodash._baseeach/index.js:80:14 Aug 16 11:42:25 xo-test xo-server[12387]: at /opt/xo-server/node_modules/lodash.foreach/index.js:26:9 Aug 16 11:42:25 xo-test xo-server[12387]: at /opt/xo-server/node_modules/xo-collection/src/collection.js:278:14 Aug 16 11:42:25 xo-test xo-server[12387]: at _combinedTickCallback (internal/process/next_tick.js:67:7) Aug 16 11:42:25 xo-test xo-server[12387]: at process._tickCallback (internal/process/next_tick.js:98:9) Aug 16 11:42:25 xo-test xo-server[12387]: Tue, 16 Aug 2016 09:42:25 GMT xo:perf blocked for 75ms Aug 16 11:42:25 xo-test xo-server[12387]: Rejected backup: XapiError: VDI_IO_ERROR(Device I/O errors) Aug 16 11:42:25 xo-test xo-server[12387]: Tue, 16 Aug 2016 09:42:25 GMT xen-api root@192.168.250.148: task.destroy(...) [46ms] ==> string Aug 16 11:42:25 xo-test xo-server[12387]: Tue, 16 Aug 2016 09:42:25 GMT xo:xapi task destroyed: VDI Export (ubuntu1) Aug 16 11:42:25 xo-test xo-server[12387]: Tue, 16 Aug 2016 09:42:25 GMT xo:api admin@admin.net | vm.rollingDeltaBackup(...) [9s] =!> Error: Rolling delta vm backup failed. Aug 16 11:42:25 xo-test xo-server[12387]: Tue, 16 Aug 2016 09:42:25 GMT xo:xapi Deleting VM XO_DELTA_BASE_VM_SNAPSHOT_DeltaBackup Aug 16 11:42:25 xo-test xo-server[12387]: Tue, 16 Aug 2016 09:42:25 GMT xo:xapi Deleting VDI ubuntu1 Aug 16 11:42:25 xo-test xo-server[12387]: Tue, 16 Aug 2016 09:42:25 GMT xo:api admin@admin.net | job.runSequence(...) [9s] ==> boolean Aug 16 11:42:25 xo-test xo-server[12387]: [Warn] Possibly unhandled rejection: Error: Rolling delta vm backup failed. Aug 16 11:42:25 xo-test xo-server[12387]: at _default._callee25$ (/source/xo-mixins/backups.js:539:12) Aug 16 11:42:25 xo-test xo-server[12387]: at tryCatch (/opt/xo-server/node_modules/regenerator-runtime/runtime.js:62:40) Aug 16 11:42:25 xo-test xo-server[12387]: at GeneratorFunctionPrototype.invoke as _invoke Aug 16 11:42:25 xo-test xo-server[12387]: at GeneratorFunctionPrototype.prototype.(anonymous function) as next Aug 16 11:42:25 xo-test xo-server[12387]: at step (/opt/xo-server/node_modules/babel-runtime/helpers/asyncToGenerator.js:17:30) Aug 16 11:42:25 xo-test xo-server[12387]: at /opt/xo-server/node_modules/babel-runtime/helpers/asyncToGenerator.js:28:20 Aug 16 11:42:25 xo-test xo-server[12387]: at run (/opt/xo-server/node_modules/babel-runtime/node_modules/core-js/library/modules/es6.promise.js:87:22) Aug 16 11:42:25 xo-test xo-server[12387]: at /opt/xo-server/node_modules/babel-runtime/node_modules/core-js/library/modules/es6.promise.js:100:28 Aug 16 11:42:25 xo-test xo-server[12387]: at flush (/opt/xo-server/node_modules/babel-runtime/node_modules/core-js/library/modules/_microtask.js:18:9) Aug 16 11:42:25 xo-test xo-server[12387]: at _combinedTickCallback (internal/process/next_tick.js:67:7) Aug 16 11:42:25 xo-test xo-server[12387]: at process._tickCallback (internal/process/next_tick.js:98:9)
Unfortunately no further details.
As far as i understand XO Delta Backup makes use of the XAPI with incremental export: https://wiki.xen.org/wiki/Disk_import/export_APIs
So basically a HTTP GET from XenServer should export delta backups: http://IP_of_XenServer/export_raw_vdi?vdi=xxx[&base=yyy]
This seems to work for local storage for what i was able to test via browser. But when used with RBDSR storage it seems to fail. Even exporting the entire disk with vdi=uuid_of_vdi (without base=yyy) seems to fail - it only exports a file with this content:
HTTP/1.1 500 Internal Error content-length: 266 content-type:text/html connection:close cache-control:no-cache, no-store
HTTP 500 internal server error
An unexpected error occurred; please wait a while and try again. If the problem persists, please contact your support representative.Additional information
VDI_IO_ERROR: [ Device I/O errors ]
I guess that's the same IO Error that XO Backup Logs from above provide.
Please note: No errors in SMlog.
@rposudnevskiy Any idea whats going on here and how to fix this?
Hi Martin,
The problem is that vhd-tool
(/usr/bin/vhd-tool) that is used by xe vdi-export
doesn't support rbd images. It is intended for work with images in VHD format but RBDSR uses raw images.
The capability to export only the changes between the VDI and the snapshot, that is used by Delta backups
can be used only with VHD format. If you try to export only deltas you can find messages like this base VDI is not a vhd; cannot compute differences
in /var/log/xensource.log
As for exporting the entire disk, the vhd-tool
uses raw
format in this case but catch error too.
Aug 17 11:06:55 HV-VMH0154AE xapi: [ info|HV-VMH0154AE|191666 INET :::80|VDI export R:03671ba94b88|vhd_tool_wrapper] Executing /bin/vhd-tool stream --source-protocol none --source-format raw --source /dev/sm/backend/5aab7115-2d2c-466d-818c-909cff689467/bff7db0c-076f-4386-ad48-fd9a2eb4d87d --destination-protocol none --destination-format raw --destination-fd d909a77e-428a-4940-a9e2-5f55396e0b98 --tar-filename-prefix --progress --machine --direct --path /dev/mapper:.
Aug 17 11:06:55 HV-VMH0154AE xapi: [error|HV-VMH0154AE|191666 INET :::80|VDI export R:03671ba94b88|vhd_tool_wrapper] vhd-tool failed, returning VDI_IO_ERROR
Aug 17 11:06:55 HV-VMH0154AE xapi: [error|HV-VMH0154AE|191666 INET :::80|VDI export R:03671ba94b88|vhd_tool_wrapper] vhd-tool output: vhd-tool: internal error, uncaught exception:#012 Unix.Unix_error(Unix.EINVAL, "lseek", "")
And I don't know the reason of this error for now. It should work as it use raw
format, but it doesn't work.
As workaround you can try to use the native ceph export rbd export-diff
rbd export
http://ceph.com/dev-notes/incremental-snapshots-with-rbd/ and write some scripts to run them from cron.
Hi Martin, I have made some fixes for Issue #14. Actually it's more a hack but not real fixes but I hope it can help you with XO. Please see Add_support_for_xe_vdi-export for details. Please note that exported images can be imported only in RBD repository as they are exported in rbd diff format.
With update 982fbf2 XenOrchestra DeltaBackup should be fully supported
It seems RBDSR is not compatible with XenOrchestra's DeltaBackup - backups simply "fail". Unfortunately no further information from XO and no obvious errors in Xen's SMlog:
Placing the VM on local storage makes it work. So it just happens with VDIs on RBDSR Storage. As mentioned in #11 XenOrchestra's Backup works now. However in DeltaBackup-mode it makes full backup on first backup and incremental on further backups by leaving one snapshot intact. Furthermore DeltaBackup uses the HTTP XenAPI to export the VDI which is a lot faster than exporting via xe, which Backup utilizes. So using DeltaBackup has a lot of improvements over Backup and i would love to see it working with RBDSR volumes.