vatesfr / xen-orchestra

The global orchestration solution to manage and backup XCP-ng and XenServer.
https://xen-orchestra.com
Other
782 stars 264 forks source link

[Backup Ng deltaBackup] VDI_IO_ERROR(Device I/O errors) #3242

Closed mike59999 closed 6 years ago

mike59999 commented 6 years ago

After the latest commit the error descriped in the issue #3205 still exists. From 20 VM's one is backed up and the other 19 are failing with the reason "VDI_IO_ERROR(Device I/O errors)". We have 2 XEN-Nodes with the latest patches applied and nfs for the backupstorage. In the attachment is the the error log from xoa, if you need more logs please tell me which files you need. Not sure if this helps but the backuptype from the delta is full because we created an new backupjob.

xo.log

PS: We using currently nfs for the vdi's with a 10Gbit-Card also for the XEN-Hosts. Only the Backupserver has currently a 1Gbitnic because we have no free slots for a 10Gbit Card.

Danp2 commented 6 years ago

I just noticed that Timeout = 3600 under the backup job's advanced settings. Could this be the cause of the issue?

olivierlambert commented 6 years ago

Very likely. Please disable it.

Danp2 commented 6 years ago

Also noticed that the VDI Export task appears to run much faster on XOA. Any suggestions to troubleshoot this on XO from source?

olivierlambert commented 6 years ago

I can't tell, depends entirely on your source environment. It's a stream, so your export speed is the speed of the slowest element in the chain.

Danp2 commented 6 years ago

@nraynaud Seems like I get the VDI_IO_ERROR(Device I/O errors) issue whenever

olivierlambert commented 6 years ago

Outside these cases, it works right?

Also works from the sources?

julien-f commented 6 years ago

@Danp2 Yes, that's normal behavior, not ideal though, the cancelation/timeout will be better reported in the future.

Danp2 commented 6 years ago

I believe so. However, I'm investigating why the VDI Export is running so slow.

AlexD2006 commented 6 years ago

Wow, so it might really be just some missunderstanding or wrong labeling in my opinion. After i upgraded to Version 5.28.0, the backup settings show up a field timeout with the tooltip/label "number of hours after which a job is considered failed / in hours". so i intuitively wrote 23 in this field cause my job is running every 24 hours. :-( now i deleted the value and started again and everything seems to be running well.

So what is this field timeout labeled "in hours" doing for real right now? timout failing in seconds?

Additional information: tested this in my "from sources" installation cause i still have no possibility to register and upgrade the downloaded XOA with my network restrictions.

Danp2 commented 6 years ago

@AlexD2006 What language are you using with XO? In English, I see "number of seconds after which a job is considered failed"

AlexD2006 commented 6 years ago

Using the english Version too. I have another environment without the last update. Version xo-server 5.27.1/xo-web 5.27.0, like the updated one was before. At this Version the tooltip is still saying "number of seconds after which a job is considered failed" and the greyed out prefill/label is not existant. (in the upgraded one it says "in hours" if it is empty)

So after the update, i checked all settings as i always do after an update to check if something changed. The label "in hours" caused me to intuitively change my old value (that was in seconds).

Sorry, i didnt communicated this in my first post, but i didnt have any context in my mind and so i just forgot the fact i changed this value.

julien-f commented 6 years ago

We changed it recently from seconds to hours in the UI but the value handling was not updated as well :confounded:

3575 will fix this.

olivierlambert commented 6 years ago

thanks for the feedback @Danp2 and @AlexD2006 !

AlexD2006 commented 6 years ago

thanks for the feedback @Danp2 and @AlexD2006 !

I have to say thanks for all your great work and your commitment in handling such reports. As always, unbelievable response times.

Kind regards Alex

olivierlambert commented 6 years ago

You are welcome. When you can afford it, please consider getting XOA with support. We got a nice remote assistance tool inside :) (so we can check your problem easily without asking you tons of questions!)