Restore problem on XenServer 7.2

[forall]

Hi,

A few days ago I upgraded XenServer from 7.0 to the latest version 7.2 and now I have problem with restore vm. When I try restore I have error messages:

DEBUG:root:PUT with following params: {'task_id': 'OpaqueRef:a063f4ef-a045-bebe-cb66-66ee74641a8a', 'session_id': 'OpaqueRef:8c73abfd-9ecd-37ae-911f-6a088c24ee55'} Error in command restore: [Errno 32] Broken pipe ERROR:root:Command restore failed with [Errno 32] Broken pipe Traceback (most recent call last): File "/usr/local/lib/python2.7/site-packages/xapi_back/cli.py", line 282, in main cmd.execute() File "/usr/local/lib/python2.7/site-packages/xapi_back/cli.py", line 127, in execute self.execute_for_one(session, s) File "/usr/local/lib/python2.7/site-packages/xapi_back/commands/restore_one.py", line 55, in execute_for_one _resp=c.put('/import', slot.get_reader(), slot.size_uncompressed, params) File "/usr/local/lib/python2.7/site-packages/xapi_back/http.py", line 86, in put 'Content-type':'application/octect-stream'}) File "/usr/local/lib/python2.7/httplib.py", line 1057, in request self._send_request(method, url, body, headers) File "/usr/local/lib/python2.7/httplib.py", line 1097, in _send_request self.endheaders(body) File "/usr/local/lib/python2.7/httplib.py", line 1053, in endheaders self._send_output(message_body) File "/usr/local/lib/python2.7/httplib.py", line 901, in _send_output self.send(message_body) File "/usr/local/lib/python2.7/httplib.py", line 870, in send self.sock.sendall(datablock) File "/usr/local/lib/python2.7/ssl.py", line 743, in sendall v = self.send(data[count:]) File "/usr/local/lib/python2.7/ssl.py", line 709, in send v = self._sslobj.write(data) error: [Errno 32] Broken pipe DEBUG:root:Progress thread error: ['INTERNAL_ERROR', '(Failure "Expected string, got \'N\'")'] DEBUG:root:Progress thread error: ['INTERNAL_ERROR', '(Failure "Expected string, got \'N\'")']

Did you check your script on XenServer 7.2?

[izderadicka]

I'm still on 7.0 I have no access to 7.2.

Do you have enough space for restore? Other functions are working? For restore - does it fail immediately? Or after some time? Problem seem to be with communication with xen server. Check logs on the server - DOM0 /var/log/xensource.log - if something went wrong with communication some errors should be there.

Try restore with xe, if that works.

I.

[forall]

Hi, Yes, I have a lot of free space for restore... Other function working, today I run a backup and everything ok, backup done. For restore problem is immediately with below error:

DEBUG:root:Executing command restore DEBUG:root:Starting progress monitor INFO:root:Starting import of VM proxytest DEBUG:root:PUT with following params: {'sr_id': 'OpaqueRef:6f75e8b9-b29d-1df7-d4fc-dd17850dada7', 'task_id': 'OpaqueRef:d7647580-f1e6-6865-92b6-ba4005644ac5', 'session_id': 'OpaqueRef:6707b8b9-c508-e119-544d-4e92f3f2b6f1'} Error in command restore: [Errno 32] Broken pipe ERROR:root:Command restore failed with [Errno 32] Broken pipe Traceback (most recent call last): File "/usr/local/lib/python2.7/site-packages/xapi_back/cli.py", line 282, in main cmd.execute() File "/usr/local/lib/python2.7/site-packages/xapi_back/cli.py", line 127, in execute self.execute_for_one(session, s) File "/usr/local/lib/python2.7/site-packages/xapi_back/commands/restore_one.py", line 55, in execute_for_one _resp=c.put('/import', slot.get_reader(), slot.size_uncompressed, params) File "/usr/local/lib/python2.7/site-packages/xapi_back/http.py", line 86, in put 'Content-type':'application/octect-stream'}) File "/usr/local/lib/python2.7/httplib.py", line 1057, in request self._send_request(method, url, body, headers) File "/usr/local/lib/python2.7/httplib.py", line 1097, in _send_request self.endheaders(body) File "/usr/local/lib/python2.7/httplib.py", line 1053, in endheaders self._send_output(message_body) File "/usr/local/lib/python2.7/httplib.py", line 901, in _send_output self.send(message_body) File "/usr/local/lib/python2.7/httplib.py", line 870, in send self.sock.sendall(datablock) File "/usr/local/lib/python2.7/ssl.py", line 743, in sendall v = self.send(data[count:]) File "/usr/local/lib/python2.7/ssl.py", line 709, in send v = self._sslobj.write(data) error: [Errno 32] Broken pipe DEBUG:root:Progress thread error: ['INTERNAL_ERROR', '(Failure "Expected string, got \'N\'")'] DEBUG:root:Progress thread error: ['INTERNAL_ERROR', '(Failure "Expected string, got \'N\'")']

It's any chance to check this problem on XS 7.2 and support it? xensource.log

[izderadicka]

Thanks for log - looking into it I found this part relevant to restore:

Nov  7 18:31:36 xhost1 xapi: [ info|xhost1|77929 INET :::80|task.create D:748d85386959|taskhelper] task VM.import R:d7647580f1e6 (uuid:858b3936-3b9d-1fa3-4aed-a881d765e0ee) created (trackid=aac5e350781971dfdff5849588e645fa) by task D:748d85386959
Nov  7 18:31:36 xhost1 xapi: [debug|xhost1|77931 INET :::80||import] import handler
Nov  7 18:31:36 xhost1 xapi: [debug|xhost1|77931 INET :::80|VM.import D:8ed52b6aa464|mscgen] xapi=>xapi [label="session.slave_login"];
Nov  7 18:31:36 xhost1 xapi: [debug|xhost1|77932 UNIX /var/lib/xcp/xapi||dummytaskhelper] task dispatch:session.slave_login D:c371492e4b25 created by task D:8ed52b6aa464
Nov  7 18:31:36 xhost1 xapi: [ info|xhost1|77932 UNIX /var/lib/xcp/xapi|session.slave_login D:595793e55223|xapi] Session.create trackid=185524ad5c5d0b970e3af1458211a0c9 pool=true uname= originator=xapi is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Nov  7 18:31:36 xhost1 xapi: [debug|xhost1|77932 UNIX /var/lib/xcp/xapi|session.slave_login D:595793e55223|mscgen] xapi=>xapi [label="pool.get_all"];
Nov  7 18:31:36 xhost1 xapi: [debug|xhost1|77933 UNIX /var/lib/xcp/xapi||dummytaskhelper] task dispatch:pool.get_all D:6c0f39c471e3 created by task D:595793e55223
Nov  7 18:31:36 xhost1 xapi: [ info|xhost1|77931 INET :::80|VM.import D:8ed52b6aa464|import] VM.import: SR = 'c901f202-0601-9bb6-65cc-3fe9d289b7cd (Data15T)'; force = false; full_restore = false
Nov  7 18:31:36 xhost1 xapi: [ warn|xhost1|77931 INET :::80|VM.import D:8ed52b6aa464|xapi] Host xhost1 cannot see SR c901f202-0601-9bb6-65cc-3fe9d289b7cd (Data15T)
Nov  7 18:31:36 xhost1 xapi: [debug|xhost1|77931 INET :::80|VM.import D:8ed52b6aa464|import] sr not available - redirecting
Nov  7 18:31:36 xhost1 xapi: [ warn|xhost1|77931 INET :::80|VM.import D:8ed52b6aa464|xapi] Host xhost1 cannot see SR c901f202-0601-9bb6-65cc-3fe9d289b7cd (Data15T)
Nov  7 18:31:36 xhost1 xapi: [debug|xhost1|77931 INET :::80|VM.import D:8ed52b6aa464|import] new location: https://192.168.12.19/import?sr_id=OpaqueRef:6f75e8b9-b29d-1df7-d4fc-dd17850dada7&task_id=OpaqueRef:d7647580-f1e6-6865-92b6-ba4005644ac5&session_id=OpaqueRef:6707b8b9-c508-e119-544d-4e92f3f2b6f1
Nov  7 18:31:36 xhost1 xapi: [debug|xhost1|77931 INET :::80|VM.import D:8ed52b6aa464|mscgen] xapi=>xapi [label="session.logout"];
Nov  7 18:31:36 xhost1 xapi: [debug|xhost1|77934 UNIX /var/lib/xcp/xapi||dummytaskhelper] task dispatch:session.logout D:b7e5c1e77e70 created by task D:8ed52b6aa464
Nov  7 18:31:36 xhost1 xapi: [ info|xhost1|77934 UNIX /var/lib/xcp/xapi|session.logout D:ee10da48bdfb|xapi] Session.destroy trackid=185524ad5c5d0b970e3af1458211a0c9
Nov  7 18:31:36 xhost1 xapi: [debug|xhost1|77931 INET :::80|VM.import D:8ed52b6aa464|import] import successful

From the log it looks like Xen server cannot find SR to restore VM to :

Host xhost1 cannot see SR c901f202-0601-9bb6-65cc-3fe9d289b7cd (Data15T)

Can you try to manually specify valid SR when doing restore? Also try xe vm-import - because xb is basically using same process.

[forall]

Thanks for help. This is a SR for second server.. If I have XS in pool, I can put only master, but why I can't see the second host (xhost2) and VMs from it?

[izderadicka]

Dont know - I do not have pool for testing - but somebody else tested on pool in past.
Can you backup from other hosts in pool? This should work. Try to xe vm-import for restore, if ti works.

[forall]

Hm, I can try backup, because I don't see VMs from the second host...and I received error:

xb --no-ssl-crt-check --debug backup --host xhost2 --vm proxytest DEBUG:root:Executing command backup Error in command backup: No such host configured: xhost2 ERROR:root:Command backup failed with No such host configured: xhost2 Traceback (most recent call last): File "/usr/local/lib/python2.7/site-packages/xapi_back/cli.py", line 282, in main cmd.execute() File "/usr/local/lib/python2.7/site-packages/xapi_back/cli.py", line 124, in execute raise CommandError('No such host configured: %s'%host_name) CommandError: No such host configured: xhost2

[izderadicka]

And if you add host to config file - hosts has to be added manually to the config? Or you can thry with existing host - in pool master should see all hosts in the pool - at least that was working for backup.
Not sure about restore ( but you can also try to add host from the pool and restore to it directly?).

[forall]

I can't add the second host to the configuration, because I received error: THIS IS A SLAVE HOST...

xb --no-ssl-crt-check list

Error in command list: ['HOST_IS_SLAVE', '192.168.12.17'] Traceback (most recent call last): File "/usr/local/lib/python2.7/site-packages/xapi_back/cli.py", line 282, in main cmd.execute() File "/usr/local/lib/python2.7/site-packages/xapi_back/cli.py", line 109, in execute with XAPISession(s['url'], s.get('user'), s.get('password')) as session: File "/usr/local/lib/python2.7/site-packages/xapi_back/cli.py", line 84, in enter self._password) File "/usr/local/lib/python2.7/site-packages/xapi_back/XenAPI.py", line 245, in call return self.send(self.name, args) File "/usr/local/lib/python2.7/site-packages/xapi_back/XenAPI.py", line 140, in xenapi_request self._login(methodname, params) File "/usr/local/lib/python2.7/site-packages/xapi_back/XenAPI.py", line 164, in _login result = _parse_result(getattr(self, 'session.%s' % method)(*params)) File "/usr/local/lib/python2.7/site-packages/xapi_back/XenAPI.py", line 219, in _parse_result raise Failure(result['ErrorDescription']) Failure: ['HOST_IS_SLAVE', '192.168.12.17']

cat xapi-config.cfg

[root@nashp1] ~# cat xapi-back.cfg { "servers": [{ "name": "xhost1", "url": "https://192.168.12.17", "user": "root", "password": "passw0rd" }, { "name": "xhost2", "url": "https://192.168.12.19", "user": "root", "password": "passw0rd" }], "storage_root": "/mnt/Data/XenBackup", "storage_retain": 3, "compress_level": 0, "log_file": "/var/log/xapi-back/log.txt", "mail_log": { "host": "", "port": , "from": "", "to": [ "" ], "user": "", "password": "", "secure": false, "subject": "Backup E-mail Report" } }

[izderadicka]

I see - that understandable, I think all communication has to go through pool master. But then you should seen all VMs in the pool or not? (when doing xb list on pool master?)

[forall]

Hi, All the time I run your script on FreeNAS, but after the latest upgrade I have problem with script, I think the problem is python3.6.. Did you test your script with the newer python version?

python setup.py install Traceback (most recent call last): File "setup.py", line 22, in m=re.search(r"version\s=\s'([\d.]+)'", file(pkg_file).read()) NameError: name 'file' is not defined

[izderadicka]

Please open above in separate issue as not related to this issue. Shortly script is only 2.7 compatible.