search

Jarli01 / xenorchestra_installer

A simple install script for Xen Orchestra
GNU General Public License v3.0
428 stars 86 forks source link

debian 8 . CPU high "node". not working #16

Closed txsastre closed 6 years ago

txsastre commented 6 years ago

Hi, I installed previously on a debian 9 and everything was fine. I tried on a debian 8 and it does not work, although it says it has finished. Then I can see that the CPU really working at almost 100%, a task named "node"

any clues ?

thanks.

Jarli01 commented 6 years ago

Type '''node -v''' and paste the output.

Node is a required dependency of XO.

txsastre commented 6 years ago

v8.11.1 nowadays I reinstaled several times. tried with debian 9 (it works) also an ubuntu server 16.04 (it works) the main problem is when trying Continous Replications, they only works with XOA the appliance.

so I investigated the XOA and I saw that it was running on a debian 8, so tried to replicate it, but its not working. Also I tried to create by hand, it worked, but once again unable to do CR.

Jarli01 commented 6 years ago

CR should work with XOCE, I know it was working at one time for certain.

I'd have to see what limit you're encountering with CR though to begin.

txsastre commented 6 years ago

hi, thanks for your interest. you can see here the thread about this problem https://xen-orchestra.com/forum/topic/638/error-continous-replications-5-18-0/34

Jarli01 commented 6 years ago

Node v 8 is LTS, which should be fine.

Please post the details as listed in the issue submission form.

Jarli01 commented 6 years ago

Are you attempting to have a CR job run from XS to XCP-ng?

txsastre commented 6 years ago

Expected Behavior

once used the script, XOCE works great exception of Continous Replicaltion.

Actual Behavior

firts backup (full) is done, but when trying another on the same VM it brings an error and its not done. Also if I try again, another Full backup is done instead an incremental one.

a large discusion and tests have been discused on the XOA forum it can be read here https://xen-orchestra.com/forum/topic/638/error-continous-replications-5-18-0/34

Confirm XOCE services are running

systemctl status xo-server.service

   Loaded: loaded (/lib/systemd/system/xo-server.service; enabled; vendor preset: enabled)
   Active: active (running) since dl 2018-04-30 12:56:16 CEST; 21min ago
 Main PID: 995 (node)
    Tasks: 16
   Memory: 155.2M
      CPU: 4min 23.878s
   CGroup: /system.slice/xo-server.service
           ├─ 995 /usr/local/bin/node ./bin/xo-server
           └─1186 /usr/local/bin/node /opt/xen-orchestra/node_modules/jest-worker/build/child.js

abr 30 13:13:04 ubuntu16 xo-server[995]:     at /opt/xen-orchestra/node_modules/core-js/modules/es6.promise.js:83:30
abr 30 13:13:04 ubuntu16 xo-server[995]:     at flush (/opt/xen-orchestra/node_modules/core-js/modules/_microtask.js:18:9)
abr 30 13:13:04 ubuntu16 xo-server[995]:     at _combinedTickCallback (internal/process/next_tick.js:131:7)
abr 30 13:13:04 ubuntu16 xo-server[995]:     at process._tickCallback (internal/process/next_tick.js:180:9)
abr 30 13:13:04 ubuntu16 xo-server[995]: 2018-04-30T11:13:04.511Z xo:xapi Deleting VM [XO Backup cr-debianjessie8] Debian Jessie 8.0
abr 30 13:13:04 ubuntu16 xo-server[995]: 2018-04-30T11:13:04.701Z xo:xapi Deleting VDI S-XO-deb8
abr 30 13:13:05 ubuntu16 xo-server[995]: 2018-04-30T11:13:05.429Z xo:xapi Deleting VM [XO Backup cr-debianjessie8] Debian Jessie 8.0
abr 30 13:13:05 ubuntu16 xo-server[995]: 2018-04-30T11:13:05.537Z xo:xapi Deleting VDI S-XO-deb8
abr 30 13:17:29 ubuntu16 xo-server[995]: 2018-04-30T11:17:29.345Z xo:main + WebSocket connection (::ffff:192.168.222.8)
abr 30 13:17:29 ubuntu16 xo-server[995]: 2018-04-30T11:17:29.642Z xo:perf blocked for 70ms

Provide the output of journalctl logs

journalctl -u xo-server -f -n 50

-- Logs begin at dl 2018-04-30 12:56:11 CEST. --
abr 30 13:07:12 ubuntu16 xo-server[995]: 2018-04-30T11:07:12.265Z xo:perf blocked for 60ms
abr 30 13:07:13 ubuntu16 xo-server[995]: 2018-04-30T11:07:13.571Z xo:perf blocked for 54ms
abr 30 13:07:14 ubuntu16 xo-server[995]: 2018-04-30T11:07:14.133Z xo:perf blocked for 58ms
abr 30 13:07:14 ubuntu16 xo-server[995]: 2018-04-30T11:07:14.598Z xo:perf blocked for 60ms
abr 30 13:07:15 ubuntu16 xo-server[995]: 2018-04-30T11:07:15.469Z xo:perf blocked for 50ms
abr 30 13:07:22 ubuntu16 xo-server[995]: 2018-04-30T11:07:22.643Z xo:perf blocked for 79ms
abr 30 13:07:42 ubuntu16 xo-server[995]: 2018-04-30T11:07:42.086Z xo:perf blocked for 55ms
abr 30 13:07:42 ubuntu16 xo-server[995]: 2018-04-30T11:07:42.683Z xo:perf blocked for 92ms
abr 30 13:07:46 ubuntu16 xo-server[995]: 2018-04-30T11:07:46.760Z xo:perf blocked for 67ms
abr 30 13:07:49 ubuntu16 xo-server[995]: 2018-04-30T11:07:49.109Z xo:perf blocked for 89ms
abr 30 13:07:49 ubuntu16 xo-server[995]: 2018-04-30T11:07:49.901Z xo:perf blocked for 84ms
abr 30 13:07:53 ubuntu16 xo-server[995]: 2018-04-30T11:07:53.662Z xo:perf blocked for 73ms
abr 30 13:07:54 ubuntu16 xo-server[995]: 2018-04-30T11:07:54.773Z xo:perf blocked for 103ms
abr 30 13:07:55 ubuntu16 xo-server[995]: 2018-04-30T11:07:55.257Z xo:perf blocked for 84ms
abr 30 13:12:17 ubuntu16 xo-server[995]: 2018-04-30T11:12:17.998Z xo:perf blocked for 97ms
abr 30 13:12:20 ubuntu16 xo-server[995]: 2018-04-30T11:12:20.209Z xo:perf blocked for 102ms
abr 30 13:13:00 ubuntu16 xo-server[995]: 2018-04-30T11:13:00.220Z xo:xapi Snapshotting VM Debian Jessie 8.0 as [XO Backup cr-debianjessie8] Debian Jessie 8.0
abr 30 13:13:04 ubuntu16 xo-server[995]: acb8d0d6-1b44-3a85-ce8b-032c42c69010
abr 30 13:13:04 ubuntu16 xo-server[995]: 2018-04-30T11:13:04.081Z xo:xapi Creating VM [XO Backup cr-debianjessie8] Debian Jessie 8.0
abr 30 13:13:04 ubuntu16 xo-server[995]: 2018-04-30T11:13:04.310Z xo:xapi Deleting VM [XO Backup cr-debianjessie8] Debian Jessie 8.0
abr 30 13:13:04 ubuntu16 xo-server[995]: Error: missing base VDI (copy of c88d4ade-786f-45a2-a425-0339755459fa)
abr 30 13:13:04 ubuntu16 xo-server[995]:     at /opt/xen-orchestra/packages/xo-server/src/xapi/index.js:1009:16
abr 30 13:13:04 ubuntu16 xo-server[995]:     at Generator.next (<anonymous>)
abr 30 13:13:04 ubuntu16 xo-server[995]:     at step (/opt/xen-orchestra/packages/xo-server/dist/xapi/index.js:82:221)
abr 30 13:13:04 ubuntu16 xo-server[995]:     at _next (/opt/xen-orchestra/packages/xo-server/dist/xapi/index.js:82:409)
abr 30 13:13:04 ubuntu16 xo-server[995]:     at /opt/xen-orchestra/packages/xo-server/dist/xapi/index.js:82:477
abr 30 13:13:04 ubuntu16 xo-server[995]:     at new Promise (/opt/xen-orchestra/node_modules/core-js/modules/es6.promise.js:168:7)
abr 30 13:13:04 ubuntu16 xo-server[995]:     at /opt/xen-orchestra/packages/xo-server/dist/xapi/index.js:82:97
abr 30 13:13:04 ubuntu16 xo-server[995]:     at /opt/xen-orchestra/packages/xo-server/src/xapi/index.js:999:26
abr 30 13:13:04 ubuntu16 xo-server[995]:     at /opt/xen-orchestra/packages/xo-server/src/utils.js:374:18
abr 30 13:13:04 ubuntu16 xo-server[995]:     at /opt/xen-orchestra/node_modules/lodash/_createBaseFor.js:17:11
abr 30 13:13:04 ubuntu16 xo-server[995]:     at baseForOwn (/opt/xen-orchestra/node_modules/lodash/_baseForOwn.js:13:20)
abr 30 13:13:04 ubuntu16 xo-server[995]:     at /opt/xen-orchestra/node_modules/lodash/_createBaseEach.js:17:14
abr 30 13:13:04 ubuntu16 xo-server[995]:     at forEach (/opt/xen-orchestra/node_modules/lodash/forEach.js:38:10)
abr 30 13:13:04 ubuntu16 xo-server[995]:     at map (/opt/xen-orchestra/packages/xo-server/src/utils.js:373:10)
abr 30 13:13:04 ubuntu16 xo-server[995]:     at /opt/xen-orchestra/packages/xo-server/src/xapi/index.js:999:30
abr 30 13:13:04 ubuntu16 xo-server[995]:     at Generator.next (<anonymous>)
abr 30 13:13:04 ubuntu16 xo-server[995]:     at step (/opt/xen-orchestra/packages/xo-server/dist/xapi/index.js:82:221)
abr 30 13:13:04 ubuntu16 xo-server[995]:     at _next (/opt/xen-orchestra/packages/xo-server/dist/xapi/index.js:82:409)
abr 30 13:13:04 ubuntu16 xo-server[995]:     at run (/opt/xen-orchestra/node_modules/core-js/modules/es6.promise.js:66:22)
abr 30 13:13:04 ubuntu16 xo-server[995]:     at /opt/xen-orchestra/node_modules/core-js/modules/es6.promise.js:83:30
abr 30 13:13:04 ubuntu16 xo-server[995]:     at flush (/opt/xen-orchestra/node_modules/core-js/modules/_microtask.js:18:9)
abr 30 13:13:04 ubuntu16 xo-server[995]:     at _combinedTickCallback (internal/process/next_tick.js:131:7)
abr 30 13:13:04 ubuntu16 xo-server[995]:     at process._tickCallback (internal/process/next_tick.js:180:9)
abr 30 13:13:04 ubuntu16 xo-server[995]: 2018-04-30T11:13:04.511Z xo:xapi Deleting VM [XO Backup cr-debianjessie8] Debian Jessie 8.0
abr 30 13:13:04 ubuntu16 xo-server[995]: 2018-04-30T11:13:04.701Z xo:xapi Deleting VDI S-XO-deb8
abr 30 13:13:05 ubuntu16 xo-server[995]: 2018-04-30T11:13:05.429Z xo:xapi Deleting VM [XO Backup cr-debianjessie8] Debian Jessie 8.0
abr 30 13:13:05 ubuntu16 xo-server[995]: 2018-04-30T11:13:05.537Z xo:xapi Deleting VDI S-XO-deb8
abr 30 13:17:29 ubuntu16 xo-server[995]: 2018-04-30T11:17:29.345Z xo:main + WebSocket connection (::ffff:192.168.222.8)
abr 30 13:17:29 ubuntu16 xo-server[995]: 2018-04-30T11:17:29.642Z xo:perf blocked for 70ms

Provide answers to these questions:

Jarli01 commented 6 years ago

Are each of your hypervisor's the same version?

All XS7.1 OR All XCP-ng 7.4

txsastre commented 6 years ago

hi, at this moment all of them are XS 7.1

Danp2 commented 6 years ago

May want to review this thread. It's from several years ago, but may help you diagnose the issue.

How much ram do you have assigned to the XO VM? Have you checked the syslog for these same "process out of memory" errors?

txsastre commented 6 years ago

I've setup the new VM XOCE as the XOA, now I added more RAM to the Ubuntu, now its dynamic 8GB-16GB tried again and fails again.

Jarli01 commented 6 years ago

@txsastre does the VM you're attempting to run CR against already have snapshots?

Jarli01 commented 6 years ago

@Danp2 based on your link it looks like an unknown issue within XS that caused this issue. Yet what seems odd is that @txsastre is saying that by using XOA the CR job operates without issue.

Danp2 commented 6 years ago

@Jarli01 There may have been multiple issues back then. One was pre-XS 7.x, so it shouldn't be coming into play here.

Jarli01 commented 6 years ago

As a side question, I'm curious if @txsastre has an unhealthy vdi chain (to many replications) like I was encountering a while back.

Here is a link to the issue I created over at XOs GH: https://github.com/vatesfr/xen-orchestra/issues/2047

Jarli01 commented 6 years ago

@txsastre Please provide logs from XOA and XOCE running the same settings for a CR job.

We won't be able to troubleshoot otherwise.

txsastre commented 6 years ago

hi everyone. I don't think so about a unhealthy vdi, 1 because on XOA works, 2 because I created new VM from scratch and no, they have no previous snapshots. Although when trying the test I previously delete them.

Which logs do you want ? where can I take from ?

Jarli01 commented 6 years ago

The logs within XOA and XOCE regarding the backup process would likely shed the most light on whatever is occurring here.

txsastre commented 6 years ago

yes, but where do I find it ? in the XOA/XOCE WegGUI ? or somewhere in /var/log ? thanks.

txsastre commented 6 years ago

by the way Continuous Replications works for you ?

Jarli01 commented 6 years ago

Copy the logging available from within XO and paste it into a code block here.

txsastre commented 6 years ago

well I watched /var/log/syslog and I found this. while trying to CR again over a VM (previously had a CR with a full backup)

Apr 30 18:23:34 ubuntu16 xo-server[993]: 2018-04-30T16:23:34.179Z xo:xapi Snapshotting VM Debian Jessie 8.0 as [XO Backup cr-debianjessie8] Debian Jessie 8.0
Apr 30 18:23:38 ubuntu16 xo-server[993]: 41d170a0-a97e-0ef2-0969-0d6099db5065
Apr 30 18:23:38 ubuntu16 xo-server[993]: 2018-04-30T16:23:38.785Z xo:xapi Creating VM [XO Backup cr-debianjessie8] Debian Jessie 8.0
Apr 30 18:23:39 ubuntu16 xo-server[993]: 2018-04-30T16:23:39.015Z xo:xapi Deleting VM [XO Backup cr-debianjessie8] Debian Jessie 8.0
Apr 30 18:23:39 ubuntu16 xo-server[993]: Error: missing base VDI (copy of b6e1f603-4981-4119-922c-e5a2ad67b6ee)
Apr 30 18:23:39 ubuntu16 xo-server[993]:     at /opt/xen-orchestra/packages/xo-server/src/xapi/index.js:1009:16
Apr 30 18:23:39 ubuntu16 xo-server[993]:     at Generator.next (<anonymous>)
Apr 30 18:23:39 ubuntu16 xo-server[993]:     at step (/opt/xen-orchestra/packages/xo-server/dist/xapi/index.js:82:221)
Apr 30 18:23:39 ubuntu16 xo-server[993]:     at _next (/opt/xen-orchestra/packages/xo-server/dist/xapi/index.js:82:409)
Apr 30 18:23:39 ubuntu16 xo-server[993]:     at /opt/xen-orchestra/packages/xo-server/dist/xapi/index.js:82:477
Apr 30 18:23:39 ubuntu16 xo-server[993]:     at new Promise (/opt/xen-orchestra/node_modules/core-js/modules/es6.promise.js:168:7)
Apr 30 18:23:39 ubuntu16 xo-server[993]:     at /opt/xen-orchestra/packages/xo-server/dist/xapi/index.js:82:97
Apr 30 18:23:39 ubuntu16 xo-server[993]:     at /opt/xen-orchestra/packages/xo-server/src/xapi/index.js:999:26
Apr 30 18:23:39 ubuntu16 xo-server[993]:     at /opt/xen-orchestra/packages/xo-server/src/utils.js:374:18
Apr 30 18:23:39 ubuntu16 xo-server[993]:     at /opt/xen-orchestra/node_modules/lodash/_createBaseFor.js:17:11
Apr 30 18:23:39 ubuntu16 xo-server[993]:     at baseForOwn (/opt/xen-orchestra/node_modules/lodash/_baseForOwn.js:13:20)
Apr 30 18:23:39 ubuntu16 xo-server[993]:     at /opt/xen-orchestra/node_modules/lodash/_createBaseEach.js:17:14
Apr 30 18:23:39 ubuntu16 xo-server[993]:     at forEach (/opt/xen-orchestra/node_modules/lodash/forEach.js:38:10)
Apr 30 18:23:39 ubuntu16 xo-server[993]:     at map (/opt/xen-orchestra/packages/xo-server/src/utils.js:373:10)
Apr 30 18:23:39 ubuntu16 xo-server[993]:     at /opt/xen-orchestra/packages/xo-server/src/xapi/index.js:999:30
Apr 30 18:23:39 ubuntu16 xo-server[993]:     at Generator.next (<anonymous>)
Apr 30 18:23:39 ubuntu16 xo-server[993]:     at step (/opt/xen-orchestra/packages/xo-server/dist/xapi/index.js:82:221)
Apr 30 18:23:39 ubuntu16 xo-server[993]:     at _next (/opt/xen-orchestra/packages/xo-server/dist/xapi/index.js:82:409)
Apr 30 18:23:39 ubuntu16 xo-server[993]:     at run (/opt/xen-orchestra/node_modules/core-js/modules/es6.promise.js:66:22)
Apr 30 18:23:39 ubuntu16 xo-server[993]:     at /opt/xen-orchestra/node_modules/core-js/modules/es6.promise.js:83:30
Apr 30 18:23:39 ubuntu16 xo-server[993]:     at flush (/opt/xen-orchestra/node_modules/core-js/modules/_microtask.js:18:9)
Apr 30 18:23:39 ubuntu16 xo-server[993]:     at _combinedTickCallback (internal/process/next_tick.js:131:7)
Apr 30 18:23:39 ubuntu16 xo-server[993]:     at process._tickCallback (internal/process/next_tick.js:180:9)
Apr 30 18:23:39 ubuntu16 xo-server[993]: 2018-04-30T16:23:39.214Z xo:xapi Deleting VM [XO Backup cr-debianjessie8] Debian Jessie 8.0
Apr 30 18:23:39 ubuntu16 xo-server[993]: 2018-04-30T16:23:39.404Z xo:xapi Deleting VDI S-XO-deb8
Apr 30 18:23:40 ubuntu16 xo-server[993]: 2018-04-30T16:23:40.105Z xo:xapi Deleting VM [XO Backup cr-debianjessie8] Debian Jessie 8.0
Apr 30 18:23:40 ubuntu16 xo-server[993]: 2018-04-30T16:23:40.231Z xo:xapi Deleting VDI S-XO-deb8
Jarli01 commented 6 years ago

Error: missing base VDI (copy of b6e1f603-4981-4119-922c-e5a2ad67b6ee)

Warning On your target host does this VM ever get created? Does a VM with the same name already exist on the CR target host?

Make sure to have a backup

If so try deleting this copy, and run the job again.

Jarli01 commented 6 years ago

What you need to do is watch both your source hypervisor and CR target hypervisor and see if a new VM is created once the CR job is run. If it never gets created this to me would indicate an issue with your hypervisors.

The issue isn't within the installation process, but something outside of it.

txsastre commented 6 years ago

ok. repeated with XOA, same initial error. now doing first copy with CR.

Apr 30 12:27:49 xoa xo-server[502]: 2018-04-30T16:27:49.028Z xo:xapi Snapshotting VM Debian Jessie 8.0 as [XO Backup debianjessi] Debian Jessie 8.0
Apr 30 12:27:52 xoa xo-server[502]: 2018-04-30T16:27:52.931Z xo:xapi Creating VM [XO Backup debianjessi] Debian Jessie 8.0
Apr 30 12:27:53 xoa xo-server[502]: 2018-04-30T16:27:53.177Z xo:xapi Creating VDI S-XO-deb8 on SR-FJ-HBA
Apr 30 12:27:54 xoa xo-server[502]: 2018-04-30T16:27:54.011Z xo:xapi Creating VBD for VDI S-XO-deb8 on VM [Importing…] Debian Jessie 8.0 (20180430T162752Z)
Apr 30 12:27:54 xoa xo-server[502]: 2018-04-30T16:27:54.016Z xo:xapi exporting VDI S-XO-deb8
Apr 30 12:27:54 xoa xo-server[502]: 2018-04-30T16:27:54.022Z xo:xapi Creating VIF for VM [XO Backup debianjessi] Debian Jessie 8.0 on network Pool-wide network associated with eth0
Apr 30 12:28:18 xoa kernel: [  588.708549] [UFW BLOCK] IN=eth0 OUT= MAC=33:33:00:00:00:01:1c:5f:2b:27:ca:80:86:dd SRC=fe80:0000:0000:0000:1e5f:2bff:fe27:ca80 DST=ff02:0000:0000:0000:0000:0000:0000:0001 LEN=377 TC=0 HOPLIMIT=255 FLOWLBL=0 PROTO=UDP SPT=62976 DPT=62976 LEN=337 
Apr 30 12:28:37 xoa xo-server[502]: 2018-04-30T16:28:37.894Z xo:perf blocked for 106ms
Apr 30 12:28:49 xoa kernel: [  618.848779] [UFW BLOCK] IN=eth0 OUT= MAC=33:33:00:00:00:01:1c:5f:2b:27:ca:80:86:dd SRC=fe80:0000:0000:0000:1e5f:2bff:fe27:ca80 DST=ff02:0000:0000:0000:0000:0000:0000:0001 LEN=377 TC=0 HOPLIMIT=255 FLOWLBL=0 PROTO=UDP SPT=62976 DPT=62976 LEN=337 
Apr 30 12:29:19 xoa kernel: [  648.989570] [UFW BLOCK] IN=eth0 OUT= MAC=33:33:00:00:00:01:1c:5f:2b:27:ca:80:86:dd SRC=fe80:0000:0000:0000:1e5f:2bff:fe27:ca80 DST=ff02:0000:0000:0000:0000:0000:0000:0001 LEN=377 TC=0 HOPLIMIT=255 FLOWLBL=0 PROTO=UDP SPT=62976 DPT=62976 LEN=337 
Apr 30 12:29:49 xoa kernel: [  679.124031] [UFW BLOCK] IN=eth0 OUT= MAC=33:33:00:00:00:01:1c:5f:2b:27:ca:80:86:dd SRC=fe80:0000:0000:0000:1e5f:2bff:fe27:ca80 DST=ff02:0000:0000:0000:0000:0000:0000:0001 LEN=377 TC=0 HOPLIMIT=255 FLOWLBL=0 PROTO=UDP SPT=62976 DPT=62976 LEN=337 
Apr 30 12:30:19 xoa kernel: [  709.259781] [UFW BLOCK] IN=eth0 OUT= MAC=33:33:00:00:00:01:1c:5f:2b:27:ca:80:86:dd SRC=fe80:0000:0000:0000:1e5f:2bff:fe27:ca80 DST=ff02:0000:0000:0000:0000:0000:0000:0001 LEN=377 TC=0 HOPLIMIT=255 FLOWLBL=0 PROTO=UDP SPT=62976 DPT=62976 LEN=337 
Apr 30 12:30:37 xoa xo-server[502]: 2018-04-30T16:30:37.311Z xo:perf blocked for 54ms
Apr 30 12:30:49 xoa kernel: [  739.405767] [UFW BLOCK] IN=eth0 OUT= MAC=33:33:00:00:00:01:1c:5f:2b:27:ca:80:86:dd SRC=fe80:0000:0000:0000:1e5f:2bff:fe27:ca80 DST=ff02:0000:0000:0000:0000:0000:0000:0001 LEN=377 TC=0 HOPLIMIT=255 FLOWLBL=0 PROTO=UDP SPT=62976 DPT=62976 LEN=337 
Apr 30 12:31:19 xoa kernel: [  769.540469] [UFW BLOCK] IN=eth0 OUT= MAC=33:33:00:00:00:01:1c:5f:2b:27:ca:80:86:dd SRC=fe80:0000:0000:0000:1e5f:2bff:fe27:ca80 DST=ff02:0000:0000:0000:0000:0000:0000:0001 LEN=377 TC=0 HOPLIMIT=255 FLOWLBL=0 PROTO=UDP SPT=62976 DPT=62976 LEN=337 
Apr 30 12:31:49 xoa kernel: [  799.677714] [UFW BLOCK] IN=eth0 OUT= MAC=33:33:00:00:00:01:1c:5f:2b:27:ca:80:86:dd SRC=fe80:0000:0000:0000:1e5f:2bff:fe27:ca80 DST=ff02:0000:0000:0000:0000:0000:0000:0001 LEN=377 TC=0 HOPLIMIT=255 FLOWLBL=0 PROTO=UDP SPT=62976 DPT=62976 LEN=337

Doing now a CR delta now. and it works

Apr 30 12:31:59 xoa xo-server[502]: 2018-04-30T16:31:59.405Z xo:xapi Snapshotting VM Debian Jessie 8.0 as [XO Backup debianjessi] Debian Jessie 8.0
Apr 30 12:32:04 xoa xo-server[502]: fc1a002c-3257-d987-8e53-15953d4c0a62
Apr 30 12:32:04 xoa xo-server[502]: 2018-04-30T16:32:04.021Z xo:xapi Creating VM [XO Backup debianjessi] Debian Jessie 8.0
Apr 30 12:32:04 xoa xo-server[502]: 2018-04-30T16:32:04.238Z xo:xapi Cloning VDI S-XO-deb8
Apr 30 12:32:07 xoa xo-server[502]: 2018-04-30T16:32:07.035Z xo:xapi Creating VBD for VDI S-XO-deb8 on VM [Importing…] Debian Jessie 8.0 (20180430T163204Z)
Apr 30 12:32:07 xoa xo-server[502]: 2018-04-30T16:32:07.038Z xo:xapi exporting VDI S-XO-deb8 (from base S-XO-deb8)
Apr 30 12:32:07 xoa xo-server[502]: 2018-04-30T16:32:07.041Z xo:xapi Creating VIF for VM [XO Backup debianjessi] Debian Jessie 8.0 on network Pool-wide network associated with eth0
Apr 30 12:32:14 xoa xo-server[502]: 2018-04-30T16:32:14.986Z xo:xapi Deleting VM [XO Backup debianjessi] Debian Jessie 8.0
Apr 30 12:32:15 xoa xo-server[502]: 2018-04-30T16:32:15.168Z xo:xapi Deleting VDI S-XO-deb8
Apr 30 12:32:20 xoa kernel: [  829.818490] [UFW BLOCK] IN=eth0 OUT= MAC=33:33:00:00:00:01:1c:5f:2b:27:ca:80:86:dd SRC=fe80:0000:0000:0000:1e5f:2bff:fe27:ca80 DST=ff02:0000:0000:0000:0000:0000:0000:0001 LEN=377 TC=0 HOPLIMIT=255 FLOWLBL=0 PROTO=UDP SPT=62976 DPT=62976 LEN=337 
Apr 30 12:32:50 xoa kernel: [  859.953326] [UFW BLOCK] IN=eth0 OUT= MAC=33:33:00:00:00:01:1c:5f:2b:27:ca:80:86:dd SRC=fe80:0000:0000:0000:1e5f:2bff:fe27:ca80 DST=ff02:0000:0000:0000:0000:0000:0000:0001 LEN=377 TC=0 HOPLIMIT=255 FLOWLBL=0 PROTO=UDP SPT=62976 DPT=62976 LEN=337
txsastre commented 6 years ago

now with XOCE creating firts backup

Apr 30 18:34:29 ubuntu16 xo-server[993]: 2018-04-30T16:34:29.706Z xo:xapi Snapshotting VM Debian Jessie 8.0 as [XO Backup cr-debianjessie8] Debian Jessie 8.0
Apr 30 18:34:33 ubuntu16 xo-server[993]: 2018-04-30T16:34:33.545Z xo:xapi Creating VM [XO Backup cr-debianjessie8] Debian Jessie 8.0
Apr 30 18:34:33 ubuntu16 xo-server[993]: 2018-04-30T16:34:33.773Z xo:xapi Creating VDI S-XO-deb8 on SR-FJ-HBA
Apr 30 18:34:34 ubuntu16 xo-server[993]: 2018-04-30T16:34:34.605Z xo:xapi exporting VDI S-XO-deb8
Apr 30 18:34:34 ubuntu16 xo-server[993]: 2018-04-30T16:34:34.608Z xo:xapi Creating VIF for VM [XO Backup cr-debianjessie8] Debian Jessie 8.0 on network Pool-wide network associated with eth0
Apr 30 18:34:42 ubuntu16 xo-server[993]: 2018-04-30T16:34:42.196Z xo:perf blocked for 63ms
Apr 30 18:34:43 ubuntu16 xo-server[993]: 2018-04-30T16:34:43.106Z xo:perf blocked for 68ms
Apr 30 18:34:47 ubuntu16 xo-server[993]: 2018-04-30T16:34:47.138Z xo:perf blocked for 74ms
Apr 30 18:34:48 ubuntu16 xo-server[993]: 2018-04-30T16:34:48.030Z xo:perf blocked for 59ms
... a lot of "xo:perf" here

and now the second copy . Fail

Apr 30 18:38:07 ubuntu16 xo-server[993]: 2018-04-30T16:38:07.388Z xo:xapi Snapshotting VM Debian Jessie 8.0 as [XO Backup cr-debianjessie8] Debian Jessie 8.0
Apr 30 18:38:11 ubuntu16 xo-server[993]: 1280a1f5-10db-4c6e-42bd-dab8b79924b8
Apr 30 18:38:11 ubuntu16 xo-server[993]: 2018-04-30T16:38:11.878Z xo:xapi Creating VM [XO Backup cr-debianjessie8] Debian Jessie 8.0
Apr 30 18:38:12 ubuntu16 xo-server[993]: 2018-04-30T16:38:12.108Z xo:xapi Deleting VM [XO Backup cr-debianjessie8] Debian Jessie 8.0
Apr 30 18:38:12 ubuntu16 xo-server[993]: Error: missing base VDI (copy of 95c17de7-f935-4aa4-873e-dd79b117afbc)
Apr 30 18:38:12 ubuntu16 xo-server[993]:     at /opt/xen-orchestra/packages/xo-server/src/xapi/index.js:1009:16
Apr 30 18:38:12 ubuntu16 xo-server[993]:     at Generator.next (<anonymous>)
Apr 30 18:38:12 ubuntu16 xo-server[993]:     at step (/opt/xen-orchestra/packages/xo-server/dist/xapi/index.js:82:221)
Apr 30 18:38:12 ubuntu16 xo-server[993]:     at _next (/opt/xen-orchestra/packages/xo-server/dist/xapi/index.js:82:409)
Apr 30 18:38:12 ubuntu16 xo-server[993]:     at /opt/xen-orchestra/packages/xo-server/dist/xapi/index.js:82:477
Apr 30 18:38:12 ubuntu16 xo-server[993]:     at new Promise (/opt/xen-orchestra/node_modules/core-js/modules/es6.promise.js:168:7)
Apr 30 18:38:12 ubuntu16 xo-server[993]:     at /opt/xen-orchestra/packages/xo-server/dist/xapi/index.js:82:97
Apr 30 18:38:12 ubuntu16 xo-server[993]:     at /opt/xen-orchestra/packages/xo-server/src/xapi/index.js:999:26
Apr 30 18:38:12 ubuntu16 xo-server[993]:     at /opt/xen-orchestra/packages/xo-server/src/utils.js:374:18
Apr 30 18:38:12 ubuntu16 xo-server[993]:     at /opt/xen-orchestra/node_modules/lodash/_createBaseFor.js:17:11
Apr 30 18:38:12 ubuntu16 xo-server[993]:     at baseForOwn (/opt/xen-orchestra/node_modules/lodash/_baseForOwn.js:13:20)
Apr 30 18:38:12 ubuntu16 xo-server[993]:     at /opt/xen-orchestra/node_modules/lodash/_createBaseEach.js:17:14
Apr 30 18:38:12 ubuntu16 xo-server[993]:     at forEach (/opt/xen-orchestra/node_modules/lodash/forEach.js:38:10)
Apr 30 18:38:12 ubuntu16 xo-server[993]:     at map (/opt/xen-orchestra/packages/xo-server/src/utils.js:373:10)
Apr 30 18:38:12 ubuntu16 xo-server[993]:     at /opt/xen-orchestra/packages/xo-server/src/xapi/index.js:999:30
Apr 30 18:38:12 ubuntu16 xo-server[993]:     at Generator.next (<anonymous>)
Apr 30 18:38:12 ubuntu16 xo-server[993]:     at step (/opt/xen-orchestra/packages/xo-server/dist/xapi/index.js:82:221)
Apr 30 18:38:12 ubuntu16 xo-server[993]:     at _next (/opt/xen-orchestra/packages/xo-server/dist/xapi/index.js:82:409)
Apr 30 18:38:12 ubuntu16 xo-server[993]:     at run (/opt/xen-orchestra/node_modules/core-js/modules/es6.promise.js:66:22)
Apr 30 18:38:12 ubuntu16 xo-server[993]:     at /opt/xen-orchestra/node_modules/core-js/modules/es6.promise.js:83:30
Apr 30 18:38:12 ubuntu16 xo-server[993]:     at flush (/opt/xen-orchestra/node_modules/core-js/modules/_microtask.js:18:9)
Apr 30 18:38:12 ubuntu16 xo-server[993]:     at _combinedTickCallback (internal/process/next_tick.js:131:7)
Apr 30 18:38:12 ubuntu16 xo-server[993]:     at process._tickCallback (internal/process/next_tick.js:180:9)
Apr 30 18:38:12 ubuntu16 xo-server[993]: 2018-04-30T16:38:12.311Z xo:xapi Deleting VM [XO Backup cr-debianjessie8] Debian Jessie 8.0
Apr 30 18:38:12 ubuntu16 xo-server[993]: 2018-04-30T16:38:12.496Z xo:xapi Deleting VDI S-XO-deb8
Apr 30 18:38:13 ubuntu16 xo-server[993]: 2018-04-30T16:38:13.249Z xo:xapi Deleting VM [XO Backup cr-debianjessie8] Debian Jessie 8.0
Apr 30 18:38:13 ubuntu16 xo-server[993]: 2018-04-30T16:38:13.332Z xo:xapi Deleting VDI S-XO-deb8
Jarli01 commented 6 years ago

I just tested CR between my two systems at my house and everything appears to work just fine for me.

Each host is standalone (so not pooled). Can you provide any other settings you might have configured?

https://i.imgur.com/u373TsM.png

Jarli01 commented 6 years ago

Here on purpose I ran the job to fast and the job was stopped to prevent filling the target host (and affecting the VDI chain)

https://i.imgur.com/4sMZvW3.png

Jarli01 commented 6 years ago

And here you can see the job ran again, without issue. As well as both of my host (ignore the mess of VM's I have here)

https://i.imgur.com/alO7ZrM.png https://i.imgur.com/dhhwmhw.png

Jarli01 commented 6 years ago

And here the job is running on it's own every 5 minutes. Stopping automatically where it must to prevent a VDI overrun.

https://i.imgur.com/7SW9rMM.png

Jarli01 commented 6 years ago

And lastly here is a continuous run, without any failed backups.

https://i.imgur.com/Jq35924.png

txsastre commented 6 years ago

First of all, thank you very much for your time. Second I see that the CR are done from a local storage to another local storage, from one server to another. I tried and also failed, it has to be the OS of the XOCE, or the VM itself, I think that there is no other explanation. how is your VM configured ? CPU, RAM, OS , NIC ? Meanwhile I will create another pool with virtualbox and try again.

Jarli01 commented 6 years ago

My XOCE host host is running on Ubuntu 17.10 with 1Gb of ram and 1 vCPU, with a single NIC (not teamed at the hypervisor - though this is the production approach).

Nothing at all special.

Jarli01 commented 6 years ago

As a side note, I also had this setup and working at a previous client without issue (some version ago) when CR was originally released.

Back then that was on Ubuntu 16.04

Jarli01 commented 6 years ago

And lastly the host I setup last night as the CR target (lazy I know) was installed with XS7.2 and not at all updated last night.

But that shouldn't affect what is occurring here, which is an issue somewhere with the way things are configured.

Can you explain how you have things setup, your Hypervisors, your XOCE installation, switching etc.

Jarli01 commented 6 years ago

What do you mean by "Meanwhile I will create another pool with virtualbox and try again."

XS is a type 1 hypervisor (and by extension so is XCP-ng). Are you creating XS/XCP-ng Hypervisors within VirtualBox and attempting to have XO run on either of these systems and then backup to the other XS/XCP-ng VirtualBox Host?

Attempting to use VirtualBox to act as the hardware for your hypervisor could cause weird issues like this. . .

Please explain how you're hardware is configured.

txsastre commented 6 years ago

hi @Jarli01 again. I haven't time enough to set up another environment, yes I understand what you saying about virtualbox. No my pools are a real powerful machines, one set is for producion and the other one intends to be a backup pool. As I said in the Xen Orchestra forum, I upgrade to 5.19 (its been a few hours since they deployed) with your update script, and now it works fine. https://xen-orchestra.com/forum/topic/638/error-continous-replications-5-18-0/40

may be they found a bug related with my problem, or they were working in another that solved my situation.

thank you very much for your time and support. :)