The latest image (5.100.1) is not working

[headwhacker]

I'm running xen-orchestra as a K8s workload. Starting the latest image I get the error below repeatedly in the logs and the page does not load even though the container goes to a running state. Rolling back to the previous image (5.100.0) solves the issue for me. This is repeatable. So something in the latest image breaks at least for my setup.

(node:32) Warning: Accessing non-existent property 'padLevels' of module exports inside circular dependency (Use node --trace-warnings ... to show where the warning was created) [2022-08-10T12:39:47+0000] info : 'rpcbind' start: '/sbin/rpcbind' [2022-08-10T12:39:48+0000] info : 'rpcbind' start action done [2022-08-10T12:39:48+0000] info : 'xo-opm-7fffb55b8f-bbt7f' start action done [2022-08-10T12:39:48+0000] info : Awakened by User defined signal 1 [2022-08-10T12:39:48+0000] info : 'redis' start action done 2022-08-10T12:39:50.600Z xo:main INFO Configuration loaded. 2022-08-10T12:39:50.607Z xo:main INFO Web server listening on http://[::]:80 2022-08-10T12:39:50.637Z xo:main INFO Web server listening on https://[::]:443 2022-08-10T12:39:50.906Z xo:mixins:hooks WARN start failure { error: Error: spawn xenstore-read ENOENT at Process.ChildProcess._handle.onexit (node:internal/child_process:283:19) at onErrorNT (node:internal/child_process:478:16) at processTicksAndRejections (node:internal/process/task_queues:83:21) { errno: -2, code: 'ENOENT', syscall: 'spawn xenstore-read', path: 'xenstore-read', spawnargs: [ 'vm' ], cmd: 'xenstore-read vm' } }

[ronivay]

Hi,

I can't reproduce this with a fresh deployment. It's possible that there is problem with existing data related to same issue reported in https://github.com/ronivay/xen-orchestra-docker/issues/15 and issues linked in there from XenOrchestraInstallerUpdater repo.

[headwhacker]

seems like #135 is the issue close enough to what I am seeing. However, I don' t have the file /root/.config/xo-server/config.toml existing on my container.

[ronivay]

Yeah this doesn’t seem like redis connection issue, so no need to edit that. There was quite a few redis related fixes during that time other issues were reported which are most likely related. Also the fact that older container works supports this. I’ll push a new latest container to get past all those redis changes. Try that when updated and let’s see if it fixes the issue.

E: new version of 5.100.1 and latest now available

[headwhacker]

Tried the latest version still same issue for me.

[ronivay]

Okay. There’s not much on the logs to work with and i’m unable to reproduce this. I’d try with fresh data directories and if that works, try to export your XO config in a working version and import that to a fresh instance of newer version and see if that works.

[pkerwien]

I don't know if my problem is the same, but I cannot start 5.100.1 (Image ID: 8ba65b00fc6b) using podman on Debian 11. I have also tried using podman on openSUSE MicroOS with the same result. When I use docker on Debian 11 with the image 5.100.1, everything works.

This problem also goes away when reverting to the previous 5.100.0 image.

The log is filled with:

2022-08-15T16:28:04.627Z xo:xo-server ERROR unhandled error event {
  error: Error: connect ECONNREFUSED ::1:6379
      at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1187:16)
      at TCPConnectWrap.callbackTrampoline (node:internal/async_hooks:130:17) {
    errno: -111,
    code: 'ECONNREFUSED',
    syscall: 'connect',
    address: '::1',
    port: 6379
  }
}

podman.log

I will also try to run the container under k3s and see what result I'm getting there.

[pkerwien]

Same problem under k3s v1.24.3+k3s1: Image 5.100.1 generates a lot of:

2022-08-15T17:09:41.827Z xo:xo-server ERROR unhandled error event {
  error: Error: connect ECONNREFUSED ::1:6379
      at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1187:16)
      at TCPConnectWrap.callbackTrampoline (node:internal/async_hooks:130:17) {
    errno: -111,
    code: 'ECONNREFUSED',
    syscall: 'connect',
    address: '::1',
    port: 6379
  }
2022-08-15T17:09:41.849Z xo:xo-server ERROR unhandled error event {
  error: Error: connect ECONNREFUSED ::1:6379
      at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1187:16)
      at TCPConnectWrap.callbackTrampoline (node:internal/async_hooks:130:17) {
    errno: -111,
    code: 'ECONNREFUSED',
    syscall: 'connect',
    address: '::1',
    port: 6379
  }
}

and won't start. Using 5.100.0 works fine. I just used a simple setup like:

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: xo-server
  namespace: default
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: local-path
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: redis
  namespace: default
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: local-path
  resources:
    requests:
      storage: 1Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: xoa
spec:
  selector:
    matchLabels:
      app: xoa
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: xoa
    spec:
      volumes:
        - name: xo-server
          persistentVolumeClaim:
            claimName: xo-server
        - name: redis
          persistentVolumeClaim:
            claimName: redis
      containers:
        - image: docker.io/ronivay/xen-orchestra:5.100.1
          name: xoa
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 80
              name: http
          volumeMounts:
            - mountPath: "/var/lib/xo-server"
              name: xo-server
            - mountPath: "/var/lib/redis"
              name: redis
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: xoa
  name: xoa
spec:
  selector:
    app: xoa
  type: NodePort
  ports:
  - port: 80
    targetPort: 80

[ronivay]

Thanks. This indicates there is indeed an issue with redis connectivity. Don't know why i wasn't able to reproduce it. Also first message in this issue doesn't show redis connection issue. Anyway, i need to do some more testing. If it's just redis connection issue, it's a trivial fix.

[ronivay]

Hi,

I pushed new version which should force redis connection to v4 localhost address as XO will prefer v6 ::1 if not defined. I think this doesn't happen on every environment as i still wasn't able to reproduce it.

Please give it a shot and let me know how it goes.

[pkerwien]

Quickly made some tests remotely from work (so I haven't properly tested login in the webUI etc), but I could see that I now am redirected to the login screen when using 5.100.1 under podman and k3s. So everything looks ok there. But I also re-tested my docker setup, and this one seems to now have problems. But I need to double-check when at home and also easier can post logs etc.

[headwhacker]

Tested the new version and is working now. Thanks

[pkerwien]

False alarm. All seems OK!

[ronivay]

Awesome. Thanks for testing.