Stats not working with installed certificate

[billgertz]

@olivierlambert @julien-f @mas90 Stay with me as this is going to take a bit of background to cover. Please reference this issue registered with Xen Orchestra for the issue history.

We installed a Let's Encrypt certificate, disabled Unauthorized Certificate and ran into immediate issues with certificate errors. After we figured out that stunnel is used on the backend we concatenated the full chain server cert, the server cert key, and the Diffie Hellman parameters:

cat fullchain.cer server.key dh.pem > /etc/xensource/xapi-ssl.pem

Followed by an xe-toolstack-restart this solved the SSL errors. However we ran into another: Host, VM and SR stats no longer work. Looking in the daemon.log we see:

syslog:Jul  9 12:54:42 bfxoa001 xo-server[444]: 2020-07-09T10:54:42.116Z xo:api WARN admin.bill | vm.stats(...) [79ms] =!> Error: unable to verify the first certificate
syslog:Jul  9 12:55:26 bfxoa001 xo-server[444]: 2020-07-09T10:55:26.721Z xo:api WARN admin.bill | host.stats(...) [47ms] =!> Error: unable to verify the first certificate

It appears that the _xenrrd service is starting without referencing the /etc/xensource/xapi-ssl.conf file and uses an internal cert. Causing the problem with stats for any of the Hosts, VMs or SRs.

We are using XCP-ng 8.1.0 and Xen Orchestra compiled from sources (xo-server 5.60.0/ xo-web 5.60.0).

This issue has been logged with Xen Orchestra on Jun 30, 2019 with no resolution. I'll be happy to investigate and help where I can, all anyone need do is ask.

[stormi]

There has been a lot of work in Citrix around TLS certificates for the 8.2 release. Maybe this will be better in 8.2? See https://docs.citrix.com/en-us/citrix-hypervisor/whats-new.html and https://docs.citrix.com/en-us/citrix-hypervisor/hosts-pools.html#install-a-tls-certificate-on-your-server

[billgertz]

@stormi Thanks for posting that, so it looks like this will can wait until XCP-ng supports 8.2. Their article point 5 makes the TLS Certificate support fairly clear:

5. Easy admin TLS Certificate install: This feature is new with the Citrix Hypervisor 8.2 LTSR and allows organizations to easily replace the Citrix Hypervisor host TLS certificates within their environments. By incorporating the functionality into XenCenter, admins now have a much easier means of adding certificates, and XenCenter will even remind you when nearing certificate expiration.

Not an ocaml coder so their code is not easy to follow, from their xapi-project repository the port is spun up in rrdd_shared.ml:

let open Xmlrpc_client in
  let transport = SSL (SSL.make (), address, !https_port) in
  with_transport transport
    (with_http request (fun (_response, fd) ->
         try Rrd_unix.to_fd rrd fd with _ -> log_backtrace ())) ;

At the tail end of a monster long chain of in expressions, ugh. Looking forward to 8.2...

[HeMaN-NL]

Xencenter.... hope this will make it into xo as well t that time

[olivierlambert]

It will, it's planned.

[olivierlambert]

@billgertz do you still have the issue in 8.2? It seems we got a XO user reporting a similar issue.

[olivierlambert]

@HeMaN-NL BTW, it's in XO already (since November release IIRC October, see https://github.com/vatesfr/xen-orchestra/blob/master/CHANGELOG.md#5520-2020-10-30)

[olivierlambert]

If it's in 8.2, we can close this issue, I'll let Sam do it :)

[stormi]

Indeed. Closing.