TrueNAS 12 fails to boot after installing xe-guest-utilities and dependencies

[BlackRockCity]

XCP-ng version 8.1 TrueNAS 12- Release

I followed instructions to install xe-guest-utilities as described here: https://xcp-ng.org/docs/guests.html#freenas-truenas

I had the simple FreeNAS 11.2 issue mentioned in the document above that required that I modify the .conf files as described here: https://www.justinsilver.com/random/fix-pkg-on-freenas-11-2/

Then the pkg installed along with dependencies (see screenshots). (I suspect the dependencies caused the system to be unbootable.)

Then I reverted the .conf files to re-enable local.conf and disable freebsd.conf

I then added the task to the init/shutdown area. /usr/local/sbin/xe-daemon -p /var/run/xe-daemon.pid &

Then I rebooted from the TrueNAS 12 web gui.

Now the console won't render in XOA and the web GUI is unreachable. I cannot ping TrueNAS IP.

I guess I will restore from a snapshot.

Force restarting from XOA generated this log:

vm.start
{
  "id": "564c67f7-5fe0-f3a8-35c2-6623d7f3c467"
}
{
  "code": "INTERNAL_ERROR",
  "params": [
    "xenopsd internal error: (Failure
  \"Error from xenguesthelper: Populate on Demand and Nested Virtualisation are mutually exclusive\")"
  ],
  "call": {
    "method": "VM.start",
    "params": [
      "OpaqueRef:209b3b1c-32fd-420d-8b47-06a5db7a3277",
      false,
      false
    ]
  },
  "message": "INTERNAL_ERROR(xenopsd internal error: (Failure
  \"Error from xenguesthelper: Populate on Demand and Nested Virtualisation are mutually exclusive\"))",
  "name": "XapiError",
  "stack": "XapiError: INTERNAL_ERROR(xenopsd internal error: (Failure
  \"Error from xenguesthelper: Populate on Demand and Nested Virtualisation are mutually exclusive\"))
    at Function.wrap (/opt/xo/xo-builds/xen-orchestra-202021090350/packages/xen-api/src/_XapiError.js:16:12)
    at /opt/xo/xo-builds/xen-orchestra-202021090350/packages/xen-api/src/transports/json-rpc.js:35:27
    at tryCatcher (/opt/xo/xo-builds/xen-orchestra-202021090350/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/opt/xo/xo-builds/xen-orchestra-202021090350/node_modules/bluebird/js/release/promise.js:547:31)
    at Promise._settlePromise (/opt/xo/xo-builds/xen-orchestra-202021090350/node_modules/bluebird/js/release/promise.js:604:18)
    at Promise._settlePromise0 (/opt/xo/xo-builds/xen-orchestra-202021090350/node_modules/bluebird/js/release/promise.js:649:10)
    at Promise._settlePromises (/opt/xo/xo-builds/xen-orchestra-202021090350/node_modules/bluebird/js/release/promise.js:729:18)
    at _drainQueueStep (/opt/xo/xo-builds/xen-orchestra-202021090350/node_modules/bluebird/js/release/async.js:93:12)
    at _drainQueue (/opt/xo/xo-builds/xen-orchestra-202021090350/node_modules/bluebird/js/release/async.js:86:9)
    at Async._drainQueues (/opt/xo/xo-builds/xen-orchestra-202021090350/node_modules/bluebird/js/release/async.js:102:5)
    at Immediate.Async.drainQueues [as _onImmediate] (/opt/xo/xo-builds/xen-orchestra-202021090350/node_modules/bluebird/js/release/async.js:15:14)
    at processImmediate (internal/timers.js:456:21)"
}

[olivierlambert]

Do you have nested virt enabled on this VM?

[BlackRockCity]

Yes, I did have nested virt enabled on this VM. I needed to enable it to get the TrueNAS jails plugins to work.

The reason I wanted xe-guest-utilities on all of my VMs was because I successfully installed nut-client on dom0 as described here: https://voice1.me/automated-shutdown-xenserver-7-x-with-network-ups-tool/

The xen-shutdown.sh that he provides requires xe-guest-utilities to be installed in order to gracefully shutdown your VMs in a power outage.

[olivierlambert]

If you use nested virt, I think you need to be sure dynamic min = dynamic max = static max memory.

[BlackRockCity]

I just tried changing the dynamic min to 32GiB and the VM still will not boot. At VM.start 10% it halts with "Start - unknown error from peer".

[olivierlambert]

Yes, so please raise Dynamic min to 32GiB. Then it should boot.

Also, please note that nested virt isn't ultra stable.

[BlackRockCity]

I just tried changing the dynamic min to 32GiB and the VM still will not boot. At VM.start 10% it halts with "Start - unknown error from peer".

[BlackRockCity]

Is there a more verbose way to witness VM start up? Perhaps by starting it from the host CLI? I'm fairly new to XCP-ng, but I have used xsconsole, XCP-ng Center, and XOA.

[olivierlambert]

Go into Settings/logs for detailed logs.

Alternatively, you can do xe vm-start uuid=<VM UUID>

[BlackRockCity]

From the CLI:

The server failed to handle your request, due to an internal error. The given message may give details useful for debugging the problem. message: xenopsd internal error: (Failure "Error from xenguesthelper: xenguest: xc_dom_boot_mem_init: [4] panic: xc_dom_boot.c:122: xc_dom_boot_mem_init: can't")

[BlackRockCity]

From Settings/Logs in XOA:

vm.start
{
  "id": "564c67f7-5fe0-f3a8-35c2-6623d7f3c467"
}
{
  "code": "INTERNAL_ERROR",
  "params": [
    "xenopsd internal error: (Failure
  \"Error from xenguesthelper: xenguest: xc_dom_boot_mem_init: [4] panic: xc_dom_boot.c:122: xc_dom_boot_mem_init: can't\")"
  ],
  "call": {
    "method": "VM.start",
    "params": [
      "OpaqueRef:209b3b1c-32fd-420d-8b47-06a5db7a3277",
      false,
      false
    ]
  },
  "message": "INTERNAL_ERROR(xenopsd internal error: (Failure
  \"Error from xenguesthelper: xenguest: xc_dom_boot_mem_init: [4] panic: xc_dom_boot.c:122: xc_dom_boot_mem_init: can't\"))",
  "name": "XapiError",
  "stack": "XapiError: INTERNAL_ERROR(xenopsd internal error: (Failure
  \"Error from xenguesthelper: xenguest: xc_dom_boot_mem_init: [4] panic: xc_dom_boot.c:122: xc_dom_boot_mem_init: can't\"))
    at Function.wrap (/opt/xo/xo-builds/xen-orchestra-202021090350/packages/xen-api/src/_XapiError.js:16:12)
    at /opt/xo/xo-builds/xen-orchestra-202021090350/packages/xen-api/src/transports/json-rpc.js:35:27
    at tryCatcher (/opt/xo/xo-builds/xen-orchestra-202021090350/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/opt/xo/xo-builds/xen-orchestra-202021090350/node_modules/bluebird/js/release/promise.js:547:31)
    at Promise._settlePromise (/opt/xo/xo-builds/xen-orchestra-202021090350/node_modules/bluebird/js/release/promise.js:604:18)
    at Promise._settlePromise0 (/opt/xo/xo-builds/xen-orchestra-202021090350/node_modules/bluebird/js/release/promise.js:649:10)
    at Promise._settlePromises (/opt/xo/xo-builds/xen-orchestra-202021090350/node_modules/bluebird/js/release/promise.js:729:18)
    at _drainQueueStep (/opt/xo/xo-builds/xen-orchestra-202021090350/node_modules/bluebird/js/release/async.js:93:12)
    at _drainQueue (/opt/xo/xo-builds/xen-orchestra-202021090350/node_modules/bluebird/js/release/async.js:86:9)
    at Async._drainQueues (/opt/xo/xo-builds/xen-orchestra-202021090350/node_modules/bluebird/js/release/async.js:102:5)
    at Immediate.Async.drainQueues [as _onImmediate] (/opt/xo/xo-builds/xen-orchestra-202021090350/node_modules/bluebird/js/release/async.js:15:14)
    at processImmediate (internal/timers.js:456:21)"
}

[olivierlambert]

Try without nested virt and see if it boots.

[BlackRockCity]

I just tried two things sequentially. First, I thought my server might be short of RAM so I lowered static, dyn min, and dyn max to 30GB (32GB is free to use so this should be OK). I got the same error as above.

Then I tried again with nested virt turned off, and I still see the same error.

I speculate that the dependencies that xe-guest-utilities required must have run over TrueNAS 12... especially because Python got suspiciously upgraded to 3.7.9 in the process.

[olivierlambert]

I don't think that's related: if you crash your OS, you still should be able to boot the guest.

[BlackRockCity]

I don't quite follow your last message.

TrueNAS 12.0 seemed bleeding edge in other ways before this happened. For example, iSCSI broke. I don't have a lot of experience with FreeNAS so I naively thought that upgrading from 11.3-U5 to 12.0 would go smoothly. Luckily, I have snapshots along the way on XCP-ng! Thanks for an impressive platform!

[olivierlambert]

Can you please try to boot with both:

• nested disabled
• 8GiB RAM for dynamic min, max and static max

[BlackRockCity]

That made a difference! The console now loaded and booting made a lot of progress....but it's hung.

[BlackRockCity]

I spoke too soon... It continued to boot! I can now login to the Web GUI, but the RAIDZ2 storage pool is not present.

From XOA console:

[BlackRockCity]

I am using a flashed HBA with LSI logic firmware and XCP-ng PCI passthrough.

[BlackRockCity]

I raised the RAM to 20 GiB and it boots. Still hangs a long time at the same place: Lo0: link state changed to UP.

I note that the management tools are not being seen by XOA.

And the BSD icon does not appear in the list of VMs which was promised in the xe-guest-utilities documentation. (arrow and ? are my additions).

[stormi]

Let's ping @pacohope who contributed the FreeBSD support in the guest tools installation scripts and the related docs.

[BlackRockCity]

I shutdown, turned on nested virt and the VM starts up. So some of the problems above had to do with my over provisioning of RAM (*which had not been a problem before because Dyn Min was set to 8.79Gib). There are no new Log files generated since the successful boot process.

There are clearly problems with the xe-guest-utilities. The storage pool is now missing and XOA doesn't see the guest tools, and the VM boot process takes many minutes longer.

[pacohope]

Thanks for pinging me. I think I can help. But not being a TrueNAS user, I can only point you in the right direction. The fix that you posted above for pkg is obviously fixing something where TrueNAS diverges from FreeBSD. It's a lot like how xcp-ng is a derivative of CentOS. There are all sorts of people out there talking about how they enable the CentOS repos to install ordinary software in dom0 on xcp-ng. The thing is, TrueNAS modifies some of the original FreeBSD operating system (much like how xcp-ng modifies things from CentOS). So standard packages that expect to find details where FreeBSD usually puts them, might not find them the same way if TrueNAS has modified it. So the same caveats apply to a TrueNAS user who uses FreeBSD packages that might apply to an xcp-ng user who goes installing public CentOS packages.

Earlier in the thread you said this was motivated by the desire to shut down a VM cleanly from the hypervisor. I don't think these guest utilities have anything to do with that. Someone like @stormi probably knows that better than I do. The signalling from hypervisor to guest VM telling it to shutdown is not dependent on having these guest utilities installed in the guest VM. In fact, once you see how these guest utilities work, you'll see they are mostly cosmetic. They're mainly about getting the right data into Xen Orchestra. You should test a few times to be sure, but I'm pretty sure that if you run xe vm-shutdown uuid=blah in your hypervisor, it will shutdown your TrueNAS or FreeBSD guest VM whether you have installed these guest utilities or not.

Deep Dive into xe-guest-utitilities

The FreeBSD version of "xe-guest-utilities" installs /usr/local/sbin/xe-daemon, which is just a shell script. Ultimately, all that shell script does is sleep 60 seconds, then calls /usr/local/sbin/xe-update-guest-attrs. The xe-update-guest-attrs program itself, is just a shell script. You can read the code. It keeps a cache of values in files down in /var/cache/xenstore so it doesn't have to update values at all in the hypervisor, if the values haven't changed. If you want to see what it's doing, do the following 4 commands:

sudo rm -rf /var/cache/xenstore/data
sudo rm -rf /var/cache/xenstore/attrs
sudo bash -x /usr/local/sbin/xe-update-guest-attrs > /tmp/attrs.out 2>&1
grep "xenstore write" /tmp/attrs.out

On my FreeBSD guest VM, I get the following output:

/usr/local/bin/xenstore write attr/eth0/ip 172.30.2.4
/usr/local/bin/xenstore write attr/eth1/ip 169.254.0.2
/usr/local/bin/xenstore write data/os_name 'FreeBSD 12.1-RELEASE-p10'
/usr/local/bin/xenstore write data/os_uname 12.1-RELEASE-p10
/usr/local/bin/xenstore write data/os_distro FreeBSD
/usr/local/bin/xenstore write attr/PVAddons/MajorVersion 6
/usr/local/bin/xenstore write attr/PVAddons/MinorVersion 2
/usr/local/bin/xenstore write attr/PVAddons/MicroVersion 0
/usr/local/bin/xenstore write attr/PVAddons/BuildVersion 76888
/usr/local/bin/xenstore write attr/PVAddons/Installed 1
/usr/local/bin/xenstore write data/updated 'Fri Oct 30 08:04:56 EDT 2020'

Take a look at the logic of what's happening in /usr/local/sbin/xe-update-guest-attrs on your system. The logic in the xe-update-guest-attrs may be parsing output of commands like ifconfig and getting the wrong values because of some differences between TrueNAS and FreeBSD.

All the os_name, os_uname, os_distro stuff is determined in /usr/local/sbin/xe-daemon, which calls uname a few times, parses the answer, and then caches the result in /var/cache/xe-linux-distribution so that other tools can just read those values. That, too, can be different between TrueNAS and FreeBSD. I have no idea.

Anyways, that's how the guest utilities work, and they're pretty basic. Hopefully this gives you a direction to go. You might have to make some local, TrueNAS-specific modifications to these scripts to get the right values visible in Xen Orchestra.

[stormi]

Thanks for pinging me. I think I can help. But not being a TrueNAS user, I can only point you in the right direction. The fix that you posted above for pkg is obviously fixing something where TrueNAS diverges from FreeBSD. It's a lot like how xcp-ng is a derivative of CentOS. There are all sorts of people out there talking about how they enable the CentOS repos to install ordinary software in dom0 on xcp-ng. The thing is, TrueNAS modifies some of the original FreeBSD operating system (much like how xcp-ng modifies things from CentOS). So standard packages that expect to find details where FreeBSD usually puts them, might not find them the same way if TrueNAS has modified it. So the same caveats apply to a TrueNAS user who uses FreeBSD packages that might apply to an xcp-ng user who goes installing public CentOS packages.

Then maybe we should change the documentation at https://xcp-ng.org/docs/guests.html#freenas-truenas because a user wrote: "But because it's based on FreeBSD, the packages from that OS can be installed, at your own risk. This is not a big issue for this particular package, because it's a leaf in the chain of dependencies - nothing in FreeNAS depends on it." However in the case of @BlackRockCity here it seems it has updated python packages. Or maybe it was a user mistake? It would be nice if that could be verified so that we could update the docs. Trying to summon @woopla and @etomm who both worked on that part of the docs!

Earlier in the thread you said this was motivated by the desire to shut down a VM cleanly from the hypervisor. I don't think these guest utilities have anything to do with that. Someone like @stormi probably knows that better than I do.

I may be mistaken, but I'm quite sure that you need the tools for XAPI to be able to shutdown a VM cleanly (to let the OS terminate by itself). Else the only option that is available is a forced shutdown (equivalent of electrical power off).

Deep Dive into xe-guest-utitilities

The FreeBSD version of "xe-guest-utilities" installs /usr/local/sbin/xe-daemon, which is just a shell script. Ultimately, all that shell script does is sleep 60 seconds, then calls /usr/local/sbin/xe-update-guest-attrs. The xe-update-guest-attrs program itself, is just a shell script. You can read the code. It keeps a cache of values in files down in /var/cache/xenstore so it doesn't have to update values at all in the hypervisor, if the values haven't changed. If you want to see what it's doing, do the following 4 commands:

sudo rm -rf /var/cache/xenstore/data
sudo rm -rf /var/cache/xenstore/attrs
sudo bash -x /usr/local/sbin/xe-update-guest-attrs > /tmp/attrs.out 2>&1
grep "xenstore write" /tmp/attrs.out

On my FreeBSD guest VM, I get the following output:

/usr/local/bin/xenstore write attr/eth0/ip 172.30.2.4
/usr/local/bin/xenstore write attr/eth1/ip 169.254.0.2
/usr/local/bin/xenstore write data/os_name 'FreeBSD 12.1-RELEASE-p10'
/usr/local/bin/xenstore write data/os_uname 12.1-RELEASE-p10
/usr/local/bin/xenstore write data/os_distro FreeBSD
/usr/local/bin/xenstore write attr/PVAddons/MajorVersion 6
/usr/local/bin/xenstore write attr/PVAddons/MinorVersion 2
/usr/local/bin/xenstore write attr/PVAddons/MicroVersion 0
/usr/local/bin/xenstore write attr/PVAddons/BuildVersion 76888
/usr/local/bin/xenstore write attr/PVAddons/Installed 1
/usr/local/bin/xenstore write data/updated 'Fri Oct 30 08:04:56 EDT 2020'

Take a look at the logic of what's happening in /usr/local/sbin/xe-update-guest-attrs on your system. The logic in the xe-update-guest-attrs may be parsing output of commands like ifconfig and getting the wrong values because of some differences between TrueNAS and FreeBSD.

All the os_name, os_uname, os_distro stuff is determined in /usr/local/sbin/xe-daemon, which calls uname a few times, parses the answer, and then caches the result in /var/cache/xe-linux-distribution so that other tools can just read those values. That, too, can be different between TrueNAS and FreeBSD. I have no idea.

Anyways, that's how the guest utilities work, and they're pretty basic. Hopefully this gives you a direction to go. You might have to make some local, TrueNAS-specific modifications to these scripts to get the right values visible in Xen Orchestra.

Maybe @etomm or @woopla can help here too.

[etomm]

Someone pinged me? So, to make the guest utilities works in TrueNAS 12.0 (and btw, if I am not wrong in 11.3 too), the only thing that should be needed is to add a tunable in System->Tunables like this:

I don't recall to have done nothing else in 11.3 and then the upgrade went smooth to 12.0

[pacohope]

Ok. So I tested. Lemme tell you what I found. I installed a really basic TrueNAS 12.0 system off the download link on the web site. All I did was boot it. So the other things in this thread about nested virtualisation don't apply. I didn't solve those.

1. I was wrong: You do need the xe-daemon and xe-guest-utilities installed to shutdown TrueNAS gracefully from Xen Orchestra. I tried shutting down from XO and it says vm lacks feature. I installed the xe-guest-utilities and then it worked properly.
2. The xe-guest-utilities worked fine following the instructions on the documents that are linked to above. To be more explicit:
   • I needed to edit both freebsd.conf and local.conf files to disable local repo and enable the FreeBSD repo
   • Then I ran pkg install xe-guest-utilities and let it install
   • Then I ran service start xenguest
   • The VM showed up in Xen Orchestra with accurate information and a cute daemon icon
3. I don't think @etomm's suggestion will work as is, because the xe-guest-utilities do not appear to be installed in FreeNAS 12.0 by default. Moreover, these utilities are a little unusual. In FreeBSD, anyways, they don't require a line in /etc/rc.conf to enable them. The /usr/local/etc/rc.d/xenguest runs sysctl kern.vm_guest and if it finds xen as the answer, it will automatically run.

Now, having rebooted, my xe-guest-utilities are no longer installed. My changes to the files related to pkg are gone. Again, I'm not a TrueNAS user. But it appears to be a lot like xcp-ng. Instead of a bunch of files, they make reference to a database. Every traditional config file includes a warning in a comment that unless you update values in the database, the changes won't persist. So my conclusion is that installing FreeBSD packages on TrueNAS has gotten more complicated than simply pretending it's FreeBSD and following FreeBSD instructions. I'd suggest that it has very little to do with xcp or Xen Orchestra and has a lot more to do with how TrueNAS/FreeNAS works. FreeBSD can be configured just by editing files. It would appear that this is not true for TrueNAS.

If a TrueNAS user wants to update our guidance, great. I don't have more definitive to recommend at the moment.

[olivierlambert]

I think we know some IXSystem people around, let me try by pinging @jaronparsons

[etomm]

@pacohope obviously I use TrueNAS 12 and xcp-ng 8.1 and I tested the graceful shutdown before writing.

This is my pkg info result from TrueNas:

root@freenas[~]# pkg info | grep xe
jailme-0.2.0                   Setuid version of jexec to allow normal users access to jails
libevent-2.1.11                API for executing callback functions on events or timeouts
pixman-0.38.4                  Low-level pixel manipulation library
tmux-3.0a_1                    Terminal Multiplexer

As I said before, I did not add anything else to enable them apart the tunable. Moreover that tunable was the one that should be enabled in FreeNAS 9.3 when there was no problem with the python version and the utilities were installed out of the box.

[jaronparsons]

I spoke with Engineering. If you can file a ticket at https://jira.ixsystems.com we will look to include the package in future releases. I told them I would appreciate it too! Until then you can follow the directions above to install the package. The package files will remain persistently, however the conf files may/will be overwritten either during an upgrade or possibly even a reboot/reset. There are things that keep these files in check under the hood.

[yocalebo]

The xe-guest-utilities was removed from our build system back in 2017 because it was failing to build at that time. I'm testing a new build on our 12 branch to see if that's been resolved upstream.

[etomm]

The support for xe-guest-utilities will be interesting, but as I said before, in my vm the graceful shutdown and even the information about ram usage and if agent is installed was available without them.

If you want I can test a fresh installation in the following days too, but I don't recall to have done something particular apart the tunable and I updated to 12 just a week ago.

[yocalebo]

The support for xe-guest-utilities will be interesting, but as I said before, in my vm the graceful shutdown and even the information about ram usage and if agent is installed was available without them.

If you want I can test a fresh installation in the following days too, but I don't recall to have done something particular apart the tunable and I updated to 12 just a week ago.

A fresh TN-12.0-RELEASE install would be great if you could test that.

[BlackRockCity]

Thank you everyone, I walk among giants!

Making sure that everyone is on the same page, FreeNAS has now become TrueNAS. So you will be seeing a lot of FreeNAS users upgrading to TrueNAS 12.x.

For those that suggest testing xe-guest-utilities on a vanilla TrueNAS Core 12.0-Release, that's sensible... but I started with FreeNAS-11.3-U2.1 --> FreeNAS-11.3-U4.1 --> FreeNAS-11.3-U5 --> TrueNAS-12.0-Release. I haven't made any under the hood modifications to my FreeNAS/TrueNAS system (–e.g., I didn't install any third party pkg from the shell or modify .conf files... with the exception of the modifications I describe above for xe-guest-utilities).

When I installed FreeNAS, I used the 'Other install media' template because nothing else seemed applicable.

Another important detail: When python upgraded to 3.7.9 it issued a message that about 5 python modules had not been updated because they were outside of the scope of the point release update. I ignored this message and did not upgrade the modules (it wasn't presented as a yes/no option). I'm going to attempt to regenerate that warning and then manually update the modules to see if that fixes anything.

I'm not an expert in these systems but I understand @pacohope that TrueNAS is not the same as FreeBSD and that .conf files get updated on the fly by settings stored in a database.

@stormi is correct that the reassurances from the xcp-ng documentation got me to this point. I know they were written with good intentions, and probably worked fine under earlier releases of FreeNAS but now the docs need to be updated.

@etomm I added the tunable you described and rebooted TrueNAS. It does not show up in /etc/rc.conf. Is that because the tunable is a database variable that gets inserted into a copy of rc.conf stored in a database record? In other words, the rc.conf I am looking at is not the one that is in play in the system database? It also seemingly had no effect on the XOA UI.. It still says 'Management Agent Not Detected'. See the screenshot:

NOTE: I see I searched for the wrong string. But the search for the correct string 'xenguest_enable' also returned 0 results.

[olivierlambert]

As soon we got a walk-through that works with TrueNAS, we will update the doc with a specific section for it :+1:

[BlackRockCity]

@jaronparsons I have added a comment and upvote to a pre-existing suggestion ticket from 4.5 months ago. Is this good enough to see engineering take action or should I file a separate bug?: https://jira.ixsystems.com/browse/NAS-106424

I'm not sure what you mean when you say, "Until then you can follow the directions above to install the package. ". I followed the directions above and they didn't lead to good results. And @pacohope says "Now, having rebooted, my xe-guest-utilities are no longer installed. My changes to the files related to pkg are gone.".

Therefore, getting xe-guest-utilities to work is now above my pay grade.

[etomm]

Total SHAME ON ME!!!!!!!! I performed the test and despite being sure to remember that when I upgraded from 11.2 to 11.3 I did not install the pkg, the tunable did not work. I found myself in the following situation:

The FreeNAS machine is my 12.0 FreeNAS installation that I upgraded from the one where I setup the first time the command that then went into the documentation and the other is the test machine that I configured. I am in the situation that I am not sure how I could achieve the agent to be seen. So I verified and in the upgraded environment I have still the pre init command configured, so my only valid idea is that the upgrade of FreeNAS and then to TrueNAS did not remove the daemon from the directory. It is unexpected to me because I remember that in the past I had always work to do after each upgrade to make it work, but I am not able to imagine another explanation. I also have a clear remembrance that when I upgraded the previous time, from 11.2 to 11.3, I felt relieved because I did not have to struggle with the pkg install and so in fact it has been something to do with the upgrade process.

@BlackRockCity yes that is my understanding of how the rc.conf works in FreeNAS/TrueNAS too.

[jaronparsons]

@jaronparsons I have added a comment and upvote to a pre-existing suggestion ticket from 4.5 months ago. Is this good enough to see engineering take action or should I file a separate bug?: https://jira.ixsystems.com/browse/NAS-106424

Should be. I spoke again with one of the engineers. He has fixed the package builds and it should be included in the next release hopefully.
I'm not sure what you mean when you say, "Until then you can follow the directions above to install the package. ". I followed the directions above and they didn't lead to good results. And @pacohope says "Now, having rebooted, my xe-guest-utilities are no longer installed. My changes to the files related to pkg are gone.".

The engineer I spoke with indicated that the instructions for FreeNAS should work to install the necessary packages, but this is unsupported and not recommended. Further, the .conf settings could disappear at anytime they are updated from the database. The correct way to make this persistent requires modifying core databases and is not supported/recommended. Therefore, getting xe-guest-utilities to work is now above my pay grade. I will attempt it in a test environment and see if I can find out why it doesnt work.

[yocalebo]

I've fixed the building of the xe-guest-utilties port in TrueNAS. The next stable release of TrueNAS (12.0-U1) will have it preinstalled by default. The supported method for activating the necessary daemon at boot is to set the xenguest_enable sysctl property that was mentioned above.

[olivierlambert]

Nice! Thank you @yocalebo

[etomm]

Thank you, from me too, @yocalebo!

In the meantime I found out the steps to have a persistent situation (seems also through updates...). Instead of installing the utilities I followed the pkg add way. I enabled the repo as per documentation and then:

• mkdir /tmp/repo
• cd /tmp/repo
• pkg fetch -o /tmp/repo/ xen-guest-tools
• pkg add -M All/xen-guest-tools-4.14.0.txz
• pkg fetch -o /tmp/repo/ xe-guest-utilities
• pkg add -M All/xe-guest-utilities-6.2.0_3.txz

Then I added just the original command I found at the first time, as per documentation, in the pre init scripts: /usr/local/sbin/xe-daemon -p /var/run/xe-daemon.pid &

This seems to persist also across the upgrades (something I really did not expect), so maybe @yocalebo can say if it must be reverted before upgrading to the TruaNAS 12.0-U1

[BlackRockCity]

Thank you everyone! @etomm your instructions above worked for me. I now see the 'management agent 6.2 detected' in XOA and the BSD icon appears next to the TrueNAS 12 VM. The next step will be to test server shutdown with nut-client.

Thank you @yocalebo for adding this into TrueNAS 12.0-U1.

[stormi]

Can someone contribute an update to the docs? Ideally taking into account the fact that people may be using different versions of FreeNAS / TrueNAS.

Direct contribute link: https://github.com/xcp-ng/xcp-ng-org/edit/master/docs/guests.md

[BlackRockCity]

@stormi I updated guests.md and submitted a pull request.

[etomm]

@stormi oh, I did the same at the same time of @BlackRockCity. The pr is this, you can discard it in case https://github.com/xcp-ng/xcp-ng-org/pull/45

[stormi]

I let you review each other's PR, see which one to keep and enrich after your review comments :)

[BlackRockCity]

@etomm @stormi I closed my pull request. @etomm's instructions are better. However, step 5 of reverting the repos to default is not needed if the user reboots the VM because of the way .conf files are reset by the database.

[BlackRockCity]

Also, I would like to point out that the sed commands on this page won't work but @etomm's should work... https://www.justinsilver.com/random/fix-pkg-on-freenas-11-2/

[etomm]

@BlackRockCity thanks, but I was liking more your incipit than mine. If you can review my English or you have suggestions to make it better, you are really welcome to add to it. Regarding step 5, I underlined in the last row of it that a reboot will reset the settings too, but I still thought to add it because if someone does not want to perform a reboot. Leaving the modified files maybe can have problem with the normal execution of TrueNAS.

[etomm]

Also, I would like to point out that the sed commands on this page won't work but @etomm's should work... https://www.justinsilver.com/random/fix-pkg-on-freenas-11-2/

The sed commands are the ones that were just in the documentation. They are also the one that I used to test and find out the new set of commands. They seem working and they are slightly different than the page that you reported, so I think we can accept them.

[BlackRockCity]

@etomm that's correct. I tested your commands and they do work. I just think people should be warned away from using the sed commands on https://www.justinsilver.com/random/fix-pkg-on-freenas-11-2/

[stormi]

It looks like the issue is solved and the documentation was updated, so I'm closing.