UniFi OS - Dream Machines 3.x

pedropombeiro commented 1 year ago

Ubiquiti recently made 3.0.19 available in EA at https://community.ui.com/releases/UniFi-OS-Dream-Machines-3-0-19/aae685bb-4b96-4016-9125-29e57d7f2844

Known aspects of 3.x

/data is preserved in the upgrade, but udm-boot_1.0.7_all.deb needs to be installed again;
ssh-proxy is not present in 3.x, but manual install runs correctly;
This release has a kernel which is compiled without CONFIG_BPF_SYSCALL (tracked in https://github.com/unifi-utilities/unifios-utilities/issues/510), meaning podman cannot run on it without a custom kernel;
~The /etc/apt/sources.list is still pointing to stretch~. It now points to bullseye.

UPDATE: I've started a discussion at https://github.com/orgs/unifi-utilities/discussions/564 to get some feedback on how the migration to 3.x went for the more adventurous users among us.

ausfestivus commented 10 months ago

Try the install script

I dont want to install any of the podman/CNI stuff though. I just need the boot script capability.

ausfestivus commented 10 months ago

Looks like something else in the on-boot.service unit file is still unhappy. When I run restart the service:

Aug 30 13:58:08 homegw systemd[1]: /lib/systemd/system/udm-boot.service:11: Ignoring unknown escape sequences: "mkdir -p /data/on_boot.d && find -L /data/on_boot.d -mindepth 1 -maxdepth 1 -type f -print0 | sort -z | xargs -0 -r -n 1 -- bash -c 'if test -x "\$0"; then echo "%n: running \$0"; "\$0"; else case "\$0" in *.sh) echo "%n: sourcing \$0"; . "\$0";; *) echo "%n: ignoring \$0";; esac; fi'"
Aug 30 14:07:18 homegw systemd[1]: udm-boot.service: Succeeded.
Aug 30 14:07:18 homegw systemd[1]: Stopped Run On Startup UDM.
Aug 30 14:07:18 homegw systemd[1]: Stopping Run On Startup UDM...
Aug 30 14:07:18 homegw systemd[1]: Started Run On Startup UDM.
Aug 30 14:07:18 homegw bash[53949]: udm-boot.service: ignoring $0

I will continue to poke at the unit file until I work out what's going on.

ausfestivus commented 10 months ago

I found the correct string to use in the systemd unit file for the udm-boot.service.

ExecStart=bash -c 'mkdir -p ${DATA_DIR}/on_boot.d && find -L ${DATA_DIR}/on_boot.d -mindepth 1 -maxdepth 1 -type f -print0 | sort -z | xargs -0 -r -n 1 -- bash -c \'if test -x "$0"; then echo "%n: running $0"; "$0"; else case "$0" in *.sh) echo "%n: sourcing $0"; . "$0";; *) echo "%n: ignoring $0";; esac; fi\''

Full unit file is as follows:

[Unit]
Description=Run On Startup UDM
Wants=network-online.target
After=network-online.target
StartLimitIntervalSec=500
StartLimitBurst=5

[Service]
Restart=on-failure
RestartSec=5s
ExecStart=bash -c 'mkdir -p /data/on_boot.d && find -L /data/on_boot.d -mindepth 1 -maxdepth 1 -type f -print0 | sort -z | xargs -0 -r -n 1 -- bash -c \'if test -x "$0"; then echo "%n: running $0"; "$0"; else case "$0" in *.sh) echo "%n: sourcing $0"; . "$0";; *) echo "%n: ignoring $0";; esac; fi\''
RemainAfterExit=true

[Install]
WantedBy=multi-user.target

fdcastel commented 9 months ago

For reference. Installed today in a UDM-PRO running 3.1.16 with:

# Download package
curl -L https://github.com/unifi-utilities/unifios-utilities/raw/main/on-boot-script-2.x/packages/udm-boot-2x_1.0.1_all.deb -o /tmp/udm-boot-2x_1.0.1_all.deb

# Install it
dpkg -i /tmp/udm-boot-2x_1.0.1_all.deb

# Patches for 'udm-boot-2x_1.0.1_all.deb' package
sed -i 's/Description=Run On Startup UDM 2.x/Description=Run On Startup UDM 3.x/g' /lib/systemd/system/udm-boot.service
sed -i '/Restart=on-failure/d' /lib/systemd/system/udm-boot.service
sed -i '/RestartSec=5s/d' /lib/systemd/system/udm-boot.service

# Enable reload and start
systemctl enable udm-boot
systemctl daemon-reload
systemctl start udm-boot

Everything is working fine. Except for a minor bump I hit with ppp interfaces and network availability: #567

bruvv commented 9 months ago

@fdcastel does not work on my UDM v3.1.16

# systemctl status udm-boot.service
● udm-boot.service - Run On Startup UDM
     Loaded: loaded (/etc/systemd/system/udm-boot.service; enabled; vendor preset: enabled)
     Active: failed (Result: timeout) since Mon 2023-09-25 19:11:11 CEST; 12s ago
        CPU: 13.433s

Sep 25 19:09:52 DreamMachineV bash[228831]: Copying blob sha256:9bb61484bb07dd2a0582926f7faf12f7ffb2e193fc9b366438c4d590268098b3
Sep 25 19:09:52 DreamMachineV bash[228831]: Copying blob sha256:6afd91e12d00f0acf46a0663794806a3983cedf5087cbf9ee2068c80b8256e22
Sep 25 19:09:52 DreamMachineV bash[228831]: Copying blob sha256:15f67f31c2afc3853f803b54e5645f2ceef21c3b63199bd6670f71c0087cc5a5
Sep 25 19:09:52 DreamMachineV bash[228831]: Copying blob sha256:0bb78c3854f74738ff40ba8b65fe34cc59af60afab322d605b69c2fca47e8ed8
Sep 25 19:09:52 DreamMachineV bash[228831]: Copying blob sha256:0577ae6f826efe27b9a9dbb972eb0f56843c9f1cd14858d660c0a49daa9736b7
Sep 25 19:11:11 DreamMachineV systemd[1]: udm-boot.service: start operation timed out. Terminating.
Sep 25 19:11:11 DreamMachineV systemd[1]: udm-boot.service: Control process exited, code=killed, status=15/TERM
Sep 25 19:11:11 DreamMachineV systemd[1]: udm-boot.service: Failed with result 'timeout'.
Sep 25 19:11:11 DreamMachineV systemd[1]: Failed to start Run On Startup UDM.
Sep 25 19:11:11 DreamMachineV systemd[1]: udm-boot.service: Consumed 13.433s CPU time.

fdcastel commented 9 months ago

Sorry @bruvv but I have no idea what could possibly be wrong just looking at this log. We will probably need help from the script authors.

Did you run exactly all the steps I described above?
Do you remember of anything going wrong (any error messages?)

All I can suggest you for now is to uninstall the .deb package and try again.

If possible, post here the output of installation process.

m4n63 commented 9 months ago

@fdcastel does not work on my UDM v3.1.16


# systemctl status udm-boot.service

● udm-boot.service - Run On Startup UDM

     Loaded: loaded (/etc/systemd/system/udm-boot.service; enabled; vendor preset: enabled)

     Active: failed (Result: timeout) since Mon 2023-09-25 19:11:11 CEST; 12s ago

        CPU: 13.433s

Sep 25 19:09:52 DreamMachineV bash[228831]: Copying blob sha256:9bb61484bb07dd2a0582926f7faf12f7ffb2e193fc9b366438c4d590268098b3

Sep 25 19:09:52 DreamMachineV bash[228831]: Copying blob sha256:6afd91e12d00f0acf46a0663794806a3983cedf5087cbf9ee2068c80b8256e22

Sep 25 19:09:52 DreamMachineV bash[228831]: Copying blob sha256:15f67f31c2afc3853f803b54e5645f2ceef21c3b63199bd6670f71c0087cc5a5

Sep 25 19:09:52 DreamMachineV bash[228831]: Copying blob sha256:0bb78c3854f74738ff40ba8b65fe34cc59af60afab322d605b69c2fca47e8ed8

Sep 25 19:09:52 DreamMachineV bash[228831]: Copying blob sha256:0577ae6f826efe27b9a9dbb972eb0f56843c9f1cd14858d660c0a49daa9736b7

Sep 25 19:11:11 DreamMachineV systemd[1]: udm-boot.service: start operation timed out. Terminating.

Sep 25 19:11:11 DreamMachineV systemd[1]: udm-boot.service: Control process exited, code=killed, status=15/TERM

Sep 25 19:11:11 DreamMachineV systemd[1]: udm-boot.service: Failed with result 'timeout'.

Sep 25 19:11:11 DreamMachineV systemd[1]: Failed to start Run On Startup UDM.

Sep 25 19:11:11 DreamMachineV systemd[1]: udm-boot.service: Consumed 13.433s CPU time.

Can it be one of the scripts in the startup folder that does not terminate timely?

EDIT: the package basically justs runs your scripts in the /data/on_boot.d folder in a sorted order. So if any of these times out it could well give that error.

EDIT2: also what file did you install? the output indicates an older version, mine reads: udm-boot.service - Run On Startup UDM 2.x (notice the trailing 2.x)

bruvv commented 9 months ago

Ooops indeed I already had a version installed, removed everything redid and it works. saidly I am running against errors with podman and zerotier.

pedropombeiro commented 9 months ago

saidly I am running against errors with podman and zerotier.

That is to be expected though, given 3.x doesn't support podman.

fdcastel commented 9 months ago

EDIT2: also what file did you install? the output indicates an older version, mine reads: udm-boot.service - Run On Startup UDM 2.x (notice the trailing 2.x)

@m4n63 Nice catch! I didn't see this.

@bruvv It seems you have installed the wrong package.

If you follow the same steps I did this should be:

● udm-boot.service - Run On Startup UDM 3.x

Note that mine says 3.x because one of the steps was literally to replace 2.x with 3.x: -- Yeah. I have O.C.D. 😄

sed -i 's/Description=Run On Startup UDM 2.x/Description=Run On Startup UDM 3.x/g' /lib/systemd/system/udm-boot.service

SGXander commented 6 months ago

For reference. Installed today in a UDM-PRO running 3.1.16 with:

# Download package
curl -L https://github.com/unifi-utilities/unifios-utilities/raw/main/on-boot-script-2.x/packages/udm-boot-2x_1.0.1_all.deb -o /tmp/udm-boot-2x_1.0.1_all.deb

# Install it
dpkg -i /tmp/udm-boot-2x_1.0.1_all.deb

# Patches for 'udm-boot-2x_1.0.1_all.deb' package
sed -i 's/Description=Run On Startup UDM 2.x/Description=Run On Startup UDM 3.x/g' /lib/systemd/system/udm-boot.service
sed -i '/Restart=on-failure/d' /lib/systemd/system/udm-boot.service
sed -i '/RestartSec=5s/d' /lib/systemd/system/udm-boot.service

# Enable reload and start
systemctl enable udm-boot
systemctl daemon-reload
systemctl start udm-boot

Everything is working fine. Except for a minor bump I hit with ppp interfaces and network availability: #567

Working for 3.1.16 for me too. Has anyone successfully gone to 3.2.7?

ausfestivus commented 6 months ago

Has anyone successfully gone to 3.2.7?

I can confirm that the above snippet works on 3.2.7.

pedropombeiro commented 6 months ago

Working for 3.1.16 for me too. Has anyone successfully gone to 3.2.7?

I upgraded and initially my Pi-hole container didn't start. I had to run machinectl enable and machinectl start. But that was all. It was probably something with the startup script and not really a problem with the upgrade though.

fdcastel commented 6 months ago

Working for 3.1.16 for me too. Has anyone successfully gone to 3.2.7?

Tonight, my UDM-PRO was upgraded from 3.1.16 to 3.2.7. Unfortunately, udm-boot didn't survive the process.

Contents of /data/on_boot.d were preserved.

Service udm-boot was removed:

# systemctl start udm-boot
Failed to start udm-boot.service: Unit udm-boot.service not found.

A reinstall (following these steps) was needed to bring it back again.

Until we have a fix for this, I'd recommend keeping the Unifi OS Console away from auto-updates.

jgrig472 commented 6 months ago

UDM nSpawn Setup.pdf PiHole_Unbound Setup nSpawn.pdf I upgraded mine tonight to 3.2.7 and was able to get everything back up and working fine. Took me about 10 minutes of reinstalling a couple broken packages and then ran through some of my documentation steps to get it back up and working. I've attached a PDF copy of my setup instructions for this. I had to meld the instructions out there together when we had podman go away. I also have separate instructions for setting up PiHole with nSpawn too that I've attached.

kpr0th commented 6 months ago

For whatever it's worth, my udm-boot service survived the upgrade from 3.1.16 to 3.2.7, on a UDMB. But I only use it to kick off a script on boot, that uses cron to re-schedule itself to run a couple times a day. And I had most recently installed it using this method: curl -fsL "https://raw.githubusercontent.com/unifi-utilities/unifios-utilities/HEAD/on-boot-script/remote_install.sh" | /bin/sh

achalker commented 6 months ago

I think I've figured out the root cause of some of the issues people are having (in particular @bruvv and @m4n63 and maybe others).

I too followed the steps from @fdcastel several times, but noticed that the comments about being labeled "Run On Startup UDM 3.x" weren't there when doing a systemctl status udm-boot. That's when I also noticed that it was loading /etc/systemd/system/udm-boot.service NOT /lib/systemd/system/udm-boot.service (notice the /etc instead of /lib).

For some reason a previous version of the install script put things in /etc, and while dpkg was reinstalling stuff in /lib, the /etc version was still being used (and the /etc version seems to still be hardcoded in the remote_install.sh [FYI @boostchicken ])

So to get it working, all I had to do was to 'back out of everything I could':

rm /tmp/udm-boot-2x_1.0.1_all.deb
rm /etc/systemd/system/udm-boot.service
rm /etc/systemd/system/multi-user.target.wants/udm-boot.service
systemctl disable udm-boot
rm /lib/systemd/system/udm-boot.service

Then run the commands multiple people have already posted:

# Download package
curl -L https://github.com/unifi-utilities/unifios-utilities/raw/main/on-boot-script-2.x/packages/udm-boot-2x_1.0.1_all.deb -o /tmp/udm-boot-2x_1.0.1_all.deb

# Install it
dpkg -i /tmp/udm-boot-2x_1.0.1_all.deb

# Patches for 'udm-boot-2x_1.0.1_all.deb' package
sed -i 's/Description=Run On Startup UDM 2.x/Description=Run On Startup UDM 3.x/g' /lib/systemd/system/udm-boot.service
sed -i '/Restart=on-failure/d' /lib/systemd/system/udm-boot.service
sed -i '/RestartSec=5s/d' /lib/systemd/system/udm-boot.service

# Enable reload and start
systemctl enable udm-boot
systemctl daemon-reload
systemctl start udm-boot

Previously, when I ran this last command it would sit forever and then give me a time out error. This time it completed nearly immediately. A systemctl status udm-boot shows it as active and most importantly has the "Run On Startup UDM 3.x" label.

Hope this helps some people!

achalker commented 6 months ago

Another minor issue I just discovered. Looks like I had one of the original podman versions of containers running, and as such I had the 04-container-common.sh script in my on_boot.d folder, which is supposed to set limits on podman container logs. However, with podman gone in unifi os 3+, that script generates an error, which cause the udm-boot service to fail and exit instead of exit and stay active. I wonder if this is also causing some of the weirdness / inconsistencies people are seeing? Bottom line is to check for various older scripts in on_boot.d folder and remove them. Also might want to make a comment about not needing this script anymore in the appropriate readme here: https://github.com/unifi-utilities/unifios-utilities/blob/main/container-common/README.md

fdcastel commented 6 months ago

@achalker could you please provide the output for the following command on your UDM-PRO?

# find / | grep -F 'udm-boot' | grep -v -e '^/sys/fs/cgroup' | sort

Edit: Added | sort to the above command for easier comparisons.

This will list every file containing udm-boot on name, except for the ones found under /sys/fs/cgroup (which doesn't matter). Mine currently is:

/etc/systemd/system/multi-user.target.wants/udm-boot.service
/lib/systemd/system/udm-boot.service
/mnt/.rwfs/data/etc/systemd/system/multi-user.target.wants/udm-boot.service
/mnt/.rwfs/data/lib/systemd/system/udm-boot.service
/mnt/.rwfs/data/usr/share/doc/udm-boot-2x
/mnt/.rwfs/data/usr/share/doc/udm-boot-2x/copyright
/mnt/.rwfs/data/var/lib/dpkg/info/udm-boot-2x.list
/mnt/.rwfs/data/var/lib/dpkg/info/udm-boot-2x.md5sums
/mnt/.rwfs/data/var/lib/dpkg/info/udm-boot-2x.postinst
/mnt/.rwfs/data/var/lib/dpkg/info/udm-boot-2x.postrm
/mnt/.rwfs/data/var/lib/dpkg/info/udm-boot-2x.preinst
/mnt/.rwfs/data/var/lib/dpkg/info/udm-boot-2x.prerm
/mnt/.rwfs/data/var/lib/systemd/deb-systemd-helper-enabled/multi-user.target.wants/udm-boot.service
/mnt/.rwfs/data/var/lib/systemd/deb-systemd-helper-enabled/udm-boot.service.dsh-also
/run/systemd/units/invocation:udm-boot.service
/usr/share/doc/udm-boot-2x
/usr/share/doc/udm-boot-2x/copyright
/var/lib/dpkg/info/udm-boot-2x.list
/var/lib/dpkg/info/udm-boot-2x.md5sums
/var/lib/dpkg/info/udm-boot-2x.postinst
/var/lib/dpkg/info/udm-boot-2x.postrm
/var/lib/dpkg/info/udm-boot-2x.preinst
/var/lib/dpkg/info/udm-boot-2x.prerm
/var/lib/systemd/deb-systemd-helper-enabled/multi-user.target.wants/udm-boot.service
/var/lib/systemd/deb-systemd-helper-enabled/udm-boot.service.dsh-also

achalker commented 6 months ago

@fdcastel happy to. Note I don't have a UDM-PRO, just a plain old UDM (I've got 3 of them are various relatives sites in fact). Keep in mind I did already manually remove the /etc/systemd/system/udm-boot.service file yesterday on all 3 of them, as well as the symlink in /etc/systemd/system/multi-user.target.wants/udm-boot.service

Here's the output you requested from the newest UDM, which appears to be identical to yours.

/etc/systemd/system/multi-user.target.wants/udm-boot.service
/lib/systemd/system/udm-boot.service
/mnt/.rwfs/data/lib/systemd/system/udm-boot.service
/mnt/.rwfs/data/etc/systemd/system/multi-user.target.wants/udm-boot.service
/mnt/.rwfs/data/usr/share/doc/udm-boot-2x
/mnt/.rwfs/data/usr/share/doc/udm-boot-2x/copyright
/mnt/.rwfs/data/var/lib/systemd/deb-systemd-helper-enabled/udm-boot.service.dsh-also
/mnt/.rwfs/data/var/lib/systemd/deb-systemd-helper-enabled/multi-user.target.wants/udm-boot.service
/mnt/.rwfs/data/var/lib/dpkg/info/udm-boot-2x.list
/mnt/.rwfs/data/var/lib/dpkg/info/udm-boot-2x.postinst
/mnt/.rwfs/data/var/lib/dpkg/info/udm-boot-2x.postrm
/mnt/.rwfs/data/var/lib/dpkg/info/udm-boot-2x.preinst
/mnt/.rwfs/data/var/lib/dpkg/info/udm-boot-2x.prerm
/mnt/.rwfs/data/var/lib/dpkg/info/udm-boot-2x.md5sums
/usr/share/doc/udm-boot-2x
/usr/share/doc/udm-boot-2x/copyright
/var/lib/dpkg/info/udm-boot-2x.list
/var/lib/dpkg/info/udm-boot-2x.postinst
/var/lib/dpkg/info/udm-boot-2x.postrm
/var/lib/dpkg/info/udm-boot-2x.preinst
/var/lib/dpkg/info/udm-boot-2x.prerm
/var/lib/dpkg/info/udm-boot-2x.md5sums
/var/lib/systemd/deb-systemd-helper-enabled/multi-user.target.wants/udm-boot.service
/var/lib/systemd/deb-systemd-helper-enabled/udm-boot.service.dsh-also
/run/systemd/units/invocation:udm-boot.service
/tmp/udm-boot-2x_1.0.1_all.deb

Running this on the oldest UDM (which I did most of my original testing / setup on), the output is similar, but there are some differences:

/etc/systemd/system/multi-user.target.wants/udm-boot.service
/lib/systemd/system/udm-boot.service
/mnt/.rwfs/data/var/lib/systemd/deb-systemd-helper-enabled/udm-boot.service.dsh-also
/mnt/.rwfs/data/var/lib/systemd/deb-systemd-helper-enabled/multi-user.target.wants/udm-boot.service
/mnt/.rwfs/data/var/lib/dpkg/info/udm-boot-2x.postinst
/mnt/.rwfs/data/var/lib/dpkg/info/udm-boot-2x.preinst
/mnt/.rwfs/data/var/lib/dpkg/info/udm-boot-2x.list
/mnt/.rwfs/data/var/lib/dpkg/info/udm-boot-2x.prerm
/mnt/.rwfs/data/var/lib/dpkg/info/udm-boot-2x.postrm
/mnt/.rwfs/data/var/lib/dpkg/info/udm-boot-2x.md5sums
/mnt/.rwfs/data/lib/systemd/system/udm-boot.service
/mnt/.rwfs/data/etc/systemd/system/multi-user.target.wants/udm-boot.service
/mnt/.rwfs/data/usr/share/doc/udm-boot-2x
/mnt/.rwfs/data/usr/share/doc/udm-boot-2x/copyright
/usr/share/doc/udm-boot-2x
/usr/share/doc/udm-boot-2x/copyright
/var/lib/dpkg/info/udm-boot-2x.postinst
/var/lib/dpkg/info/udm-boot-2x.preinst
/var/lib/dpkg/info/udm-boot-2x.list
/var/lib/dpkg/info/udm-boot-2x.prerm
/var/lib/dpkg/info/udm-boot-2x.postrm
/var/lib/dpkg/info/udm-boot-2x.md5sums
/var/lib/systemd/deb-systemd-helper-enabled/multi-user.target.wants/udm-boot.service
/var/lib/systemd/deb-systemd-helper-enabled/udm-boot.service.dsh-also
/persistent/dpkg/bullseye/actions/udm-boot
/tmp/udm-boot-2x_1.0.1_all.deb

fdcastel commented 6 months ago

Here's the output you requested from the newest UDM, which appears to be identical to yours.

It is. Except for /tmp/udm-boot-2x_1.0.1_all.deb which should go away in the next few days. 👍

Running this on the oldest UDM (which I did most of my original testing / setup on), the output is similar, but there are some differences:

There are two differences:

/persistent/dpkg/bullseye/actions/udm-boot on yours
/run/systemd/units/invocation:udm-boot.service on mine

Unfortunately I don't have any UDM to inspect.

Anybody could chime in about these 2 files and tell us if this is the expected state on a UDM?

SGXander commented 5 months ago

From 3.1.16 to 3.2.9 and on_boot scripts weren't deleted... Confirmed working with 3.2.9 on UDM-Pro.

mrk3767 commented 5 months ago

Also running into the podman issues. Is there a workaround for this? Trying to get ZeroTier running on my UDM.

fdcastel commented 4 months ago

Manually upgraded from 3.2.7 to 3.2.9. Unfortunately, once again, udm-boot didn't survive the process.

Contents of /data/on_boot.d were preserved.

Service udm-boot was removed:

# systemctl start udm-boot
Failed to start udm-boot.service: Unit udm-boot.service not found.

A reinstall (following these steps) was needed to bring it back again.

Until we have a fix for this, I'd recommend keeping the Unifi OS Console away from auto-updates.

waffles0042 commented 4 months ago

Manually upgraded from 3.2.7 to 3.2.9. Unfortunately, once again, udm-boot didn't survive the process.
* Contents of `/data/on_boot.d` were preserved.

* Service `udm-boot` was removed:
  ```shell
  # systemctl start udm-boot
  Failed to start udm-boot.service: Unit udm-boot.service not found.
  ```
A reinstall (following these steps) was needed to bring it back again.

Until we have a fix for this, I'd recommend keeping the Unifi OS Console away from auto-updates.

Just thought I'd share that an abbreviated version of above linked 'recipe' has worked for me in my recent OS upgrade efforts:

systemctl enable udm-boot systemctl daemon-reload systemctl start udm-boot

charkins commented 4 months ago

I spent a little time digging into this yesterday. Unfortunately, I don't have a solution yet, but do have a pretty good understanding of why udm-boot is not surviving updates. I know that udm-boot used to survive firmware updates, but I don't have a udmpro running really old firmware and don't want to risk trying to downgrade that far, so I haven't determined what might have been different on the older firmwares.

Here is what I've figured out, which I thought might help if others also want to dig in...

TLDR;

Only these packages will be restored on firmware update:

DPKG_CACHE_UBNT_PKGS="unifi unifi-protect ulp-go unifi-access uid-agent unifi-drive unifi-innerspace unifi-connect unifi-talk"

ubnt-dpkg-cache/restore

When a package is installed (e.g. udm-boot-2x), a hook (/etc/dpkg/dpkg.cfg.d/020-ubnt-dpkg-cache) triggers execution of the /sbin/ubnt-dpkg-cache script, which is responsible for cacheing packages for restoring on firmware updates. In the default configuration, it will only cache packages these packages:

DPKG_CACHE_UBNT_PKGS="unifi unifi-protect ulp-go unifi-access uid-agent unifi-drive unifi-innerspace unifi-connect unifi-talk"

This behavior can be changed by adding DPKG_CACHE_ALL_PKG=yes to /etc/default/ubnt-dpkg-cache before the hook executes (i.e. before installing the udm-boot package). If we do so, then the package does get cached in /persistent/dpkg/bullseye/packages.

Unfortunately, this isn't enough. The /sbin/ubnt-dpkg-restore script only installs packages with an "install" action in /persistent/dpkg/bullseye/actions. The ubnt-dpkg-cache script creates the action file only for the DPKG_CACHE_UBNT_PKGS packages, regardless of the DPKG_CACHE_ALL_PKG setting.

I manually created the "install" action, hoping this would be enough:

echo "install" > /persistent/dpkg/bullseye/actions/udm-boot-2x

Unfortunately, I was thwarted again by the ubnt-dpkg-restore script also honoring the DPKG_CACHE_ALL_PKG setting. A firmware update overwrites /etc/default/ubnt-dpkg-cache, so when the restore script is executed after a firmware update, despite now having the cached package and an "install" action, it won't restore it because udm-boot-2x is not in DPKG_CACHE_UBNT_PKGS and DPKG_CACHE_ALL_PKG is unset.

Test Setup

To help others that have a udmpro that they can experiment with, here are the steps I used to facilitate repeated testing of the firmware update from v3.1.16 to v3.2.12.

I started with a udmpro fully updated to v3.2.12. I disconnected the WAN cable from the udmpro and am wired to a LAN port from my computer, which is also connected to internet via wifi. This ensures that firmware will NOT update during the initial setup wizard!

I downloaded copies of the the v3.1.16 and v3.2.12 firmwares on my laptop using the download links from the corresponding releases posts:

curl -o udmpro-v3.1.16.bin https://fw-download.ubnt.com/data/unifi-dream/84e6-UDMPRO-3.1.16-54b0d2b8-e966-4dbf-973e-bbc84c58ce47.bin

curl -o udmpro-v3.2.12.bin https://fw-download.ubnt.com/data/unifi-dream/b1a0-UDMPRO-3.2.12-24a7e106-d7e6-4c63-aefa-046c7eaf5a8e.bin

Factory Reset to v3.1.16

The udmpro can be downgraded to v3.1.16 by first scp'ing the firmware to the udmpro:

scp udmpro-v3.1.16.bin root@192.168.1.1:/tmp/fwupdate.bin

Then from the udmpro, apply the firmware update:

ubnt-systool fwupdate /tmp/fwupdate.bin

After the update completes, hold the hardware reset button down to force a factory reset.

The setup wizard will allow you to complete an offline setup. Enable ssh and set a password and you have a fresh v3.1.16 udmpro to test with.

Update to v3.2.12

After installing udm-boot and making any other tweaks to try and get it to survive the firmware update, the update can be applied in the same manner as v3.1.16 (without the factory reset). First scp the firmware to the udmpro:

scp udmpro-v3.2.12.bin root@192.168.1.1:/tmp/fwupdate.bin

Then from the udmpro, apply the firmware update:

ubnt-systool fwupdate /tmp/fwupdate.bin

After confirming the failure to preserve udm-boot, repeat the downgrade / factory reset to v3.1.16 to try again!

unifi-utilities / unifios-utilities