fichtner
commented
8 years ago Requires multiple things:
- Installer option for ZFS
- rc bootup mount awareness
- lots of GUI love in the form of "System: Firmware: Snapshots"
Closed fraenki closed 7 years ago
fichtner
commented
8 years ago Requires multiple things:
pv2b
commented
8 years ago I'm a big fan of ZFS, but I wonder if it's really the correct solution for an embedded system like OPNsense. Perhaps UFS snapshots might be worth considering for this?
Also, you'd have to consider that you're going to write your disk full if you keep writing to a disk with a snapshot on it, no matter what filesystem you use. That means that log files need to go on a different partition which is not snapshotted. On ZFS that's trivial, on UFS you probably need a new partition for it.
pv2b
commented
8 years ago After investigating the UFS option, it would seem it's not usable in this scenario. UFS snapshots do not have the functionality to "roll back" a filesystem. You'd end up having to copy all the changed files back onto the main filesystem, taking up twice the change space in the meantime. You'd also need to do this pre-boot.
A better approach might be something akin to what the nanobsd version does, i.e. keep two copies of the OS on two different partitions, and swap between them - in a "dual system" approach. That might still use the UFS snapshot functionality in order to clone the boot filesystem prior to upgrade.
Again, seperating out logging would probably still be a good idea.
pv2b
commented
8 years ago I just happened to run across this project: https://github.com/freqlabs/nbd-init
This may be worth looking at in order to provide a pre-boot environment that allows image switching.
fraenki
commented
8 years ago I'm a big fan of ZFS, but I wonder if it's really the correct solution for an embedded system like OPNsense.
I don't see why this should be an issue. We're talking about very small boot devices and I'm using USB/SD bootable disks for some time now on ZFS with no issues. Keeping snapshots for several releases does not consume a lot disk space in this case:
# zpool list
NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
freenas-boot 15G 5.06G 9.94G - - 33% 1.00x ONLINE -
People will use pretty big hardware in larger networks, RAM is cheap and plenty in amd64... In general there is nothing wrong with ZFS except that it needs this work get properly done.
UFS and embedded is trickier, but NanoBSD two-slice is old school and has proven to be of no particular interest in the time we started using it with OPNsense 15.1. The second slice has been dormant, unused, and the problems were not with data-loss or general desire to roll back upgrades. It also wastes 50% of the 4GB images. Thus, the second slice will die for 17.1 also because I asked for feedback in the forum and there was none:
https://forum.opnsense.org/index.php?topic=3446.0
The basic questions is what scenario is it supposed to help with? In this ticket's scope it's "flexible storage space, big servers, seamless rollbacks". In the case of embedded systems I don't really know what it is. In any case it probably needs a different approach than ZFS for sure.
fraenki
commented
8 years ago The basic questions is what scenario is it supposed to help with? In this ticket's scope it's "flexible storage space, big servers, seamless rollbacks".
Yes, that's exactly my use-case. Currently running all OPNsense boxes on amd64 hardware.
In the case of embedded systems I don't really know what it is.
I agree that ZFS may not be the right approach on embedded systems. Maybe we should just focus on amd64 as the scope of this issue for now.
bb-mitch
commented
8 years ago Hi guys - I've long disliked the old slice system... but I'm not sure abandoning the idea is a solution either.
The system never did work like it was supposed to (from the beginning I think), but I've seen people concerned about the issue regardless:
https://forum.opnsense.org/index.php?topic=1553.0
Is the vision for the new system going to address some sort of fail-safe for upgrading single system installs? In "big" installs, we use CARP, and upgrade one system at a time - but in a small install we are often limited to a single router. If an upgrade fails, or a slice fails to boot, it would be great to be able to recover - similar to a motherboard with a failsafe BIOS. If you try to flash and it fails, a hard reboot will detect the failed boot and a second hard reboot will cause the secondary / saved BIOS to be used. I'm not sure if the new system will have some kind of recovery feature for failures... the process can be as reliable as possible (and much respect and appreciation for the work you've done) but there will always be system crashes, power failures, and losses of connectivity during operations and upgrades.
If we could do something with the effect of logging in the boot table what action is happening, and then using that information to trigger the switch to the alternate slice, we'd actually have a recoverable system. Without having to put in a USB key and reinstalling.
I like the suggestion Franco made - I've long thought that myself - sharing the config partition was just moving the single point of failure - it makes more sense to sync the images at a known good point - once the upgrade is complete - and sync the config to the backup slice similarly.
I see you're on the path to removing it completely - not sure if my comments come too late in the process to have any effect. If the new systems will somehow solve this problem that's great - I just don't see any single filesystem that can do what needs to be done in an embedded / remotely managed way.
People often talk about "cheap" hardware, etc. - I'm more worried about recovery in a remote location without extended downtime or having to talk a non-techie through a re-install and recovery.
It would be much easier to say "Hey - the firmware upgrade last night crashed... I'd like you to power cycle your router - it won't boot, but it will detect the failure, Just give it 5 minutes and then reboot again and the system will boot from the last known good version".
Am I overreaching? Can I help?
Thanks!
fichtner
commented
8 years ago Hi @bb-mitch ,
It's on the roadmap, but you're not to late. I am impartial on the matter, I would only think doing something is better than doing nothing. Nano works well without the second slice, our firmware upgrades are very stable since 16.1 for Nano, with the last improvements that landed in 16.7. I haven't heard of a single Nano-Install issue in the past few months.
We need a maintainer with a vision, I'm happy to give the torch to somebody else. Right now I simply see we can do without the second slice that is dormant ever since 15.7 and very few people spoke up. Although most probably found another project that supports both slices, it's interesting to know that pfSense likely discontinues the approach with 2.4.
The practicality of the Raspberry-Pi FreeBSD images made me think this could be something that is worth the work for Nano, but as I said that's just one way of looking at it.
In OPNsense we have multiple mitigation techniques for failed updates: recover known states, opnsense-bootstrap, remote backup of configurations, remote sync of configurations. One of the fears that I have is that a broken SD card will take both slices down, which narrows the failure model further. It's only safe if we had two SD cards, but which device offers that?
There are two ways to proceed: we keep the status quo for 17.1 in order to get work into making the second slice usable through GUI commands, or we will go ahead with the revamp into an embedded-one-size-fits-all image with a singe slice.
I don't think you're overreacting, yet there are things to consider before going either way. :)
I heard from a guy they use fixed images with configs on USB sticks, so the acual upgrades are done by physically swapping the SD card, keeping the config on USB storage. This way you can send e.g. two SD cards for recovery and not lose your config, which can also be backed up by cloning the USB stick during maintenance or something.
As I said, lots of ways to fix "it". Maybe we can find a way to go ahead as planned and make your use case even more safe. It just takes a bit of talking and eventually coding.
Cheers, Franco
fichtner
commented
8 years ago (this also refs #1195)
ghost
commented
8 years ago so..... here Ill say it... I am currently working on such integrations. Things take time ... Im willing to contribute. compile a short feature list, and ill work through it. right now, its integration a form of backup. integrate DISKS / ZFS and SNAPSHOTS. then look at BE environments. Thoughts welcome.
bb-mitch
commented
8 years ago Hi guys :-)
Actually, the pcengines hardware (example only) support CF or SD, DOM (in some boards),and miniPCIX, So a single device could easily have multiple boot devices - just not multiple "SD". The system could boot from at least two internal "drives" plus the external USB port. What's missing is an easy way to backup / sync / fail over. I'd imagine that other platforms would work similarly (multiple boot devices at least including the internal device and USB if nothing else). But the lack of BIOS support for detecting the failed boot will break the process - which is outside the scope of the project right?
I'm not committed to the "slice" model of fail over - but it would seem that model of recovery that spans physical devices would have to involve the BIOS in processing a fail over. If we keep the slice we do have the possibility to boot off the primary or backup slice (and perhaps read the associated primary and backup config).
While the old model didn't make any sense to me (the the shared config seemed to be the weak point) at least it could allow a failback without BIOS support.
What's needed is a way of determining "success" or "failure" of a boot. Or a way to "force" fail over using something simple like timing a hard reboot.
My worry about the upgrade process or desire to have a way to recover from a failure is not related to the reliability of the process - but for the number of things go wrong "outside the lab".
IF the slice model works, the process could work without BIOS support. Once that works, we could look for BIOS support in popular embedded platforms. BIOS support would likely involve enabling a hardware watchdog, and then switching to the secondary boot device if the watchdog times out.
Without a way of determining a successful boot though, there's no point in the slice etc.
Right?
ghost
commented
8 years ago i also use BSDRP and I agree that the slice model does work for me, I can upgrade the routers, and they boot from another mount point, if i need to i can switch back to the previous
bb-mitch
commented
8 years ago Hey @outbackdingo
I think the piece that is / always was missing was the mechanism to make the multi-slice system handle a backed up config - the ability to both set your "check points" and to recover from a failed boot with something "simple" like a watchdog timer or other. Have you seen any elegant recovery mechanisms in other tools you have used?
The old pfsense slice modem had a shared config - which was bad IMHO - because if an upgrade had problems with a config, the failed upgrade would render the backup slice unbootable. Sounds like many of those issues have been corrected according to @fichtner - my goal is probably closer to the way you are using slices...
1) to be able to roll back even after a "success" 2) to be able to make a "point in time" backup and be able to choose to revert to that
Those were available before - but might have required manual selection of the boot slice at start time (unless the system was working and you could set the "boot X slice next time".
They seemed limited though because they still could require that person at the keyboard or serial console and the shared config could mean the entire system was screwed until you restore from backup.
I think if we started with the old system, but
We'd have a way (ignoring the potential for CF failure) to recover from a bricked device - even if the fault is with the user (config), or power fail, etc.
Add to that the smarts in the boot manager or BIOS to try a secondary device in the same failure cases noted above, and we'd have a way to handle CF failure too.
What do you think?
fichtner
commented
7 years ago Nano gave some users file system headaches and errors, I went ahead and changed the approach as previously discussed: https://github.com/opnsense/tools/commit/54fa4c1dc72
I don't have anything against the idea of a double slice. If there is work in this area I can imagine a "nano2" image which has two slices. We're not breaking compatibility by changing the current approach, visibly the outcome is the same, but better. For "nano2" we can use the same code as a base, just need to add a second partition in the script again and the appropriate code in core.git.
AdSchellevis
commented
7 years ago timeout due to inactivity
FreeNAS uses (ZFS) snapshots to allow for easy rollback of software updates:
It's possible to select an old release and "activate" it, which means the snapshot will be selected as new boot device. It does not just restore the old software versions but also the configuration. So you have the configuration which works for this software release.
It's also possible (and default) to keep several old versions with no major drawback. Of course, this would require an option to install OPNsense on ZFS.
I'd love to have this feature, because updates may always fail. It can be related to my configuration or my hardware or just a software bug. IMHO it would be a great advantage to go back to the working state with just a few clicks.