search

ubuntu / zsys

ZSys daemon and client for zfs systems
GNU General Public License v3.0
302 stars 43 forks source link

ZSys alters unmanaged datasets #203

Open lgosselin opened 3 years ago

lgosselin commented 3 years ago

Let's consider a machine that is storing backups from other machines within a dataset rpool/backups. When a new package is installed, for example, a new autozsys_xxx snapshot is created but it is also created in some unmanaged datasets like rpool/backups/machineX/rpool/USERDATA/username_uuid.

We should have no snapshot created in any unmanaged datasets because they are by definition not managed by ZSys. It is especially important since interfering with snapshots present in a structure maintained via "zfs send/receive" can cause some difficulties to receive incremental backups afterwards. Edit: I also had user account completely removed from rpool/backups/machineX/rpool/USERDATA. I guess this may be linked to #103 somehow.

All datasets outside the usual /BOOT/, /ROOT/, /USERDATA/ hierarchy managed closely by ZSys for the local server should not be tempered with it. This report is about a caveat that occurs when there is a ZSys-like structure present on the server at another location (typical for a backup server)

Installed versions:

lgosselin commented 3 years ago

I was experimenting between two VMs freshly installed and heavily inspired by OpenZFS - ZFS on Root for Ubuntu 20.04. I installed a first VM sender meant to be backuped. I cloned it and renamed the clone receiver. This is important since the problem might be exacerbated by the fact that there is a shared uuid here since the VMs have not yet diverged enough.

Upon installing a package on the receiver side, I realized that rpool/backups/sender/rpool/USERDATA/root_xxx had a new snapshot.

zsysctl output tends to suggest that at least some child datasets of rpool/backups are well persistent datasets but are considered for a special processing?

root@receiver:~# zsysctl show --full
Name:               rpool/ROOT/ubuntu_l2zibg
ZSys:               true
Last Used:          current
Last Booted Kernel: vmlinuz-5.8.0-48-generic
System Datasets:
 - bpool/BOOT/ubuntu_l2zibg
 - rpool/ROOT/ubuntu_l2zibg
 - rpool/ROOT/ubuntu_l2zibg/srv
 - rpool/ROOT/ubuntu_l2zibg/tmp
 - rpool/ROOT/ubuntu_l2zibg/usr
 - rpool/ROOT/ubuntu_l2zibg/var
 - rpool/ROOT/ubuntu_l2zibg/usr/local
 - rpool/ROOT/ubuntu_l2zibg/var/cache
 - rpool/ROOT/ubuntu_l2zibg/var/games
 - rpool/ROOT/ubuntu_l2zibg/var/lib
 - rpool/ROOT/ubuntu_l2zibg/var/log
 - rpool/ROOT/ubuntu_l2zibg/var/mail
 - rpool/ROOT/ubuntu_l2zibg/var/snap
 - rpool/ROOT/ubuntu_l2zibg/var/spool
 - rpool/ROOT/ubuntu_l2zibg/var/www
 - rpool/ROOT/ubuntu_l2zibg/var/lib/AccountsService
 - rpool/ROOT/ubuntu_l2zibg/var/lib/NetworkManager
 - rpool/ROOT/ubuntu_l2zibg/var/lib/apt
 - rpool/ROOT/ubuntu_l2zibg/var/lib/dpkg
User Datasets:
 User: root
 - rpool/backups/sender/rpool/USERDATA/root_l2zibg
Persistent Datasets:
                        - &{rpool/backups/sender %!!(MISSING)s(bool=false) {/backups/sender on %!!(MISSING)s(bool=true) %!!(MISSING)s(bool=false) %!!(MISSING)s(int=0)    {inherited     }} [%!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool false {none off false false 0    {local     }} [0xc000097a00 0xc00015f800 0xc0001be000 0xc0001be100 0xc0001be200 0xc0001be300 0xc0001be400] {0xc00014e360}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/bpool false {none off false false 0    {local     }} [0xc0001be600 0xc0001bec00 0xc0001bf900 0xc0001bfa00 0xc0001bfb00 0xc0001bfc00] {0xc0001c04e0}})] {%!!(MISSING)s(*zfs.Dataset=&{0x7f886c02b320 1 map[1:{1616707097 none} 6:{yes none} 13:{/backups/sender inherited} 27:{rpool/backups/sender none} 28:{on default}] []})}}
                        - &{rpool/backups/sender/bpool/BOOT/ubuntu_l2zibg %!!(MISSING)s(bool=false) {none on %!!(MISSING)s(bool=false) %!!(MISSING)s(bool=false) %!!(MISSING)s(int=1616742292)    {local   inherited  }} [%!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/bpool/BOOT/ubuntu_l2zibg@zrepl_20210326_094058_000 true {  false false 1616751658    {     }} [] {0xc0001c12f0}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/bpool/BOOT/ubuntu_l2zibg@zrepl_20210326_104057_000 true {  false false 1616755257    {     }} [] {0xc0001c1440}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/bpool/BOOT/ubuntu_l2zibg@zrepl_20210326_072058_000 true {  false false 1616743258    {     }} [] {0xc0001c1590}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/bpool/BOOT/ubuntu_l2zibg@zrepl_20210326_075102_000 true {  false false 1616745062    {     }} [] {0xc0001c16e0}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/bpool/BOOT/ubuntu_l2zibg@zrepl_20210326_085101_000 true {  false false 1616748661    {     }} [] {0xc0001c1830}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/bpool/BOOT/ubuntu_l2zibg@autozsys_hfhg76 true {  false false 1616743959    {     }} [] {0xc0001c1980}})] {%!!(MISSING)s(*zfs.Dataset=&{0x7f886c256820 1 map[1:{1616743320 none} 6:{no none} 13:{none local} 27:{rpool/backups/sender/bpool/BOOT/ubuntu_l2zibg none} 28:{on received}] []})}}
                        - &{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg %!!(MISSING)s(bool=false) {none on %!!(MISSING)s(bool=false) %!!(MISSING)s(bool=true) %!!(MISSING)s(int=1616742292) vmlinuz-5.8.0-48-generic   {local  inherited inherited inherited }} [%!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/usr false {none off false false 1616742292 vmlinuz-5.8.0-48-generic   {local  inherited inherited inherited }} [0xc00015fd00 0xc00016e400 0xc00016e500 0xc00016e600 0xc00016e700 0xc00016e800 0xc00016e900] {0xc000160ed0}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/srv false {none on false false 1616742292 vmlinuz-5.8.0-48-generic   {local  inherited inherited inherited }} [0xc00016ec00 0xc00016ed00 0xc00016ee00 0xc00016ef00 0xc00016f000 0xc00016f100] {0xc0001703f0}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/tmp false {none on false false 1616742292 vmlinuz-5.8.0-48-generic   {local  inherited inherited inherited }} [0xc00016f400 0xc00016f500 0xc00016f600 0xc00016f700 0xc00016f800 0xc00016f900] {0xc000170e10}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var false {none off false false 1616742292 vmlinuz-5.8.0-48-generic   {local  inherited inherited inherited }} [0xc00016fe00 0xc000182600 0xc000182e00 0xc000195d00 0xc0001a4500 0xc0001a4d00 0xc0001a5500 0xc0001a5d00 0xc0001b6400 0xc0001b6600 0xc0001b6800 0xc0001b6a00 0xc0001b6c00 0xc0001b6e00] {0xc000171830}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg@zrepl_20210326_104101_000 true {  false false 1616755261    {     }} [] {0xc0001bade0}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg@zrepl_20210326_072057_000 true {  false false 1616743257    {     }} [] {0xc0001baff0}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg@autozsys_hfhg76 true {  false false 1616743953    {     }} [] {0xc0001bb200}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg@zrepl_20210326_095059_000 true {  false false 1616752259    {     }} [] {0xc0001bb410}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg@zrepl_20210326_075102_000 true {  false false 1616745062    {     }} [] {0xc0001bb620}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg@zrepl_20210326_085058_000 true {  false false 1616748658    {     }} [] {0xc0001bb830}})] {%!!(MISSING)s(*zfs.Dataset=&{0x7f886c0862f0 1 map[1:{1616743323 none} 6:{no none} 13:{none local} 27:{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg none} 28:{on received}] []})}}
                        - &{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/srv %!!(MISSING)s(bool=false) {none on %!!(MISSING)s(bool=false) %!!(MISSING)s(bool=false) %!!(MISSING)s(int=1616742292) vmlinuz-5.8.0-48-generic   {local  inherited inherited inherited }} [%!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/srv@zrepl_20210326_075058_000 true {  false false 1616745058    {     }} [] {0xc0001706f0}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/srv@zrepl_20210326_085059_000 true {  false false 1616748659    {     }} [] {0xc000170840}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/srv@zrepl_20210326_104102_000 true {  false false 1616755262    {     }} [] {0xc000170990}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/srv@zrepl_20210326_095057_000 true {  false false 1616752257    {     }} [] {0xc000170ae0}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/srv@autozsys_hfhg76 true {  false false 1616743953    {     }} [] {0xc000170c30}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/srv@zrepl_20210326_072056_000 true {  false false 1616743256    {     }} [] {0xc000170d80}})] {%!!(MISSING)s(*zfs.Dataset=&{0x7f886c0a1270 1 map[1:{1616743360 none} 6:{no none} 13:{none local} 27:{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/srv none} 28:{on default}] []})}}
                        - &{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/tmp %!!(MISSING)s(bool=false) {none on %!!(MISSING)s(bool=false) %!!(MISSING)s(bool=false) %!!(MISSING)s(int=1616742292) vmlinuz-5.8.0-48-generic   {local  inherited inherited inherited }} [%!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/tmp@zrepl_20210326_104058_000 true {  false false 1616755258    {     }} [] {0xc0001710e0}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/tmp@zrepl_20210326_094101_000 true {  false false 1616751661    {     }} [] {0xc000171230}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/tmp@autozsys_hfhg76 true {  false false 1616743953    {     }} [] {0xc000171380}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/tmp@zrepl_20210326_075059_000 true {  false false 1616745059    {     }} [] {0xc0001714d0}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/tmp@zrepl_20210326_072057_000 true {  false false 1616743257    {     }} [] {0xc000171620}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/tmp@zrepl_20210326_085057_000 true {  false false 1616748657    {     }} [] {0xc000171770}})] {%!!(MISSING)s(*zfs.Dataset=&{0x7f886c09cc00 1 map[1:{1616743364 none} 6:{no none} 13:{none local} 27:{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/tmp none} 28:{on default}] []})}}
                        - &{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/usr/local %!!(MISSING)s(bool=false) {none on %!!(MISSING)s(bool=false) %!!(MISSING)s(bool=false) %!!(MISSING)s(int=1616742292) vmlinuz-5.8.0-48-generic   {local  inherited inherited inherited }} [%!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/usr/local@zrepl_20210326_072058_000 true {  false false 1616743258    {     }} [] {0xc000161500}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/usr/local@zrepl_20210326_074058_000 true {  false false 1616744458    {     }} [] {0xc000161650}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/usr/local@zrepl_20210326_095056_000 true {  false false 1616752256    {     }} [] {0xc0001617a0}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/usr/local@zrepl_20210326_085102_000 true {  false false 1616748662    {     }} [] {0xc0001618f0}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/usr/local@autozsys_hfhg76 true {  false false 1616743954    {     }} [] {0xc000161a40}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/usr/local@zrepl_20210326_104057_000 true {  false false 1616755257    {     }} [] {0xc000161b90}})] {%!!(MISSING)s(*zfs.Dataset=&{0x7f886c0bf390 1 map[1:{1616743380 none} 6:{no none} 13:{none local} 27:{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/usr/local none} 28:{on default}] []})}}
                        - &{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/cache %!!(MISSING)s(bool=false) {none on %!!(MISSING)s(bool=false) %!!(MISSING)s(bool=false) %!!(MISSING)s(int=1616742292) vmlinuz-5.8.0-48-generic   {local  inherited inherited inherited }} [%!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/cache@zrepl_20210326_075102_000 true {  false false 1616745062    {     }} [] {0xc0001a8360}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/cache@zrepl_20210326_072100_000 true {  false false 1616743260    {     }} [] {0xc0001a84b0}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/cache@autozsys_hfhg76 true {  false false 1616743954    {     }} [] {0xc0001a8600}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/cache@zrepl_20210326_094101_000 true {  false false 1616751661    {     }} [] {0xc0001a8750}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/cache@zrepl_20210326_085102_000 true {  false false 1616748662    {     }} [] {0xc0001a88a0}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/cache@zrepl_20210326_104100_000 true {  false false 1616755260    {     }} [] {0xc0001a89f0}})] {%!!(MISSING)s(*zfs.Dataset=&{0x7f886c0e81a0 1 map[1:{1616743376 none} 6:{no none} 13:{none local} 27:{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/cache none} 28:{on default}] []})}}
                        - &{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/games %!!(MISSING)s(bool=false) {none on %!!(MISSING)s(bool=false) %!!(MISSING)s(bool=false) %!!(MISSING)s(int=1616742292) vmlinuz-5.8.0-48-generic   {local  inherited inherited inherited }} [%!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/games@zrepl_20210326_075057_000 true {  false false 1616745057    {     }} [] {0xc000180090}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/games@zrepl_20210326_104057_000 true {  false false 1616755257    {     }} [] {0xc0001801e0}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/games@zrepl_20210326_084057_000 true {  false false 1616748058    {     }} [] {0xc000180330}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/games@zrepl_20210326_095056_000 true {  false false 1616752256    {     }} [] {0xc000180480}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/games@autozsys_hfhg76 true {  false false 1616743955    {     }} [] {0xc0001805d0}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/games@zrepl_20210326_072058_000 true {  false false 1616743258    {     }} [] {0xc000180720}})] {%!!(MISSING)s(*zfs.Dataset=&{0x7f886c1260a0 1 map[1:{1616743367 none} 6:{no none} 13:{none local} 27:{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/games none} 28:{on default}] []})}}
                        - &{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/lib %!!(MISSING)s(bool=false) {none on %!!(MISSING)s(bool=false) %!!(MISSING)s(bool=false) %!!(MISSING)s(int=1616742292) vmlinuz-5.8.0-48-generic   {local  inherited inherited inherited }} [%!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/lib/dpkg false {none on false false 1616742292 vmlinuz-5.8.0-48-generic   {local  inherited inherited inherited }} [0xc000183200 0xc000183300 0xc000183400 0xc000183500 0xc000183600 0xc000183700] {0xc0001816b0}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/lib/AccountsService false {none on false false 1616742292 vmlinuz-5.8.0-48-generic   {local  inherited inherited inherited }} [0xc000183a00 0xc000183b00 0xc000183c00 0xc000183d00 0xc000183e00 0xc000183f00] {0xc00018e0f0}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/lib/apt false {none on false false 1616742292 vmlinuz-5.8.0-48-generic   {local  inherited inherited inherited }} [0xc000194200 0xc000194300 0xc000194400 0xc000194500 0xc000194600 0xc000194700] {0xc00018eb10}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/lib/NetworkManager false {none on false false 1616742292 vmlinuz-5.8.0-48-generic   {local  inherited inherited inherited }} [0xc000194a00 0xc000194b00 0xc000194c00 0xc000194d00 0xc000194e00 0xc000194f00] {0xc00018f530}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/lib@zrepl_20210326_075103_000 true {  false false 1616745063    {     }} [] {0xc00018ff50}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/lib@zrepl_20210326_085057_000 true {  false false 1616748657    {     }} [] {0xc00019e180}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/lib@zrepl_20210326_095059_000 true {  false false 1616752259    {     }} [] {0xc00019e390}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/lib@zrepl_20210326_072056_000 true {  false false 1616743256    {     }} [] {0xc00019e5a0}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/lib@zrepl_20210326_104100_000 true {  false false 1616755260    {     }} [] {0xc00019e7b0}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/lib@autozsys_hfhg76 true {  false false 1616743955    {     }} [] {0xc00019e9c0}})] {%!!(MISSING)s(*zfs.Dataset=&{0x7f886c08bfb0 1 map[1:{1616743361 none} 6:{no none} 13:{none local} 27:{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/lib none} 28:{on default}] []})}}
                        - &{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/log %!!(MISSING)s(bool=false) {none on %!!(MISSING)s(bool=false) %!!(MISSING)s(bool=false) %!!(MISSING)s(int=1616742292) vmlinuz-5.8.0-48-generic   {local  inherited inherited inherited }} [%!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/log@zrepl_20210326_072100_000 true {  false false 1616743261    {     }} [] {0xc0001a9890}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/log@zrepl_20210326_095100_000 true {  false false 1616752260    {     }} [] {0xc0001a99b0}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/log@zrepl_20210326_084059_000 true {  false false 1616748059    {     }} [] {0xc0001a9ad0}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/log@zrepl_20210326_104058_000 true {  false false 1616755258    {     }} [] {0xc0001a9bf0}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/log@autozsys_hfhg76 true {  false false 1616743956    {     }} [] {0xc0001a9d10}})] {%!!(MISSING)s(*zfs.Dataset=&{0x7f886c122b00 1 map[1:{1616743383 none} 6:{no none} 13:{none local} 27:{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/log none} 28:{on default}] []})}}
                        - &{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/mail %!!(MISSING)s(bool=false) {none on %!!(MISSING)s(bool=false) %!!(MISSING)s(bool=false) %!!(MISSING)s(int=1616742292) vmlinuz-5.8.0-48-generic   {local  inherited inherited inherited }} [%!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/mail@autozsys_hfhg76 true {  false false 1616743956    {     }} [] {0xc00019ede0}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/mail@zrepl_20210326_104056_000 true {  false false 1616755256    {     }} [] {0xc00019ef30}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/mail@zrepl_20210326_075056_000 true {  false false 1616745056    {     }} [] {0xc00019f080}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/mail@zrepl_20210326_094100_000 true {  false false 1616751660    {     }} [] {0xc00019f1d0}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/mail@zrepl_20210326_072059_000 true {  false false 1616743259    {     }} [] {0xc00019f320}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/mail@zrepl_20210326_085058_000 true {  false false 1616748658    {     }} [] {0xc00019f470}})] {%!!(MISSING)s(*zfs.Dataset=&{0x7f886c103d30 1 map[1:{1616743369 none} 6:{no none} 13:{none local} 27:{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/mail none} 28:{on default}] []})}}
                        - &{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/snap %!!(MISSING)s(bool=false) {none on %!!(MISSING)s(bool=false) %!!(MISSING)s(bool=false) %!!(MISSING)s(int=1616742292) vmlinuz-5.8.0-48-generic   {local  inherited inherited inherited }} [%!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/snap@zrepl_20210326_072101_000 true {  false false 1616743261    {     }} [] {0xc0001a8e10}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/snap@autozsys_hfhg76 true {  false false 1616743956    {     }} [] {0xc0001a8f60}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/snap@zrepl_20210326_074059_000 true {  false false 1616744459    {     }} [] {0xc0001a90b0}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/snap@zrepl_20210326_084059_000 true {  false false 1616748059    {     }} [] {0xc0001a9200}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/snap@zrepl_20210326_104056_000 true {  false false 1616755256    {     }} [] {0xc0001a9350}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/snap@zrepl_20210326_094058_000 true {  false false 1616751658    {     }} [] {0xc0001a94a0}})] {%!!(MISSING)s(*zfs.Dataset=&{0x7f886c077af0 1 map[1:{1616743385 none} 6:{no none} 13:{none local} 27:{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/snap none} 28:{on default}] []})}}
                        - &{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/spool %!!(MISSING)s(bool=false) {none on %!!(MISSING)s(bool=false) %!!(MISSING)s(bool=false) %!!(MISSING)s(int=1616742292) vmlinuz-5.8.0-48-generic   {local  inherited inherited inherited }} [%!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/spool@zrepl_20210326_072057_000 true {  false false 1616743257    {     }} [] {0xc00019f890}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/spool@zrepl_20210326_085056_000 true {  false false 1616748656    {     }} [] {0xc00019f9e0}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/spool@autozsys_hfhg76 true {  false false 1616743957    {     }} [] {0xc00019fb30}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/spool@zrepl_20210326_075101_000 true {  false false 1616745061    {     }} [] {0xc00019fc80}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/spool@zrepl_20210326_104056_000 true {  false false 1616755256    {     }} [] {0xc00019fdd0}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/spool@zrepl_20210326_094100_000 true {  false false 1616751660    {     }} [] {0xc00019ff20}})] {%!!(MISSING)s(*zfs.Dataset=&{0x7f886c0814c0 1 map[1:{1616743362 none} 6:{no none} 13:{none local} 27:{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/spool none} 28:{on default}] []})}}
                        - &{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/www %!!(MISSING)s(bool=false) {none on %!!(MISSING)s(bool=false) %!!(MISSING)s(bool=false) %!!(MISSING)s(int=1616742292) vmlinuz-5.8.0-48-generic   {local  inherited inherited inherited }} [%!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/www@zrepl_20210326_085056_000 true {  false false 1616748656    {     }} [] {0xc000180b40}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/www@zrepl_20210326_072101_000 true {  false false 1616743261    {     }} [] {0xc000180c90}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/www@zrepl_20210326_104056_000 true {  false false 1616755256    {     }} [] {0xc000180de0}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/www@zrepl_20210326_094058_000 true {  false false 1616751658    {     }} [] {0xc000180f30}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/www@autozsys_hfhg76 true {  false false 1616743957    {     }} [] {0xc000181080}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/www@zrepl_20210326_075059_000 true {  false false 1616745060    {     }} [] {0xc0001811d0}})] {%!!(MISSING)s(*zfs.Dataset=&{0x7f886c054c30 1 map[1:{1616743384 none} 6:{no none} 13:{none local} 27:{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/www none} 28:{on default}] []})}}
                        - &{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/lib/AccountsService %!!(MISSING)s(bool=false) {none on %!!(MISSING)s(bool=false) %!!(MISSING)s(bool=false) %!!(MISSING)s(int=1616742292) vmlinuz-5.8.0-48-generic   {local  inherited inherited inherited }} [%!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/lib/AccountsService@zrepl_20210326_072058_000 true {  false false 1616743258    {     }} [] {0xc00018e3f0}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/lib/AccountsService@zrepl_20210326_095100_000 true {  false false 1616752260    {     }} [] {0xc00018e540}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/lib/AccountsService@autozsys_hfhg76 true {  false false 1616743957    {     }} [] {0xc00018e690}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/lib/AccountsService@zrepl_20210326_104100_000 true {  false false 1616755260    {     }} [] {0xc00018e7e0}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/lib/AccountsService@zrepl_20210326_084058_000 true {  false false 1616748058    {     }} [] {0xc00018e930}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/lib/AccountsService@zrepl_20210326_094057_000 true {  false false 1616751657    {     }} [] {0xc00018ea80}})] {%!!(MISSING)s(*zfs.Dataset=&{0x7f886c18f990 1 map[1:{1616743366 none} 6:{no none} 13:{none local} 27:{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/lib/AccountsService none} 28:{on default}] []})}}
                        - &{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/lib/NetworkManager %!!(MISSING)s(bool=false) {none on %!!(MISSING)s(bool=false) %!!(MISSING)s(bool=false) %!!(MISSING)s(int=1616742292) vmlinuz-5.8.0-48-generic   {local  inherited inherited inherited }} [%!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/lib/NetworkManager@zrepl_20210326_104057_000 true {  false false 1616755257    {     }} [] {0xc00018f7d0}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/lib/NetworkManager@zrepl_20210326_085103_000 true {  false false 1616748663    {     }} [] {0xc00018f920}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/lib/NetworkManager@autozsys_hfhg76 true {  false false 1616743958    {     }} [] {0xc00018fa70}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/lib/NetworkManager@zrepl_20210326_072058_000 true {  false false 1616743258    {     }} [] {0xc00018fbc0}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/lib/NetworkManager@zrepl_20210326_095101_000 true {  false false 1616752261    {     }} [] {0xc00018fd10}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/lib/NetworkManager@zrepl_20210326_074059_000 true {  false false 1616744459    {     }} [] {0xc00018fe60}})] {%!!(MISSING)s(*zfs.Dataset=&{0x7f886c181880 1 map[1:{1616743368 none} 6:{no none} 13:{none local} 27:{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/lib/NetworkManager none} 28:{on default}] []})}}
                        - &{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/lib/apt %!!(MISSING)s(bool=false) {none on %!!(MISSING)s(bool=false) %!!(MISSING)s(bool=false) %!!(MISSING)s(int=1616742292) vmlinuz-5.8.0-48-generic   {local  inherited inherited inherited }} [%!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/lib/apt@zrepl_20210326_095059_000 true {  false false 1616752259    {     }} [] {0xc00018ede0}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/lib/apt@zrepl_20210326_072100_000 true {  false false 1616743260    {     }} [] {0xc00018ef30}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/lib/apt@zrepl_20210326_094056_000 true {  false false 1616751656    {     }} [] {0xc00018f080}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/lib/apt@autozsys_hfhg76 true {  false false 1616743958    {     }} [] {0xc00018f1d0}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/lib/apt@zrepl_20210326_084059_000 true {  false false 1616748059    {     }} [] {0xc00018f320}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/lib/apt@zrepl_20210326_104100_000 true {  false false 1616755260    {     }} [] {0xc00018f470}})] {%!!(MISSING)s(*zfs.Dataset=&{0x7f886c0d3020 1 map[1:{1616743381 none} 6:{no none} 13:{none local} 27:{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/lib/apt none} 28:{on default}] []})}}
                        - &{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/lib/dpkg %!!(MISSING)s(bool=false) {none on %!!(MISSING)s(bool=false) %!!(MISSING)s(bool=false) %!!(MISSING)s(int=1616742292) vmlinuz-5.8.0-48-generic   {local  inherited inherited inherited }} [%!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/lib/dpkg@zrepl_20210326_075057_000 true {  false false 1616745058    {     }} [] {0xc0001819e0}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/lib/dpkg@zrepl_20210326_094059_000 true {  false false 1616751659    {     }} [] {0xc000181b30}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/lib/dpkg@autozsys_hfhg76 true {  false false 1616743959    {     }} [] {0xc000181c80}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/lib/dpkg@zrepl_20210326_104058_000 true {  false false 1616755258    {     }} [] {0xc000181dd0}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/lib/dpkg@zrepl_20210326_072059_000 true {  false false 1616743259    {     }} [] {0xc000181f20}}) %!!(MISSING)s(*zfs.Dataset=&{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/lib/dpkg@zrepl_20210326_085058_000 true {  false false 1616748658    {     }} [] {0xc00018e090}})] {%!!(MISSING)s(*zfs.Dataset=&{0x7f886c194090 1 map[1:{1616743372 none} 6:{no none} 13:{none local} 27:{rpool/backups/sender/rpool/ROOT/ubuntu_l2zibg/var/lib/dpkg none} 28:{on default}] []})}}
History:
  - Name:               rpool/ROOT/ubuntu_l2zibg@autozsys_uqgiju
    Created on:         2021-03-26 13:14:22
    Last Booted Kernel: vmlinuz-5.8.0-48-generic
    System Datasets:
     - bpool/BOOT/ubuntu_l2zibg@autozsys_uqgiju
     - rpool/ROOT/ubuntu_l2zibg@autozsys_uqgiju
     - rpool/ROOT/ubuntu_l2zibg/srv@autozsys_uqgiju
     - rpool/ROOT/ubuntu_l2zibg/tmp@autozsys_uqgiju
     - rpool/ROOT/ubuntu_l2zibg/usr@autozsys_uqgiju
     - rpool/ROOT/ubuntu_l2zibg/var@autozsys_uqgiju
     - rpool/ROOT/ubuntu_l2zibg/usr/local@autozsys_uqgiju
     - rpool/ROOT/ubuntu_l2zibg/var/cache@autozsys_uqgiju
     - rpool/ROOT/ubuntu_l2zibg/var/games@autozsys_uqgiju
     - rpool/ROOT/ubuntu_l2zibg/var/lib@autozsys_uqgiju
     - rpool/ROOT/ubuntu_l2zibg/var/log@autozsys_uqgiju
     - rpool/ROOT/ubuntu_l2zibg/var/mail@autozsys_uqgiju
     - rpool/ROOT/ubuntu_l2zibg/var/snap@autozsys_uqgiju
     - rpool/ROOT/ubuntu_l2zibg/var/spool@autozsys_uqgiju
     - rpool/ROOT/ubuntu_l2zibg/var/www@autozsys_uqgiju
     - rpool/ROOT/ubuntu_l2zibg/var/lib/AccountsService@autozsys_uqgiju
     - rpool/ROOT/ubuntu_l2zibg/var/lib/NetworkManager@autozsys_uqgiju
     - rpool/ROOT/ubuntu_l2zibg/var/lib/apt@autozsys_uqgiju
     - rpool/ROOT/ubuntu_l2zibg/var/lib/dpkg@autozsys_uqgiju
    User Datasets:
                        User: root
                        - rpool/backups/sender/rpool/USERDATA/root_l2zibg@autozsys_uqgiju
  - Name:               rpool/ROOT/ubuntu_l2zibg@autozsys_zvjs9e
    Created on:         2021-03-26 08:39:03
    Last Booted Kernel: vmlinuz-5.8.0-48-generic
    System Datasets:
                        - bpool/BOOT/ubuntu_l2zibg@autozsys_zvjs9e
                        - rpool/ROOT/ubuntu_l2zibg@autozsys_zvjs9e
                        - rpool/ROOT/ubuntu_l2zibg/srv@autozsys_zvjs9e
                        - rpool/ROOT/ubuntu_l2zibg/tmp@autozsys_zvjs9e
                        - rpool/ROOT/ubuntu_l2zibg/usr@autozsys_zvjs9e
                        - rpool/ROOT/ubuntu_l2zibg/var@autozsys_zvjs9e
                        - rpool/ROOT/ubuntu_l2zibg/usr/local@autozsys_zvjs9e
                        - rpool/ROOT/ubuntu_l2zibg/var/cache@autozsys_zvjs9e
                        - rpool/ROOT/ubuntu_l2zibg/var/games@autozsys_zvjs9e
                        - rpool/ROOT/ubuntu_l2zibg/var/lib@autozsys_zvjs9e
                        - rpool/ROOT/ubuntu_l2zibg/var/log@autozsys_zvjs9e
                        - rpool/ROOT/ubuntu_l2zibg/var/mail@autozsys_zvjs9e
                        - rpool/ROOT/ubuntu_l2zibg/var/snap@autozsys_zvjs9e
                        - rpool/ROOT/ubuntu_l2zibg/var/spool@autozsys_zvjs9e
                        - rpool/ROOT/ubuntu_l2zibg/var/www@autozsys_zvjs9e
                        - rpool/ROOT/ubuntu_l2zibg/var/lib/AccountsService@autozsys_zvjs9e
                        - rpool/ROOT/ubuntu_l2zibg/var/lib/NetworkManager@autozsys_zvjs9e
                        - rpool/ROOT/ubuntu_l2zibg/var/lib/apt@autozsys_zvjs9e
                        - rpool/ROOT/ubuntu_l2zibg/var/lib/dpkg@autozsys_zvjs9e
  - Name:               rpool/ROOT/ubuntu_l2zibg@autozsys_eav20h
    Created on:         2021-03-25 22:00:15
    Last Booted Kernel: vmlinuz-5.8.0-48-generic
    System Datasets:
     - bpool/BOOT/ubuntu_l2zibg@autozsys_eav20h
     - rpool/ROOT/ubuntu_l2zibg@autozsys_eav20h
     - rpool/ROOT/ubuntu_l2zibg/srv@autozsys_eav20h
     - rpool/ROOT/ubuntu_l2zibg/tmp@autozsys_eav20h
     - rpool/ROOT/ubuntu_l2zibg/usr@autozsys_eav20h
     - rpool/ROOT/ubuntu_l2zibg/var@autozsys_eav20h
     - rpool/ROOT/ubuntu_l2zibg/usr/local@autozsys_eav20h
     - rpool/ROOT/ubuntu_l2zibg/var/cache@autozsys_eav20h
     - rpool/ROOT/ubuntu_l2zibg/var/games@autozsys_eav20h
     - rpool/ROOT/ubuntu_l2zibg/var/lib@autozsys_eav20h
     - rpool/ROOT/ubuntu_l2zibg/var/log@autozsys_eav20h
     - rpool/ROOT/ubuntu_l2zibg/var/mail@autozsys_eav20h
     - rpool/ROOT/ubuntu_l2zibg/var/snap@autozsys_eav20h
     - rpool/ROOT/ubuntu_l2zibg/var/spool@autozsys_eav20h
     - rpool/ROOT/ubuntu_l2zibg/var/www@autozsys_eav20h
     - rpool/ROOT/ubuntu_l2zibg/var/lib/AccountsService@autozsys_eav20h
     - rpool/ROOT/ubuntu_l2zibg/var/lib/NetworkManager@autozsys_eav20h
     - rpool/ROOT/ubuntu_l2zibg/var/lib/apt@autozsys_eav20h
     - rpool/ROOT/ubuntu_l2zibg/var/lib/dpkg@autozsys_eav20h
    User Datasets:
     User: root
     - rpool/USERDATA/root_l2zibg@autozsys_eav20h
Users:
  - Name:    root
    History:
     - rpool/backups/sender/rpool/USERDATA/root_l2zibg@autozsys_qxj5td (2021-03-26 13:30:17): rpool/backups/sender/rpool/USERDATA/root_l2zibg@autozsys_qxj5td
     - rpool/backups/sender/rpool/USERDATA/root_l2zibg@autozsys_uqgiju (2021-03-26 13:14:23): rpool/backups/sender/rpool/USERDATA/root_l2zibg@autozsys_uqgiju
     - rpool/backups/sender/rpool/USERDATA/root_l2zibg@autozsys_wnzte3 (2021-03-26 12:29:17): rpool/backups/sender/rpool/USERDATA/root_l2zibg@autozsys_wnzte3
     - rpool/backups/sender/rpool/USERDATA/root_l2zibg@zrepl_20210326_104057_000 (2021-03-26 11:40:57): rpool/backups/sender/rpool/USERDATA/root_l2zibg@zrepl_20210326_104057_000
     - rpool/backups/sender/rpool/USERDATA/root_l2zibg@autozsys_dnhwqi (2021-03-26 11:10:18): rpool/backups/sender/rpool/USERDATA/root_l2zibg@autozsys_dnhwqi
     - rpool/backups/sender/rpool/USERDATA/root_l2zibg@zrepl_20210326_095058_000 (2021-03-26 10:50:58): rpool/backups/sender/rpool/USERDATA/root_l2zibg@zrepl_20210326_095058_000
     - rpool/backups/sender/rpool/USERDATA/root_l2zibg@autozsys_ys4osp (2021-03-26 10:09:18): rpool/backups/sender/rpool/USERDATA/root_l2zibg@autozsys_ys4osp
     - rpool/backups/sender/rpool/USERDATA/root_l2zibg@zrepl_20210326_085100_000 (2021-03-26 09:51:00): rpool/backups/sender/rpool/USERDATA/root_l2zibg@zrepl_20210326_085100_000
     - rpool/backups/sender/rpool/USERDATA/root_l2zibg@autozsys_8az0om (2021-03-26 09:08:18): rpool/backups/sender/rpool/USERDATA/root_l2zibg@autozsys_8az0om
     - rpool/backups/sender/rpool/USERDATA/root_l2zibg@autozsys_hfhg76 (2021-03-26 08:32:39): rpool/backups/sender/rpool/USERDATA/root_l2zibg@autozsys_hfhg76
     - rpool/backups/sender/rpool/USERDATA/root_l2zibg@zrepl_20210326_072057_000 (2021-03-26 08:20:57): rpool/backups/sender/rpool/USERDATA/root_l2zibg@zrepl_20210326_072057_000
     - rpool/USERDATA/root_l2zibg@autozsys_eav20h (2021-03-25 22:00:16): rpool/USERDATA/root_l2zibg@autozsys_eav20h

Important as well that during the zfs send, I'm pushing out the ZFS properties (zfs send -p). So if the selection of the ZSys datasets is based at least partially on ZFS properties, the fact that I use a clone is also to take into account.

lgosselin commented 3 years ago

I finally found a little time to experiment a little further. I'm still running with Ubuntu 20-04 LTS, but with zsysctl 0.4.8 & zsysd 0.4.8 (so upgraded compared to the initial post).

I have a machine2 which is a server installed with ZSys enabled. zfs-auto-snapshot is setup in cron to produce snapshot periodically. These snapshots are brought back to a backup server also running ZSys. The snapshots are transfered with a simple send/recv over ssh script in cron (roughly equivalent to zfs send -cRI from pool@to | zfs receive -dFu rpool/backup/machine2/pool).

On the backup server, I have :

## Backup dataset is a placeholder for backups coming from several machines
# zfs get canmount,mountpoint rpool/backup
NAME          PROPERTY    VALUE       SOURCE
rpool/backup  canmount    off         local
rpool/backup  mountpoint  /backup     inherited from rpool

##  A child dataset per machine
# zfs get canmount,mountpoint rpool/backup/machine2
NAME                PROPERTY    VALUE          SOURCE
rpool/backup/machine2 canmount    off            local
rpool/backup/machine2 mountpoint  /backup/machine2  inherited from rpool

## machine2 replicated pools and datasets, ZSys enabled
# zfs list -H -o name -d 2 rpool/backup/machine2
rpool/backup/machine2
rpool/backup/machine2/bpool
rpool/backup/machine2/bpool/BOOT
rpool/backup/machine2/rpool
rpool/backup/machine2/rpool/ROOT
rpool/backup/machine2/rpool/USERDATA
rpool/backup/machine2/rpool/data

## ZSys inspects `backup` because of the familiar structure
# zsysctl list
ID                                           ZSys  Last Used
--                                           ----  ---------
rpool/ROOT/ubuntu_8ok5om                     true  current
rpool/backup/machine2/rpool/ROOT/ubuntu_weigra  true  2021-03-25 12:30:14
# The last entry should not be considered. It is a state from another machine.

## A little more details
# zsysctl machine show --full
Name:               rpool/ROOT/ubuntu_8ok5om
ZSys:               true
Last Used:          current
Last Booted Kernel: vmlinuz-5.8.0-48-generic
System Datasets:
 - bpool/BOOT/ubuntu_8ok5om
 - rpool/ROOT/ubuntu_8ok5om
 - ...
...
User Datasets:
 User: root
 - rpool/USERDATA/root_8ok5om
...
Persistent Datasets:
 - rpool/backup/machine2/rpool/data
 - rpool/backup/machine2/rpool/data/dumps
...
# Only `data` and its children are flagged as persistent, here. It would help to have ROOT, BOOT, USERDATA as well.

## Grub scripts seems to also inspect unmanaged datasets
# update-grub
Sourcing file `/etc/default/grub'
Sourcing file `/etc/default/grub.d/init-select.cfg'
Generating grub configuration file ...
Found linux image: vmlinuz-5.8.0-50-generic in rpool/ROOT/ubuntu_8ok5om
Found initrd image: initrd.img-5.8.0-50-generic in rpool/ROOT/ubuntu_8ok5om
...
cannot open 'bpool/BOOT/ubuntu_weigra@autozsys_bqb173': dataset does not exist
Warning: didn't find any valid initrd or kernel.
cannot open 'bpool/BOOT/ubuntu_weigra@autozsys_4xq7jp': dataset does not exist
Warning: didn't find any valid initrd or kernel.
...
cannot open 'bpool/BOOT/ubuntu_weigra@zfs-auto-snap_daily-2021-05-10-1325': dataset does not exist
Warning: didn't find any valid initrd or kernel.
Adding boot menu entry for UEFI Firmware Settings
done
# These `ubuntu_weigra` entries are from `machine2` datasets located in `rpool/backup/machine2/bpool/BOOT/ubuntu_weigra`.
# Also non `autozsys` snapshots are considered?

I made some tests on another backup server. ZSys machine is also looking into datasets of older machines (16.04 upgraded to 18.04) not running ZSys (but are ZFS On Root with a dataset ROOT/ubuntu)

# zsysctl list
ID                                          ZSys   Last Used
--                                          ----   ---------
rpool/ROOT/ubuntu_mhz675                    true   current
# machine1 is Ubuntu 18.04, not running ZSys and has no zsys properties at all
rpool/backup/machine1/rpool/ROOT/ubuntu         false  0001-01-01 00:00:00
# machine 4 is a ZSys enabled Ubuntu 20.04
rpool/backup/machine4/rpool/ROOT/ubuntu_1ml921  true   2021-03-22 10:46:04

From the details gathered so far

  1. Why is rpool/backup and all its children not considered as unmanaged datasets? (only some children datasets). Is there a combination of ZFS properties that could instruct ZSys to stop introspecting them?
  2. Every day I have observed that all children datasets of rpool/backup/machine2/rpool/USERDATA are removed along with their snapshots. (Is this the GC getting out of its way? Linked to #103?). That makes any incremental send/receive fail afterwards and needs repairing.
  3. Still related to not picking up state from other machine, I don't expect the GRUB installed on the backup server to consider anything else than its own state/snapshots for building the menu.

From some extra trials and errors, it looks like forcing ZFS mountpoint properties to none on the datasets may help. At least the output of zsysctl list does no longer displaying a state, but I still need to figure out what it means for the whole replication process and the implication in case of full restore.