Need a command to manipulate the zsys database

Popyllol commented 4 years ago

Describe the bug No method to edit the zsys database found

To Reproduce Steps to reproduce the behavior:

Create a user
```
# useradd -m -s /bin/bash testuser
```

Get the dataset name of users home directory

# df -h ~testuser
Filesystem                       Size  Used Avail Use% Mounted on
rpool/USERDATA/testuser_bdqp54  356G  128K  356G   1% /home/testuser

Rename the zfs dataaset

# zfs rename rpool/USERDATA/testuser_bdqp54 rpool/USERDATA/testuser2_bdqp54

4.Check the dataset name

# df -h ~testuser
Filesystem                       Size  Used Avail Use% Mounted on
rpool/USERDATA/testuser2_bdqp54  356G  128K  356G   1% /home/testuser

Dump what zsys thinks about the user

# zsysctl service dump
...
"Users": {
...
        "testuser2": {
           "ID": "rpool/USERDATA/testuser2_bdqp54",
           "LastUsed": "2020-04-21T15:28:05+02:00",
           "Datasets": {
              "rpool/USERDATA/testuser2_bdqp54": [
                 {
                    "Name": "rpool/USERDATA/testuser2_bdqp54",
                    "Mountpoint": "/home/testuser",
                    "CanMount": "on",
                    "Mounted": true,
                    "LastUsed": 1587475685,
                    "BootfsDatasets": "rpool/ROOT/ubuntu_pu2ihh"
                 }
              ]
           }
        }
...

Switch to the user
```
# sudo -iu testuser
```

Take a snapshot

$ zsysctl save
ERROR couldn't save state for user "testuser":user "testuser" doesn't exist

Expected behavior

$ zsysctl save
Successfully saved as "autozsys_xyz"

I think the problem is here that the name of the user is taken from the dataset name as the changed username show in the output from zsysctl service dump above.

What I would love to have is a key in the database that really matches the user and not a part of the dataset name. And further any subcommand to correct settings in that database.

Installed versions:

OS: (/etc/os-release)

NAME="Ubuntu"
VERSION="20.04 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

Zsysd running version: (zsysctl version output)
```
zsysctl 0.4.5
zsysd   0.4.5
```

didrocks commented 4 years ago

Thanks for the bug report!

We are trying really hard in zsys to avoid having a database. All properties are taken from the ZFS datasets properties itself, with some user properties.

It’s true that right now, we don’t integrate at all with user renames, but in any case, I don’t think we should support manual zfs rename out of the box. I think:

either the user has a clue about the internals (we will blog about it soon) and can apply any needed transformation himself (which is basically what you want by "manipulating the database")
the user doesn’t want to use zsys and have a way to opt-out
we integrate with usermod -l to track user rename. It means that we will have to set a property in some way on the datasets.

I would tend to transform this bug as "provide a way to rename users" and keep it tracked. It will certainly greatly complexifies the code for an edge case, but I think we can achieve it. But it needs design: what to do with older clones and such, what if the user state is switched between 2 systems (one with a rename, one without rename)? and such and such… What do you think? Then, the internal zfs properties that will be used would be

Popyllol commented 4 years ago

Hey didrocks, thank you for your fast reply!

To read all from the ZFS is better than having a parallel universe in a database!

So what about another ZFS property which is holding the username for a dataset? I know there might be problems of uniqueness, but to derive it from the dataset name seems to hold more possible problems for the future in my opinion.

What do you think about using more ZFS properties for zsys?

Greetings, Lars

didrocks commented 4 years ago

So what about another ZFS property which is holding the username for a dataset? I know there might be problems of uniqueness, but to derive it from the dataset name seems to hold more possible problems for the future in my opinion.

What do you think about using more ZFS properties for zsys?

The issues there is that it needs careful design:

you can rename an user:
- old state save should keep the older user, new state save should have the new user
- if you revert without reverting user dataset, then 2 states share the same user datasets. It means it needs to track the 2 names and apply the correct name to each user state (this has an impact on uniqueness of internal name in our current model)
you can attach this user datasets to multiples machines having different user names (a bit similar to 2 states in the history associated with it: one before, one after)
what if an older state or 2 machines have the 2 users attached (the nameA and nameB) and what you have done is deleting nameA, then renaming nameB to nameA? We need to keep the 2 history of datasets still separated when looking at them, and introspecting/reading each state /etc/passwd can introduce huge slowdown as it means mounting and inspecting for each available system states, for all machines!

This is indeed complex and a known issue. If I’m correct, the only impact (that we anticipated) is only in zsysctl show output, correct? There is nothing changing in term of mounting and such (usermod -d -m to change user home directory path is supported). Only ofc sysctl save will fail for a given user until you rename the dataset (but you zfs rename somewhat of purpose to broke it, right? ;) I mean, it’s not something that will happen under people’s feet contrary to use a tool to rename an user which ends up with the same effect)

If you confirm that there is no other side-effect, it seems minor for now and can be workarounded with zfs rename every impacted datasets (you should do the same ofc if you revert). Maybe this is something we can put on a WIKI? Interested to contribute there?

I would to not put that one in the quick fix list as you can see the impact can be tremendous and dangerous for an edge case which can be manually workarounded. Then, once we refine the model and have answers for all the above case, this is something we should fix, for sure. Does it make sense?

mcdnl90 commented 4 years ago

What about using user IDs instead of user names? That shouldn't change on account rename.

didrocks commented 4 years ago

The issue with ID is that some people will still want to use different systems with them and want explicit dataset names. We can use a combination of ID + zsys metadata on the dataset but that would need more thought. I think any contribution on the WIKI in the meantime is the best approach before we go on the wrong path which will be harder to revert :)

mskarbek commented 4 years ago

And here it comes systemd-userdb and systemd-homed since you "are trying really hard in zsys to avoid having a database" ;) You don't have to add another user database, you already have one in 20.04 in systemd ready to use.

ddnexus commented 3 years ago

I have a possibly simpler but similar problem:

I installed a new system and I realized that I missed the overridden of the user name in the install form, so I changed the user and the directory names to the right one. (domizio is the wrong one, and dd is the right one)

Now the user dataset contains the wrong name but it is pointing to the right directory. In that condition it cannot save the state.

# zsysctl service dump
...
         "Users": {
            "domizio": {
               "ID": "rpool/USERDATA/domizio_hbdxnx",
               "LastUsed": "2021-05-31T07:31:14+07:00",
               "Datasets": {
                  "rpool/USERDATA/domizio_hbdxnx": [
                     {
                        "Name": "rpool/USERDATA/domizio_hbdxnx",
                        "Mountpoint": "/home/dd",
                        "CanMount": "on",
                        "Mounted": true,
                        "LastUsed": 1622421074,
                        "BootfsDatasets": "rpool/ROOT/ubuntu_9k5zqb"
                     }
                  ]
               }
            },
...

I don' t want to break anything making the system unusable, so I better ask to somebody that know better (e.g. @didrocks, @Popyllol, ...):

Could I fix it simply renaming the dataset? like:

zfs rename rpool/USERDATA/domizio_hbdxnx rpool/USERDATA/dd_hbdxnx

Should I do something else? (Notice, it' s a very new system, so I don' t have any history to preserve, hence I would have no problem to nuke the old zsys history just to fix it)

Any help is very appreciated. Thanks!

didrocks commented 3 years ago

Now the user dataset contains the wrong name but it is pointing to the right directory. In that condition it cannot save the state.

Are you sure it doesn’t save state? It should (using the mountpoint to know which user has a matching set), or this is a bug (can you open one after double checking?)

Could I fix it simply renaming the dataset? like: zfs rename rpool/USERDATA/domizio_hbdxnx rpool/USERDATA/dd_hbdxnx

Yes, if you hit this bug, this could be fixed that way. Nothing else is needed :)

Just then stop the daemon, and restart it (or reboot)

ddnexus commented 3 years ago

Thank you @didrocks

Yes, I am sure it does not save the user state because it is repeated in the log many times:

Failed to start Save current user state periodically.

besides it gives an explicit error when I try to do it manually:

zsysctl save                        
ERROR couldn't save state for user "dd": user "dd" doesn't exist

However something else also saves it, probably while it saves the system state, because there are saved snapshots for the old wrong user (which have the same timestamp and id for ROOT, and USERDATA user and root):

zsysctl show
Name:           rpool/ROOT/ubuntu_9k5zqb
ZSys:           true
Last Used:      current
History:        
  - Name:       rpool/ROOT/ubuntu_9k5zqb@autozsys_i5h4ih
    Created on: 2021-06-02 18:52:18
  - Name:       rpool/ROOT/ubuntu_9k5zqb@autozsys_5l6zl2
    Created on: 2021-06-02 18:52:18
  - Name:       rpool/ROOT/ubuntu_9k5zqb@autozsys_myvrlo
    Created on: 2021-06-02 18:47:17
  - Name:       rpool/ROOT/ubuntu_9k5zqb@autozsys_czyjpg
    Created on: 2021-06-02 18:47:17
  - Name:       rpool/ROOT/ubuntu_9k5zqb@autozsys_7nxojz
    Created on: 2021-05-31 09:55:43
  - Name:       rpool/ROOT/ubuntu_9k5zqb@autozsys_ugicug
    Created on: 2021-05-30 08:50:49
  - Name:       rpool/ROOT/ubuntu_9k5zqb@autozsys_dvo91v
    Created on: 2021-05-27 19:39:18
  - Name:       rpool/ROOT/ubuntu_9k5zqb@autozsys_gjmeil
    Created on: 2021-05-27 19:23:33
  - Name:       rpool/ROOT/ubuntu_9k5zqb@autozsys_ovnkmk
    Created on: 2021-05-27 19:19:23
  - Name:       rpool/ROOT/ubuntu_9k5zqb@autozsys_zjw6zm
    Created on: 2021-05-27 18:24:36
  - Name:       rpool/ROOT/ubuntu_9k5zqb@autozsys_8q9kg1
    Created on: 2021-05-27 18:24:35
  - Name:       rpool/ROOT/ubuntu_9k5zqb@autozsys_8ge6gu
    Created on: 2021-05-27 16:35:42
  - Name:       rpool/ROOT/ubuntu_9k5zqb@autozsys_kxuuj1
    Created on: 2021-05-27 16:28:36
  - Name:       rpool/ROOT/ubuntu_9k5zqb@autozsys_2dxe5s
    Created on: 2021-05-27 16:05:30
  - Name:       rpool/ROOT/ubuntu_9k5zqb@autozsys_qb1s4q
    Created on: 2021-05-27 15:55:20
  - Name:       rpool/ROOT/ubuntu_9k5zqb@autozsys_zan1li
    Created on: 2021-05-27 14:49:28
  - Name:       rpool/ROOT/ubuntu_9k5zqb@autozsys_roxla1
    Created on: 2021-05-27 13:08:31
  - Name:       rpool/ROOT/ubuntu_9k5zqb@autozsys_49hqr1
    Created on: 2021-05-27 12:38:39
  - Name:       rpool/ROOT/ubuntu_9k5zqb@autozsys_e78npk
    Created on: 2021-05-27 10:43:35
  - Name:       rpool/ROOT/ubuntu_9k5zqb@autozsys_551oir
    Created on: 2021-05-27 10:43:35
  - Name:       rpool/ROOT/ubuntu_9k5zqb@autozsys_jayx6t
    Created on: 2021-05-26 20:33:43
Users:
  - Name:    domizio
    History: 
     - rpool/USERDATA/domizio_hbdxnx@autozsys_5l6zl2 (2021-06-02 18:52:19)
     - rpool/USERDATA/domizio_hbdxnx@autozsys_i5h4ih (2021-06-02 18:52:18)
     - rpool/USERDATA/domizio_hbdxnx@autozsys_myvrlo (2021-06-02 18:47:17)
     - rpool/USERDATA/domizio_hbdxnx@autozsys_czyjpg (2021-06-02 18:47:17)
     - rpool/USERDATA/domizio_hbdxnx@autozsys_7nxojz (2021-05-31 09:55:43)
     - rpool/USERDATA/domizio_hbdxnx@autozsys_ugicug (2021-05-30 08:50:49)
     - rpool/USERDATA/domizio_hbdxnx@autozsys_dvo91v (2021-05-27 19:39:18)
     - rpool/USERDATA/domizio_hbdxnx@autozsys_gjmeil (2021-05-27 19:23:33)
     - rpool/USERDATA/domizio_hbdxnx@autozsys_ovnkmk (2021-05-27 19:19:23)
     - rpool/USERDATA/domizio_hbdxnx@autozsys_zjw6zm (2021-05-27 18:24:36)
     - rpool/USERDATA/domizio_hbdxnx@autozsys_8q9kg1 (2021-05-27 18:24:36)
     - rpool/USERDATA/domizio_hbdxnx@autozsys_8ge6gu (2021-05-27 16:35:42)
     - rpool/USERDATA/domizio_hbdxnx@autozsys_kxuuj1 (2021-05-27 16:28:37)
     - rpool/USERDATA/domizio_hbdxnx@autozsys_2dxe5s (2021-05-27 16:05:30)
     - rpool/USERDATA/domizio_hbdxnx@autozsys_qb1s4q (2021-05-27 15:55:20)
     - rpool/USERDATA/domizio_hbdxnx@autozsys_zan1li (2021-05-27 14:49:29)
     - rpool/USERDATA/domizio_hbdxnx@autozsys_roxla1 (2021-05-27 13:08:31)
     - rpool/USERDATA/domizio_hbdxnx@autozsys_49hqr1 (2021-05-27 12:38:39)
     - rpool/USERDATA/domizio_hbdxnx@autozsys_e78npk (2021-05-27 10:43:35)
     - rpool/USERDATA/domizio_hbdxnx@autozsys_551oir (2021-05-27 10:43:35)
     - rpool/USERDATA/domizio_hbdxnx@autozsys_jayx6t (2021-05-26 20:33:44)
  - Name:    root
    History: 
     - rpool/USERDATA/root_hbdxnx@autozsys_5l6zl2 (2021-06-02 18:52:19)
     - rpool/USERDATA/root_hbdxnx@autozsys_i5h4ih (2021-06-02 18:52:18)
     - rpool/USERDATA/root_hbdxnx@autozsys_myvrlo (2021-06-02 18:47:17)
     - rpool/USERDATA/root_hbdxnx@autozsys_czyjpg (2021-06-02 18:47:17)
     - rpool/USERDATA/root_hbdxnx@autozsys_7nxojz (2021-05-31 09:55:43)
     - rpool/USERDATA/root_hbdxnx@autozsys_ugicug (2021-05-30 08:50:49)
     - rpool/USERDATA/root_hbdxnx@autozsys_dvo91v (2021-05-27 19:39:18)
     - rpool/USERDATA/root_hbdxnx@autozsys_gjmeil (2021-05-27 19:23:33)
     - rpool/USERDATA/root_hbdxnx@autozsys_ovnkmk (2021-05-27 19:19:23)
     - rpool/USERDATA/root_hbdxnx@autozsys_zjw6zm (2021-05-27 18:24:36)
     - rpool/USERDATA/root_hbdxnx@autozsys_8q9kg1 (2021-05-27 18:24:36)
     - rpool/USERDATA/root_hbdxnx@autozsys_8ge6gu (2021-05-27 16:35:42)
     - rpool/USERDATA/root_hbdxnx@autozsys_kxuuj1 (2021-05-27 16:28:37)
     - rpool/USERDATA/root_hbdxnx@autozsys_2dxe5s (2021-05-27 16:05:30)
     - rpool/USERDATA/root_hbdxnx@autozsys_qb1s4q (2021-05-27 15:55:20)
     - rpool/USERDATA/root_hbdxnx@autozsys_zan1li (2021-05-27 14:49:29)
     - rpool/USERDATA/root_hbdxnx@autozsys_roxla1 (2021-05-27 13:08:31)
     - rpool/USERDATA/root_hbdxnx@autozsys_49hqr1 (2021-05-27 12:38:39)
     - rpool/USERDATA/root_hbdxnx@autozsys_e78npk (2021-05-27 10:43:35)
     - rpool/USERDATA/root_hbdxnx@autozsys_551oir (2021-05-27 10:43:35)
     - rpool/USERDATA/root_hbdxnx@autozsys_jayx6t (2021-05-26 20:33:44)

So it looks like different triggers may hit or avoid the problem. And anyway, according to what I understood, it should save a as dd_* instead it saves it as domizio_*.

If you notice, there are also double snapshots taken at the same exact time. It's a long time that I observed it in another system, and I was wondering why. It looks like the trigger hits twice. I notice that while installing packages in synaptic, but sometime synaptic takes just one snapshot, so I have no idea why it happens. Is that known/normal?

can you open one after double checking?

What do you mean? Open with what?

ddnexus commented 3 years ago

Just updated the system with the Software Updater and it triggered 2 snapshots, so it' s definitely not synaptic

didrocks commented 3 years ago

Ok, so system state save is working, not individual user ones. This is where a bug will be useful (if the name doesn’t match).

I have already seen semdomly 2 snapshots being taken at the same time (like if the apt hook was racy). Something else to raise as an issue so that we can have a look?

Thanks a lot, at least your immediate problem should be solved now :)

ddnexus commented 3 years ago

I have a few other things to report (and potentially dangerous) but I will post that in a separate issue ASAP, however nothing related to the current bug.

Thanks a lot, at least your immediate problem should be solved now :)

Thanks a lot to you, very responsive as usual! However I have some problem renaming the dataset. I stop the service with zsysctl service stop. Then:

zfs rename rpool/USERDATA/domizio_hbdxnx rpool/USERDATA/dd_hbdxnx
cannot unmount '/home/dd': unmount failed

I even tried the same logging in as root, just in case it was some problem being the dd user, but no avail. It keeps complaining about the failed unmount. I tried to zfs unmount /home/dd but failed as well.

Not sure whether that could be caused by 3 persistent datasets under home/dd. I created them with:

zfs create -o canmount=off rpool/home
zfs create -o canmount=off rpool/home/dd
zfs create rpool/home/dd/Downloads
zfs create rpool/home/dd/repos
zfs create rpool/home/dd/tmp

Now I am not sure that is correct. It works, but I have doubts about the second statement. Is that OK?

And... how could I finally rename the dataset?

Thanks

didrocks commented 3 years ago

Thanks a lot to you, very responsive as usual! yw!

On the rename issue, I think zfs rename has a force option or something along the line. The issue is probably due to the fact you are logged in as this user (even if you sudo afterwards, you still have some processes running under this user). So:

either you have a secondary user (who is a sudoer) and you don’t log in or start a graphical session with original dd user.
you can start a live session and zfs rename on your (unused disk).

Or if this is not the case, try unmounting the subdatasets first (but I thought zfs would handle that for you)

ddnexus commented 3 years ago

The issue is probably due to the fact you are logged in as this user (even if you sudo afterwards, you still have some processes running under this user)

I thought the same as well, so I already logged out as dd, then I opened a console with ctrl-alt-f4 logged in as root. Login: root ...

That didn't change a thing. the logout should close all the processes, but I will try to reboot instead of logout.

Then if it does not work, a live session would fix the problem... however I should first load zfs and all the suff in the live disk... complicated stuff :). Thank you again, I will let you know.

ddnexus commented 3 years ago

OK, just learned something.

The persistent datasets ARE relevant: without unmounting them you cannot proceed. Here is what I did.

I rebooted and logged in a console as dd, enabled root login, exit, logged in as root, and then I had to zfs unmount ... all the persistent datasets under the /home/dd/* mountpoint first, then I could unmount /home/dd, and the rename finally worked. No need to use -f.

Thank you again for your time!

ddnexus commented 3 years ago

I wanted to try whether the bug is triggered by the persistent datasets, and try other commands but I don't want to mess up my working system for that. Is there any recommended sand-boxed environment for reproducing bugs or trying zsys? Thanks

didrocks commented 3 years ago

As you have some kernel components in it and the mock doesn‘t reproduce the exact issue, I would suggest using a VM to find a good reproducer. That will be easier for us as well then to debug :)

ubuntu / zsys

Need a command to manipulate the zsys database #100