Consolidation broken - Githubissues

ghost commented 3 years ago

Console output:

btrfs subvolume snapshot: exactly 2 arguments expected, 1 given
btrfs subvolume show: exactly 1 argument expected, 0 given
Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/buttermanager/buttermanager/window/windows.py", line 275, in consolidate
    commandline_output = subprocess.check_output(command, shell=True)
  File "/usr/lib/python3.9/subprocess.py", line 424, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['sudo btrfs subvolume show ']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/buttermanager/buttermanager/window/windows.py", line 278, in consolidate
    str(called_process_error_exception.reason))
AttributeError: 'CalledProcessError' object has no attribute 'reason'
Aborted

Relevant log:


2021-09-23 17:11:16,835 INFO:ConfigManager. Checking OS. ARCH found
2021-09-23 17:11:16,835 INFO:ConfigManager. Checking Desktop Environment. kde found
2021-09-23 17:11:16,835 INFO:ConfigManager. Installation type: native
2021-09-23 17:11:16,835 INFO:ConfigManager. Creating PropertiesManager...
2021-09-23 17:11:16,836 INFO:ConfigManager. Migration process has finished successfully!
2021-09-23 17:11:16,836 INFO:ConfigManager. Retrieving user's configuration from buttermanager.yaml file and loading it in memory...
2021-09-23 17:11:16,836 INFO:VersionChecker. Checking for a new version of ButterManager. Please wait...
2021-09-23 17:11:16,852 INFO:PasswordWindow. Calculating appropriated base font size for UI elements...
2021-09-23 17:11:16,853 INFO:PasswordWindow. Base font size = 10
2021-09-23 17:11:17,064 INFO:VersionChecker. Last version is 2.4.2 and current version is 2.4.2
2021-09-23 17:11:20,571 INFO:RootSnapshotChecker. Checking if the current snapshot used for root is the default. Please wait...
2021-09-23 17:11:20,783 INFO:ButtermanagerMainWindow. BTRFS filesystems found in the system:
2021-09-23 17:11:20,783 INFO:ButtermanagerMainWindow. BTRFS Filesystem -> UUID: 28a88c3b-5018-40f5-b628-70a519155731; Devices: ['/dev/nvme1n1p2']; Mounted Points: ['/', '/home', '/mnt/defvol']
2021-09-23 17:11:23,095 INFO:ConsolidateSnapshotWindow. Consolidating default root snapshot. The system has booted in /mnt/defvol/_snapshots/root-20210923-0 and it will be consolidated into 
2021-09-23 17:11:23,095 INFO:Subvolume. Deleting subvolume from origin /. Please wait...
2021-09-23 17:11:23,095 INFO:Subvolume. Deleting subvolume from origin /. Please wait...
2021-09-23 17:11:23,103 INFO:Subvolume. Snapshot / deleted.

2021-09-23 17:11:23,103 INFO:Subvolume. Snapshot / deleted.

btrfs-list:

NAME                                 TYPE     EXCL  MOUNTPOINT
28a88c3b                               fs  137.47G (single, 788.23G free)
   [main]                         mainvol       -  /
   _active                         subvol       -  
   _active/home                    subvol       -  /home
      _snapshots/home.20210919_1   rosnap       -  
      _snapshots/home.20210923     rosnap       -  
      _snapshots/home-20210923-0   rosnap       -  
   _snapshots                      subvol       -  
   _active/root                    subvol       -  
_active/root.BROKEN                  snap       -  
_snapshots/root-20210923-0           snap       -  /
_snapshots/root.20210923           rosnap       -  
_snapshots/root.20210919_1         rosnap       -

It is probably because I somehow fucked my filesystem while trying to use btrbk and their restore guide. But I don't really know so I wanted to share the broken case if you want to add some safety checks.

egara commented 3 years ago

Hi @Deckweiss and thank you for the report.

Today, I've been testing the consolidation process in my own machine and it worked flawlessly. This is the log during the operation:

2021-09-24 08:51:57,438 INFO:ConfigManager. Checking OS. ARCH found
2021-09-24 08:51:57,444 INFO:ConfigManager. Checking Desktop Environment. generic found
2021-09-24 08:51:57,445 INFO:ConfigManager. Installation type: native
2021-09-24 08:51:57,445 INFO:ConfigManager. Creating PropertiesManager...
2021-09-24 08:51:57,446 INFO:ConfigManager. Migration process has finished successfully!
2021-09-24 08:51:57,447 INFO:ConfigManager. Retrieving user's configuration from buttermanager.yaml file and loading it in memory...
2021-09-24 08:51:57,447 INFO:VersionChecker. Checking for a new version of ButterManager. Please wait...
2021-09-24 08:51:57,500 INFO:PasswordWindow. Calculating appropriated base font size for UI elements...
2021-09-24 08:51:57,501 INFO:PasswordWindow. Base font size = 10
2021-09-24 08:51:57,620 INFO:VersionChecker. Last version is 2.4.2 and current version is 2.4.2
2021-09-24 08:51:59,682 INFO:RootSnapshotChecker. Checking if the current snapshot used for root is the default. Please wait...
2021-09-24 08:51:59,886 INFO:ButtermanagerMainWindow. BTRFS filesystems found in the system:
2021-09-24 08:51:59,886 INFO:ButtermanagerMainWindow. BTRFS Filesystem -> UUID: bbad1b9c-49dc-4405-9c90-b034005e673b; Devices: ['/dev/sda2']; Mounted Points: ['/', '/mnt/defvol', '/home']
2021-09-24 08:52:04,350 INFO:ConsolidateSnapshotWindow. Consolidating default root snapshot. The system has booted in /mnt/defvol/snapshots/root-20210924-0 and it will be consolidated into /mnt/defvol/@

There is something that caught my attention comparing my log with yours and it is the line

2021-09-24 08:52:04,350 INFO:ConsolidateSnapshotWindow. Consolidating default root snapshot. The system has booted in /mnt/defvol/snapshots/root-20210924-0 and it will be consolidated into /mnt/defvol/@

As you can see, the destination is /mnt/defvol/@ but there is no destination in your log. This is my configuration file:

aur_repository: 0
check_at_startup: 1
grub_btrfs: 1
path_to_consolidate_root_snapshot: '@'
save_log: 1
snap_packages: 1
subvolumes_dest: /mnt/defvol/snapshots/|/mnt/defvol/snapshots/
subvolumes_orig: /mnt/defvol/@/|/mnt/defvol/@home/
subvolumes_prefix: root|home
subvolumes_snapshots_to_keep: 2|1

Could you share yours? I suspect there is no value for path_to_consolidate_root_snapshot parameter.

Thanks! :)

ghost commented 3 years ago

These are the settings

aur_repository: 0
check_at_startup: 0
grub_btrfs: 1
path_to_consolidate_root_snapshot: _active/root
save_log: 1
snap_packages: 0
subvolumes_dest: /mnt/defvol/_snapshots/|/mnt/defvol/_snapshots/
subvolumes_orig: /|/home/
subvolumes_prefix: root|home
subvolumes_snapshots_to_keep: 3|3

I did a fresh install now and it works, I have kept the same setting file and tried booting into a snapshot.

I feel like @ might be a fallback value if the buttermanager doesn't find the configured one.

It is probably my bad for trying to restore a backup manually by following https://github.com/digint/btrbk#restoring-backups And something in that guide is not compatible. The difference seems to be that after restoring with buttermanager, the / is still a subvolume, while with btrbk guide it became a snap

egara commented 3 years ago

Hi @Deckweiss

Thanks again for the feedback and I'm glad that on a fresh installation everything works fine.

Yes, it's totally possible that if you were tinkering with several BTRFS tools, something could cause collateral effects.

Again, thanks for your interest in ButterManager :)

ghost commented 2 years ago

Hi @egara

Today, I had a bug after updating the system, so I wanted to restore a previous snapshot and I have the same error as in this issue. This time, I strictly followed your guide and did not fuck around with the subvolumes. I also restored a snapshot at some point in time on this very setup and it worked fine, but now it is broken for some reason.

commandline output:

deckweiss@BIG-Chungus:~$ buttermanager
[sudo] password for deckweiss: btrfs subvolume snapshot: exactly 2 arguments expected, 1 given
btrfs subvolume show: exactly 1 argument expected, 0 given
Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/buttermanager/buttermanager/window/windows.py", line 275, in consolidate
    commandline_output = subprocess.check_output(command, shell=True)
  File "/usr/lib/python3.9/subprocess.py", line 424, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['sudo btrfs subvolume show ']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/buttermanager/buttermanager/window/windows.py", line 278, in consolidate
    str(called_process_error_exception.reason))
AttributeError: 'CalledProcessError' object has no attribute 'reason'
Aborted

buttermanager.log:

2021-11-10 12:39:31,703 INFO:ConfigManager. Checking OS. ARCH found
2021-11-10 12:39:31,703 INFO:ConfigManager. Checking Desktop Environment. kde found
2021-11-10 12:39:31,703 INFO:ConfigManager. Installation type: native
2021-11-10 12:39:31,703 INFO:ConfigManager. Creating PropertiesManager...
2021-11-10 12:39:31,704 INFO:ConfigManager. Migration process has finished successfully!
2021-11-10 12:39:31,704 INFO:ConfigManager. Retrieving user's configuration from buttermanager.yaml file and loading it in memory...
2021-11-10 12:39:31,704 INFO:VersionChecker. Checking for a new version of ButterManager. Please wait...
2021-11-10 12:39:31,721 INFO:PasswordWindow. Calculating appropriated base font size for UI elements...
2021-11-10 12:39:31,721 INFO:PasswordWindow. Base font size = 10
2021-11-10 12:39:31,923 INFO:VersionChecker. Last version is 2.4.2 and current version is 2.4.2
2021-11-10 12:39:35,248 INFO:RootSnapshotChecker. Checking if the current snapshot used for root is the default. Please wait...
2021-11-10 12:39:35,413 INFO:ButtermanagerMainWindow. BTRFS filesystems found in the system:
2021-11-10 12:39:35,413 INFO:ButtermanagerMainWindow. BTRFS Filesystem -> UUID: 8d49b3d9-7842-4db7-bc15-bbf64b8771e2; Devices: ['/dev/nvme1n1p2']; Mounted Points: ['/', '/home', '/mnt/defvol', '/var/lib/anbox/rootfs/cache', '/var/lib/anbox/rootfs/data']
2021-11-10 12:39:37,060 INFO:ConsolidateSnapshotWindow. Consolidating default root snapshot. The system has booted in /mnt/defvol/_snapshots/root-20211109-1 and it will be consolidated into 
2021-11-10 12:39:37,060 INFO:Subvolume. Deleting subvolume from origin /. Please wait...
2021-11-10 12:39:37,060 INFO:Subvolume. Deleting subvolume from origin /. Please wait...
2021-11-10 12:39:37,070 INFO:Subvolume. Snapshot / deleted.

2021-11-10 12:39:37,070 INFO:Subvolume. Snapshot / deleted.

btrfs-list output (currently booted into _snapshots/root-20211109-1):

NAME                                 TYPE     EXCL  MOUNTPOINT
8d49b3d9                               fs  227.70G (single, 698.00G free)
   [main]                         mainvol       -  /
   _active                         subvol       -  
   _active/root                    subvol       -  
      _snapshots/root-20211108-1     snap       -  
      _snapshots/root-20211109-0     snap       -  
      _snapshots/root-20211109-1     snap       -  /
      _snapshots/root-20211110-0     snap       -  
      _snapshots/root-20211110-1     snap       -  
   _active/home                    subvol       -  /home
      _snapshots/home-20211108-1   rosnap       -  
      _snapshots/home-20211109-0   rosnap       -  
      _snapshots/home-20211109-1   rosnap       -  
      _snapshots/home-20211110-0   rosnap       -  
      _snapshots/home-20211110-1   rosnap       -  
   _snapshots                      subvol       -

btrfs-list output (normal boot):

NAME                                 TYPE     EXCL  MOUNTPOINT
8d49b3d9                               fs  228.31G (single, 696.39G free)
   [main]                         mainvol       -  /
   _active                         subvol       -  
   _active/root                    subvol       -  /
      _snapshots/root-20211108-1     snap       -  
      _snapshots/root-20211109-0     snap       -  
      _snapshots/root-20211109-1     snap       -  
      _snapshots/root-20211110-0     snap       -  
      _snapshots/root-20211110-1     snap       -  
   _active/home                    subvol       -  /home
      _snapshots/home-20211108-1   rosnap       -  
      _snapshots/home-20211109-0   rosnap       -  
      _snapshots/home-20211109-1   rosnap       -  
      _snapshots/home-20211110-0   rosnap       -  
      _snapshots/home-20211110-1   rosnap       -  
   _snapshots                      subvol       -

I have the project cloned and tried stepping in with the debugger, but it was difficult to understand for me. It seemed like some variables just weren't there at all. If you want to take a look at it yourself, let me know.

Thanks and cheers

egara commented 2 years ago

Hi @Deckweiss

Oh, again this error :(

Could you attach again the config file please? (it is in ~/.buttermanager/buttermanager.yaml)

It seems that it doesn't find where to create the consolidated snapshot when restoring.

Thanks!

ghost commented 2 years ago

Here is the buttermanager.yaml (git doesnt allow yaml file extensions, so I pasted the content)

aur_repository: 1
check_at_startup: 0
grub_btrfs: 1
path_to_consolidate_root_snapshot: _active/root
save_log: 1
snap_packages: 0
subvolumes_dest: /mnt/defvol/_snapshots/|/mnt/defvol/_snapshots/
subvolumes_orig: /|/home/
subvolumes_prefix: root|home
subvolumes_snapshots_to_keep: 5|5

egara commented 2 years ago

Hi! This is my configuration file:

aur_repository: 0
check_at_startup: 1
grub_btrfs: 1
path_to_consolidate_root_snapshot: '@'
save_log: 1
snap_packages: 1
subvolumes_dest: /mnt/defvol/snapshots/|/mnt/defvol/snapshots/
subvolumes_orig: /mnt/defvol/@/|/mnt/defvol/@home/
subvolumes_prefix: root|home
subvolumes_snapshots_to_keep: 2|1

I think this is the relevant line:

subvolumes_orig: /mnt/defvol/@/|/mnt/defvol/@home/

Ok, Try to change this line. Instead of pointing directly to / and /home use the path to the actual subvolume location. In your case, I guess it should be subvolumes_orig: /mnt/defvol/_active/root/|/mnt/defvol/_active/home/ (Please, check it out before).

Then, reboot the system and boot into a previous snapshot. Then, try to consolidate and tell me.

Best,

Eloy

ghost commented 2 years ago

This did the trick!

Thank you for your support ❤️

egara commented 2 years ago

Hi @Deckweiss

Great!! I have to implement a way to restrict this. I mean, when a user configures a subvolume to create a snapshot in ButterManager, he/she selects / as origin. This is a very natural way of thinking. But it is not correct, because it is only the mount point and the original subvolume is below main BTRFS volume with ID 5. Because of this. it is necessary to have a mount point for volume 5 (for example, at /mnt/defvol). Then, if active root subvolume is /mnt/defvol/_active/rootvol, then origin should be this.

I want to implement a wizard in the next big release of ButterManager that do all this stuff automatically.

Meanwhile, I have to introduce some checks when user defines a subvolume to create snapshot for root partition and complete the documentation in order to explain all better.

Best,

Eloy

egara / buttermanager

Consolidation broken #40