search

egara / buttermanager

ButterManager is a BTRFS tool for managing snapshots, balancing filesystems and upgrading the system safetly.
GNU General Public License v3.0
129 stars 6 forks source link

Consolidation not working if docker uses btrfs subvolumes #46

Closed ghost closed 2 years ago

ghost commented 2 years ago

When trying to restore the previous snapshot I get

Error removing root snapshot because it is not empty and there are subvolumes within it. The consolidation process couldn't be done.

I think this is due to docker using btrfs storage driver when your system is btrfs. https://docs.docker.com/storage/storagedriver/btrfs-driver/

btrfs-list

NAME                                                                                                                                                             TYPE     EXCL  MOUNTPOINT
8d49b3d9                                                                                                                                                           fs  524.44G (single, 397.26G free)
   [main]                                                                                                                                                     mainvol       -  /
   _active                                                                                                                                                     subvol       -  
   _active/home                                                                                                                                                subvol       -  /home
      _snapshots/home-20220509-0                                                                                                                               rosnap       -  
      _snapshots/home-20220510-0                                                                                                                               rosnap       -  
      _snapshots/home-20220510-1                                                                                                                               rosnap       -  
      _snapshots/home-20220515-0                                                                                                                               rosnap       -  
      _snapshots/home-20220516-0                                                                                                                               rosnap       -  
      _snapshots/home-20220522-0                                                                                                                               rosnap       -  
   _snapshots                                                                                                                                                  subvol       -  
   _active/root/var/lib/docker/btrfs/subvolumes/5c0c577fbc807d18f455b25dbf15dd8f2206340ac3219cac28705ddb445870b1                                               subvol       -  
      _active/root/var/lib/docker/btrfs/subvolumes/89da89fb79c534966bdbddfd90f71d75c2787ee5356caf26519c9ef376f40d4f                                              snap       -  
         _active/root/var/lib/docker/btrfs/subvolumes/4f71a21eed30b27a35a7af560304d4c631b5385ae1f0d842f11cb9188c0191ea                                           snap       -  
            _active/root/var/lib/docker/btrfs/subvolumes/53faadbc170d31293d35fc818b6061fc70bad40b84349864fef2664b19ca9230                                        snap       -  
               _active/root/var/lib/docker/btrfs/subvolumes/33458e412d4db56fa713e35e35ff9084c49d4bf73e4b01c78eb81bdf6e471f53                                     snap       -  
                  _active/root/var/lib/docker/btrfs/subvolumes/97122f575ec6ab5203b3d292a0b1c7be1d18cfd60ee27c0b43797614bb6f2e44                                  snap       -  
                     _active/root/var/lib/docker/btrfs/subvolumes/64a1becc16557148dc72eedfcafb898b35d63f0bd421b0ce724a9f7347f45d99                               snap       -  
                        _active/root/var/lib/docker/btrfs/subvolumes/f578126cce5500bab66683244214169cc47e33a60313a95599a4f8df450ead3d                            snap       -  
                           _active/root/var/lib/docker/btrfs/subvolumes/bf3b32b5f38bc6bc10ceafc07dc04fbd4b5cd9b94da94e602f7b361895e7b396                         snap       -  
                              _active/root/var/lib/docker/btrfs/subvolumes/c11b92d4bbd019cd8e2a688d51445327f0b9644ecf90d07e44d5c4e76b168ed3                      snap       -  
                                 _active/root/var/lib/docker/btrfs/subvolumes/07cabea4bd28aaf01f9fd759f3aad7d3baf33fcb46ea88928c250bcad7518db4                   snap       -  
                                    _active/root/var/lib/docker/btrfs/subvolumes/9d6cec32ee4ea01d553b0697507c3158dd24cb21ecf116af80d4ac1471c36ea2                snap       -  
                                       _active/root/var/lib/docker/btrfs/subvolumes/ba9fccbaa568553450a59ebaf885bb48f6d60698ea30754e00d5ff24adb5bf4a             snap       -  
                                          _active/root/var/lib/docker/btrfs/subvolumes/b5132c68ca5ecc28cac84934c1c8b29221808eeee62a16d9a02b9051fac9afdc-init     snap       -  
                                             _active/root/var/lib/docker/btrfs/subvolumes/b5132c68ca5ecc28cac84934c1c8b29221808eeee62a16d9a02b9051fac9afdc       snap       -  
_snapshots/root-20220522-0                                                                                                                                       snap       -  /
_snapshots/root-20220510-1                                                                                                                                       snap       -  
_active/root                                                                                                                                                     snap       -  
_snapshots/root-20220510-0                                                                                                                                       snap       -  
_snapshots/root-20220516-0                                                                                                                                       snap       -  
_snapshots/root-20220515-0                                                                                                                                       snap       -  
_snapshots/root-20220509-0                                                                                                                                       snap       -

btrfs subvolume list /mnt/defvol/_active/root/var/lib/docker/btrfs/ 

ID 256 gen 249506 top level 5 path _active
ID 258 gen 249550 top level 256 path _active/home
ID 259 gen 249506 top level 5 path _snapshots
ID 1154 gen 249543 top level 256 path _active/root
ID 2161 gen 249539 top level 1154 path var/lib/docker/btrfs/subvolumes/5c0c577fbc807d18f455b25dbf15dd8f2206340ac3219cac28705ddb445870b1
ID 2162 gen 249540 top level 1154 path var/lib/docker/btrfs/subvolumes/89da89fb79c534966bdbddfd90f71d75c2787ee5356caf26519c9ef376f40d4f
ID 2163 gen 249540 top level 1154 path var/lib/docker/btrfs/subvolumes/4f71a21eed30b27a35a7af560304d4c631b5385ae1f0d842f11cb9188c0191ea
ID 2164 gen 249540 top level 1154 path var/lib/docker/btrfs/subvolumes/53faadbc170d31293d35fc818b6061fc70bad40b84349864fef2664b19ca9230
ID 2165 gen 249540 top level 1154 path var/lib/docker/btrfs/subvolumes/33458e412d4db56fa713e35e35ff9084c49d4bf73e4b01c78eb81bdf6e471f53
ID 2166 gen 249540 top level 1154 path var/lib/docker/btrfs/subvolumes/97122f575ec6ab5203b3d292a0b1c7be1d18cfd60ee27c0b43797614bb6f2e44
ID 2167 gen 249540 top level 1154 path var/lib/docker/btrfs/subvolumes/64a1becc16557148dc72eedfcafb898b35d63f0bd421b0ce724a9f7347f45d99
ID 2168 gen 249540 top level 1154 path var/lib/docker/btrfs/subvolumes/f578126cce5500bab66683244214169cc47e33a60313a95599a4f8df450ead3d
ID 2169 gen 249541 top level 1154 path var/lib/docker/btrfs/subvolumes/bf3b32b5f38bc6bc10ceafc07dc04fbd4b5cd9b94da94e602f7b361895e7b396
ID 2170 gen 249541 top level 1154 path var/lib/docker/btrfs/subvolumes/c11b92d4bbd019cd8e2a688d51445327f0b9644ecf90d07e44d5c4e76b168ed3
ID 2171 gen 249541 top level 1154 path var/lib/docker/btrfs/subvolumes/07cabea4bd28aaf01f9fd759f3aad7d3baf33fcb46ea88928c250bcad7518db4
ID 2172 gen 249542 top level 1154 path var/lib/docker/btrfs/subvolumes/9d6cec32ee4ea01d553b0697507c3158dd24cb21ecf116af80d4ac1471c36ea2
ID 2173 gen 249542 top level 1154 path var/lib/docker/btrfs/subvolumes/ba9fccbaa568553450a59ebaf885bb48f6d60698ea30754e00d5ff24adb5bf4a
ID 2178 gen 249542 top level 1154 path var/lib/docker/btrfs/subvolumes/b5132c68ca5ecc28cac84934c1c8b29221808eeee62a16d9a02b9051fac9afdc-init
ID 2179 gen 249543 top level 1154 path var/lib/docker/btrfs/subvolumes/b5132c68ca5ecc28cac84934c1c8b29221808eeee62a16d9a02b9051fac9afdc
ID 2319 gen 249507 top level 259 path _snapshots/root-20220509-0
ID 2320 gen 234208 top level 259 path _snapshots/home-20220509-0
ID 2321 gen 249507 top level 259 path _snapshots/root-20220510-0
ID 2322 gen 235498 top level 259 path _snapshots/home-20220510-0
ID 2323 gen 249507 top level 259 path _snapshots/root-20220510-1
ID 2324 gen 235785 top level 259 path _snapshots/home-20220510-1
ID 2647 gen 249507 top level 259 path _snapshots/root-20220515-0
ID 2648 gen 240582 top level 259 path _snapshots/home-20220515-0
ID 2649 gen 249507 top level 259 path _snapshots/root-20220516-0
ID 2650 gen 242162 top level 259 path _snapshots/home-20220516-0
ID 2651 gen 249545 top level 259 path _snapshots/root-20220522-0
ID 2652 gen 248927 top level 259 path _snapshots/home-20220522-0

This is such a mess ... not sure if buttermanager can really do much about that.

ghost commented 2 years ago

How I solved it in a bruteforce way:

  1. Backed up the /mnt/defvol/_active/root/var/lib/docker directory to a tar.gz file just in case
  2. sudo -s
  3. cd /mnt/defvol/_active/root/var/lib/docker
  4. btrfs subvolume delete btrfs/subvolumes/*

After that, docker could not start the container. Luckily I had nothing too important there, so I ran docker system prune But I will have to set up the data in a db again.

It does worry me though about how to restore a snapshot in the future.

ghost commented 2 years ago

After this I changed the docker storage driver to fuse-overlayfs instead of btrfs by following https://docs.docker.com/storage/storagedriver/overlayfs-driver/

So that this issue will not occur in the future.

egara commented 2 years ago

Hi @Deckweiss

Thanks for reporting :)

You are right, that problem is more a layout problem than a ButterManager issue 'per se'. Some time ago, I created a cheatsheet and documentation regarding to a simple BTRFS layout for a new system and you can see here this is something I pointed it out, because it is a real problem for people that use Docker in a BTRFS filesystem.

So your solution is the easiest one I found too: If you are running Docker in a BTRFS filesystem, the best option is to change the configuration of the storage driver that Docker will use by default. Otherwise, some problems could arise sooner or later as you mentioned in this issue.

Best wishes!