Incorrect status/diff output

[carbn]

System information:

root@ubuntu:/# uname -a
Linux ubuntu 4.4.0-59-generic #80-Ubuntu SMP Fri Jan 6 17:47:47 UTC 2017 x86_64 x86_64 x86_64 
root@ubuntu:/# snapper --version
snapper 0.4.1
flags btrfs,lvm,ext4,xattrs,rollback,btrfs-quota,no-selinux

The problem is that sometimes when the user creates/deletes folders and moves files around, snapper can become confused what the changes between two snapshots are.

Here are the contents of the two problematic snapshots:

root@ubuntu:/# sh -c 'cd /mnt/shared/.snapshots/43/snapshot; find .'
.
./.snapshots
./a
./a/Lake McDonald.jpg
./a/d
./a/d/173.jpg
./b

root@ubuntu:/# sh -c 'cd /mnt/shared/.snapshots/47/snapshot; find .'
.
./.snapshots
./d
./d/173.jpg
./d/Lake McDonald.jpg
./d/Lake Tahoe Colors.jpg
./b

snapper output:

root@ubuntu:~# snapper list
Type   | #  | Pre # | Date                            | User | Cleanup  | Description | Userdata
-------+----+-------+---------------------------------+------+----------+-------------+---------
single | 0  |       |                                 | root |          | current     |
single | 22 |       | Tue 24 Jan 2017 12:00:08 PM EET | root | timeline | timeline    |
single | 43 |       | Tue 24 Jan 2017 12:14:28 PM EET | root |          |             |
single | 47 |       | Tue 24 Jan 2017 12:15:03 PM EET | root |          |             |
single | 48 |       | Tue 24 Jan 2017 12:15:18 PM EET | root |          |             |

root@ubuntu:/# snapper status 43..47
+..... /mnt/shared/b/b/d/Lake McDonald.jpg
-..... /mnt/shared/b/Lake McDonald.jpg
+..... /mnt/shared/d/Lake Tahoe Colors.jpg

root@ubuntu:/# snapper diff 43..47
/usr/bin/diff: /mnt/shared/.snapshots/43/snapshot/b/b/d/Lake McDonald.jpg: No such file or directory
/usr/bin/diff: /mnt/shared/.snapshots/47/snapshot/b/b/d/Lake McDonald.jpg: No such file or directory
/usr/bin/diff: /mnt/shared/.snapshots/43/snapshot/b/Lake McDonald.jpg: No such file or directory
/usr/bin/diff: /mnt/shared/.snapshots/47/snapshot/b/Lake McDonald.jpg: No such file or directory
Binary files /mnt/shared/.snapshots/43/snapshot/d/Lake Tahoe Colors.jpg and /mnt/shared/.snapshots/47/snapshot/d/Lake Tahoe Colors.jpg differ

What is incorrect here:

• status line 1: no such path exists
• status line 2: no such path exists
• diff line 5: path in snapshot 43 does not exist

I tested sending the same incremental snapshot using btrfs send/receive and the result looked ok to me, and it makes me believe this is an issue with snapper. I can provide the original btrfs disk image (~8MB compressed) if needed for further debugging.

[carbn]

Here's another incorrect output. The paths for deleted files include some kind of inode-derived path component that do not exist in the real file system.

root@ubuntu:/# snapper list
Type   | #  | Pre # | Date                            | User | Cleanup  | Description | Userdata
-------+----+-------+---------------------------------+------+----------+-------------+---------
single | 0  |       |                                 | root |          | current     |
single | 3  |       | Wed 25 Jan 2017 10:00:03 AM EET | root | timeline | timeline    |
single | 20 |       | Wed 25 Jan 2017 10:11:28 AM EET | root |          |             |
single | 21 |       | Wed 25 Jan 2017 10:11:53 AM EET | root |          |             |
single | 23 |       | Wed 25 Jan 2017 10:12:13 AM EET | root |          |             |

Problem is between 20..21. The files and the parent folder were removed. Here's the contents:

root@ubuntu:/# sh -c 'cd /mnt/shared/.snapshots/20/snapshot; find .'
.
./.snapshots
./f
./f/Star River.jpg
./f/Horseshoe Bend Sunset.jpg

root@ubuntu:/# sh -c 'cd /mnt/shared/.snapshots/21/snapshot; find .'
.
./.snapshots

And the status & diff outputs:

root@ubuntu:/# snapper status 20..21
-..... /mnt/shared/o264-492-0/Horseshoe Bend Sunset.jpg
-..... /mnt/shared/o264-492-0/Star River.jpg

root@ubuntu:/# snapper diff 20..21
/usr/bin/diff: /mnt/shared/.snapshots/20/snapshot/o264-492-0/Horseshoe Bend Sunset.jpg: No such file or directory
/usr/bin/diff: /mnt/shared/.snapshots/21/snapshot/o264-492-0/Horseshoe Bend Sunset.jpg: No such file or directory
/usr/bin/diff: /mnt/shared/.snapshots/20/snapshot/o264-492-0/Star River.jpg: No such file or directory
/usr/bin/diff: /mnt/shared/.snapshots/21/snapshot/o264-492-0/Star River.jpg: No such file or directory

As with the original post, I can provide the fs image if needed.

[aschnell]

Using the btrfs send/receive to get the file changes was done in a hury and also many years ago (2013). Maybe the code was incorrect from the beginning or the btrfs send/receive stream changed meanwhile. In any case I won't have time to work in it.

[carbn]

Thanks for the response. If I understood correctly, the btrfs send stream is ordered by inode numbers instead of being in the chronological order, and this is why some directories are renamed to temporary names before they are deleted. The idea behind the current implementation is IMO good (tracking the changes and outputting the result), but it can't handle these delayed directory removes properly. It wasn't obvious to me how this could be fixed as the code behind the send stream parsing is quite complex without too many comments.

Here are examples on how anyone can produce these errors:

Case 1 (non-existing path):

snapper create

touch file1
mkdir dir1
mkdir dir1/dir2
mkdir dir3
mv file1 dir1/

snapper create

mv dir1/file1 dir1/dir2/
rmdir dir3
mv dir1 dir3
mv dir3/dir2 dir2

snapper create

Case 2 (temporary oNNN-NNN-N folder names):

snapper create

touch file1
mkdir dir1
mv file1 dir1/
touch dir1/file2

snapper create

rm -rf dir1

snapper create

As snapper::cmpDirs fallback gives a correct result in both cases maybe it would be beneficial to have tests that verify that both libbtrfs and fallback cmpDirs produce the same result.