Closed spitfire closed 11 years ago
This looks very similar to #7; please read my comment over there for troubleshooting. Also, what kernel are you running, was it recently upgraded?
I'm currently running x86_64 Gentoo on 3.8.0-rc2 kernel (you can say I've upgraded it recently, I did it the same day it was released).
python -V Python 2.7.3
'btrfs subvolume list -ap' listed (properly) all subvolumes for every volume I ran it for.
bedup release from pip (that you asked me to install) has different syntax, and doesn't recognize --debug parameter:
??[root@server01] - [~/bedup] - [2013-01-28 10:38:51] ??[$] ~/.local/bin/bedup --debug show /strg01 usage: python -m bedup [-h]
{scan-vol,dedup-vol,forget-vol,show-vols,dedup-files,generation,find-new}
...
python -m bedup: error: argument command: invalid choice: 'show' (choose from 'scan-vol', 'dedup-vol', 'forget-vol', 'show-vols', 'dedup-files', 'generation', 'find-new') ??[root@server01] - [~/bedup] - [2013-01-28 10:39:04] ??[$] ~/.local/bin/bedup --debug show-vols /strg01 usage: python -m bedup [-h]
{scan-vol,dedup-vol,forget-vol,show-vols,dedup-files,generation,find-new}
...
python -m bedup: error: unrecognized arguments: --debug /strg01
But I can say that this release 'just works' unlike the one from git, but it crashes too:
??[root@server01] - [~] - [2013-01-28 10:49:02] ??[$] ~/.local/bin/bedup dedup-vol /strg01/backup/timemachine/mieszko Scanning volume '/strg01/backup/timemachine/mieszko' generations from 0 to 222119, with size cutoff 8388608 16.80 Updated 2296 items: mieszko-macbook1.sparsebundle/bands/cc7 (ino 3092 outer gen 222098 inner gen 216144 size 8388608) (many more files) Deduplicated: 'mieszko-macbook1.sparsebundle/bands/1a' 'mieszko-macbook1.sparsebundle/bands/c9e' 01:19.7 Size group 1/1 Traceback (most recent call last): File "/root/.local/bin/bedup", line 9, in <module> load_entry_point('bedup==0.0.7', 'console_scripts', 'bedup')() File "/root/.local/lib64/python2.7/site-packages/bedup/__main__.py", line 236, in script_main sys.exit(main(sys.argv)) File "/root/.local/lib64/python2.7/site-packages/bedup/__main__.py", line 231, in main return args.action(args) File "/root/.local/lib64/python2.7/site-packages/bedup/__main__.py", line 127, in vol_cmd dedup_tracked(sess, volset, tt) File "/root/.local/lib64/python2.7/site-packages/bedup/tracking.py", line 399, in dedup_tracked dedup_tracked1(sess, tt, ofile_reserved, query, fs, skipped) File "/root/.local/lib64/python2.7/site-packages/bedup/tracking.py", line 417, in dedup_tracked1 for comm1 in query: File "/root/.local/lib64/python2.7/site-packages/bedup/tracking.py", line 355, in windowed_query clear_updates(window_start, window_end) File "/root/.local/lib64/python2.7/site-packages/bedup/tracking.py", line 378, in clear_updates window_start >= Inode.size >= window_end File "/root/.local/lib64/python2.7/site-packages/sqlalchemy/sql/expression.py", line 3740, in __nonzero__ raise TypeError("Boolean value of this clause is not defined") TypeError: Boolean value of this clause is not defined
And every next time I run it crashes too:
??[root@server01] - [~] - [2013-01-28 10:53:55] ??[$] ~/.local/bin/bedup dedup-vol /strg01/backup/timemachine/mieszko Scanning volume '/strg01/backup/timemachine/mieszko' generations from 222128 to 222129, with size cutoff 8388608 00.23 Updated 0 items: Did not deduplicate (same extents): 'mieszko-macbook1.sparsebundle/bands/1a' 'mieszko-macbook1.sparsebundle/bands/1b' (Many more files) Did not deduplicate (same extents): 'mieszko-macbook1.sparsebundle/bands/1a' 'mieszko-macbook1.sparsebundle/bands/c9e' 07.49 Size group 1/1 Traceback (most recent call last): File "/root/.local/bin/bedup", line 9, in <module> load_entry_point('bedup==0.0.7', 'console_scripts', 'bedup')() File "/root/.local/lib64/python2.7/site-packages/bedup/__main__.py", line 236, in script_main sys.exit(main(sys.argv)) File "/root/.local/lib64/python2.7/site-packages/bedup/__main__.py", line 231, in main return args.action(args) File "/root/.local/lib64/python2.7/site-packages/bedup/__main__.py", line 127, in vol_cmd dedup_tracked(sess, volset, tt) File "/root/.local/lib64/python2.7/site-packages/bedup/tracking.py", line 399, in dedup_tracked dedup_tracked1(sess, tt, ofile_reserved, query, fs, skipped) File "/root/.local/lib64/python2.7/site-packages/bedup/tracking.py", line 417, in dedup_tracked1 for comm1 in query: File "/root/.local/lib64/python2.7/site-packages/bedup/tracking.py", line 355, in windowed_query clear_updates(window_start, window_end) File "/root/.local/lib64/python2.7/site-packages/bedup/tracking.py", line 378, in clear_updates window_start >= Inode.size >= window_end File "/root/.local/lib64/python2.7/site-packages/sqlalchemy/sql/expression.py", line 3740, in __nonzero__ raise TypeError("Boolean value of this clause is not defined") TypeError: Boolean value of this clause is not defined
I didn't mean you should go back to the PyPI version, excuse my poor instructions. The idea was to include ipdb in the dependencies and keep the git version of bedup.
Ok, so with ipdb, it lists all subvolumes on volume mounted as root '/', which were working well already, but when it comes to /strg01 volume, I get this:
Label: None UUID: 68417477-82e6-4b7f-86a9-2d33bdfa1698 Device: /dev/sdb Device: /dev/sdc Device: /dev/sdd Device: /dev/sde KeyError(1993L,)
/root/.local/lib64/python2.7/site-packages/bedup-0.0.7-py2.7-linux-x86_64.egg/bedup/platform/btrfs.py(520)read_root_tree() 519 posixpath.join( --> 520 root_info[parent_root_id].path, parent_path, name), 521 parent_root_id,
Yup. What does root_info contain so far?
What is root_info/how do I get it?
--debug drops you into an interactive ipdb shell, just type the name
ipdb> root_info {5L: RootInfo(path='/', parent_root_id=None, is_frozen=False), 1680L: RootInfo(path='/lxc/containers/base', parent_root_id=1637L, is_frozen=False), 1683L: RootInfo(path='/lxc/containers/quassel01', parent_root_id=1637L, is_frozen=False), 1684L: RootInfo(path='/lxc/containers/ziproxy01', parent_root_id=1637L, is_frozen=False), 1685L: RootInfo(path='/lxc/containers/xbmcold', parent_root_id=1637L, is_frozen=False), 1942L: RootInfo(path='/distribution/updatecache', parent_root_id=346L, is_frozen=False), 1945L: RootInfo(path='/lxc/containers/guacamole01', parent_root_id=1637L, is_frozen=False), 1947L: RootInfo(path='/backup/clonezilla', parent_root_id=1185L, is_frozen=False), 1948L: RootInfo(path='/lxc/containers/oldshare01', parent_root_id=1637L, is_frozen=False), 1693L: RootInfo(path='/lxc/containers/tor01', parent_root_id=1637L, is_frozen=False), 1951L: RootInfo(path='/lxc/containers/share02', parent_root_id=1637L, is_frozen=False), 1952L: RootInfo(path='/lxc/containers/timemachine01', parent_root_id=1637L, is_frozen=False), 1185L: RootInfo(path='/backup', parent_root_id=5L, is_frozen=False), 1186L: RootInfo(path='/backup/timemachine', parent_root_id=1185L, is_frozen=False), 1955L: RootInfo(path='/db/mysql', parent_root_id=1954L, is_frozen=False), 1956L: RootInfo(path='/db/psql', parent_root_id=1954L, is_frozen=False), 1701L: RootInfo(path='/lxc/containers/squid01', parent_root_id=1637L, is_frozen=False), 1702L: RootInfo(path='/distribution/osx/macports', parent_root_id=346L, is_frozen=False), 1703L: RootInfo(path='/distribution/osx/macports/distfiles', parent_root_id=1702L, is_frozen=False), 1704L: RootInfo(path='/lxc/containers/openswan01', parent_root_id=1637L, is_frozen=False), 1705L: RootInfo(path='/lxc/containers/openvpn01', parent_root_id=1637L, is_frozen=False), 1706L: RootInfo(path='/lxc/containers/bacula01', parent_root_id=1637L, is_frozen=False), 1707L: RootInfo(path='/lxc/containers/mediatomb01', parent_root_id=1637L, is_frozen=False), 1708L: RootInfo(path='/lxc/containers/forked-daapd01', parent_root_id=1637L, is_frozen=False), 1711L: RootInfo(path='/lxc/containers/audio01', parent_root_id=1637L, is_frozen=False), 1968L: RootInfo(path='/pentium-m', parent_root_id=5L, is_frozen=False), 1969L: RootInfo(path='/pentium-m/system', parent_root_id=1968L, is_frozen=False), 1970L: RootInfo(path='/pentium-m/lxc', parent_root_id=1968L, is_frozen=False), 1971L: RootInfo(path='/pentium-m/smb4ad02', parent_root_id=1968L, is_frozen=False), 1716L: RootInfo(path='/lxc/containers/share01', parent_root_id=1637L, is_frozen=False), 1717L: RootInfo(path='/lxc/containers/bitcoin01', parent_root_id=1637L, is_frozen=False), 1974L: RootInfo(path='/openwrt', parent_root_id=5L, is_frozen=False), 1975L: RootInfo(path='/distribution/openwrt', parent_root_id=346L, is_frozen=False), 1941L: RootInfo(path='/virtualbox/machines/osxsrv-01', parent_root_id=1727L, is_frozen=False), 1972L: RootInfo(path='/share/music/frommacbook', parent_root_id=5L, is_frozen=False), 1727L: RootInfo(path='/virtualbox/machines', parent_root_id=5L, is_frozen=False), 1728L: RootInfo(path='/virtualbox/machines/winsrv-01', parent_root_id=1727L, is_frozen=False), 1967L: RootInfo(path='/distribution/gentoo/stages', parent_root_id=347L, is_frozen=False), 1990L: RootInfo(path='/base', parent_root_id=5L, is_frozen=False), 1949L: RootInfo(path='/lxc/containers/share01/.snapshots/samba-4.0.0-rc4', parent_root_id=1716L, is_frozen=False), 1954L: RootInfo(path='/db', parent_root_id=5L, is_frozen=False), 1959L: RootInfo(path='/lxc/containers/nagios01', parent_root_id=1637L, is_frozen=False), 346L: RootInfo(path='/distribution', parent_root_id=5L, is_frozen=False), 347L: RootInfo(path='/distribution/gentoo', parent_root_id=346L, is_frozen=False), 348L: RootInfo(path='/distribution/gentoo/distfiles', parent_root_id=347L, is_frozen=False), 349L: RootInfo(path='/distribution/gentoo/portage', parent_root_id=347L, is_frozen=False), 351L: RootInfo(path='/distribution/gentoo/snapshots', parent_root_id=347L, is_frozen=False), 1936L: RootInfo(path='/lxc/containers/strongswan01', parent_root_id=1637L, is_frozen=False), 1636L: RootInfo(path='/lxc', parent_root_id=5L, is_frozen=False), 1637L: RootInfo(path='/lxc/containers', parent_root_id=1636L, is_frozen=False), 1638L: RootInfo(path='/lxc/containers/db01', parent_root_id=1637L, is_frozen=False), 1639L: RootInfo(path='/lxc/containers/distcc01', parent_root_id=1637L, is_frozen=False), 1640L: RootInfo(path='/lxc/containers/p2p01', parent_root_id=1637L, is_frozen=False), 1641L: RootInfo(path='/lxc/containers/reposado01', parent_root_id=1637L, is_frozen=False), 1643L: RootInfo(path='/p2p', parent_root_id=5L, is_frozen=False), 1644L: RootInfo(path='/p2p/transmission', parent_root_id=1643L, is_frozen=False), 1645L: RootInfo(path='/p2p/transmission/downloads', parent_root_id=1644L, is_frozen=False), 1646L: RootInfo(path='/p2p/mldonkey', parent_root_id=1643L, is_frozen=False), 1648L: RootInfo(path='/p2p/mldonkey/incoming', parent_root_id=1646L, is_frozen=False), 1651L: RootInfo(path='/lxc/containers/virtualbox01', parent_root_id=1637L, is_frozen=False), 1973L: RootInfo(path='/lxc/containers/havp01', parent_root_id=1637L, is_frozen=False), 1957L: RootInfo(path='/portagetmp', parent_root_id=5L, is_frozen=False), 1663L: RootInfo(path='/systemd', parent_root_id=5L, is_frozen=False)}
Basically I want to compare it to btrfs subvolume list to check that everything fills up in the right order.
It seems wrong. I'll list subvolumes for you. volume '/' doesn't contain '/lxc/' subvolumes, but '/strg01' does
Also, I'd like to confirm that root_id > parent_root_id.
parent_root_id 1993L
List (correct one) of subvolumes, on both volumes. I don't think it matches info, from root_id:
??[root@server01] - [~/bedup] - [2013-01-28 11:51:15] ??[$] btrfs subvolume list / ID 259 gen 123765 top level 5 path containers ID 262 gen 125811 top level 5 path containers/xbmc01 ID 264 gen 125777 top level 5 path containers/p2p01 ID 266 gen 125811 top level 5 path containers/reposado01 ID 274 gen 125814 top level 5 path containers/crashplan01 ID 277 gen 125809 top level 5 path containers/timemachine01 ID 279 gen 124856 top level 5 path containers/syslog01 ID 280 gen 9367 top level 5 path containers/bitcoin01 ID 281 gen 124856 top level 5 path containers/theos01 ID 314 gen 124856 top level 5 path containers/minecraft01 ID 316 gen 124856 top level 5 path containers/squid01 ID 317 gen 124856 top level 5 path containers/prey01 ID 318 gen 124856 top level 5 path containers/havp01 ID 321 gen 124856 top level 5 path containers/sync01 ID 324 gen 72671 top level 5 path containers/old ID 327 gen 125814 top level 5 path containers/smb4ad01 ID 328 gen 124856 top level 5 path containers/smb4fs01 ID 332 gen 125814 top level 5 path containers/owncloud01 ID 334 gen 105048 top level 5 path containers/kolab01 ID 336 gen 90792 top level 5 path containers/opensuse ID 337 gen 91026 top level 5 path containers/ubuntu ID 338 gen 124856 top level 5 path containers/print01 ID 339 gen 124856 top level 5 path containers/netatalk01 ID 340 gen 105014 top level 5 path containers/wheezy ID 341 gen 125744 top level 5 path containers/backuppc01 ID 342 gen 121535 top level 5 path c7 ID 343 gen 125752 top level 5 path c7/base ??[root@server01] - [~/bedup] - [2013-01-28 11:52:33] ??[$] btrfs subvolume list /strg01 ID 346 gen 222138 top level 5 path distribution ID 347 gen 36342 top level 5 path distribution/gentoo ID 348 gen 221415 top level 5 path distribution/gentoo/distfiles ID 349 gen 222207 top level 5 path distribution/gentoo/portage ID 351 gen 222196 top level 5 path distribution/gentoo/snapshots ID 1185 gen 220293 top level 5 path backup ID 1186 gen 197858 top level 5 path backup/timemachine ID 1636 gen 220295 top level 5 path lxc ID 1637 gen 96038 top level 5 path lxc/containers ID 1638 gen 220824 top level 5 path lxc/containers/db01 ID 1639 gen 220824 top level 5 path lxc/containers/distcc01 ID 1640 gen 220824 top level 5 path lxc/containers/p2p01 ID 1641 gen 220825 top level 5 path lxc/containers/reposado01 ID 1643 gen 207150 top level 5 path p2p ID 1644 gen 207150 top level 5 path p2p/transmission ID 1645 gen 221349 top level 5 path p2p/transmission/downloads ID 1646 gen 11496 top level 5 path p2p/mldonkey ID 1648 gen 146706 top level 5 path p2p/mldonkey/incoming ID 1651 gen 220826 top level 5 path lxc/containers/virtualbox01 ID 1663 gen 12219 top level 5 path systemd ID 1680 gen 220827 top level 5 path lxc/containers/base ID 1683 gen 222226 top level 5 path lxc/containers/quassel01 ID 1684 gen 220829 top level 5 path lxc/containers/ziproxy01 ID 1685 gen 220840 top level 5 path lxc/containers/xbmcold ID 1693 gen 220827 top level 5 path lxc/containers/tor01 ID 1701 gen 220828 top level 5 path lxc/containers/squid01 ID 1702 gen 149175 top level 5 path distribution/osx/macports ID 1703 gen 149175 top level 5 path distribution/osx/macports/distfiles ID 1704 gen 220828 top level 5 path lxc/containers/openswan01 ID 1705 gen 220829 top level 5 path lxc/containers/openvpn01 ID 1706 gen 220829 top level 5 path lxc/containers/bacula01 ID 1707 gen 220830 top level 5 path lxc/containers/mediatomb01 ID 1708 gen 220831 top level 5 path lxc/containers/forked-daapd01 ID 1711 gen 220832 top level 5 path lxc/containers/audio01 ID 1716 gen 220832 top level 5 path lxc/containers/share01 ID 1717 gen 220833 top level 5 path lxc/containers/bitcoin01 ID 1727 gen 25529 top level 5 path virtualbox/machines ID 1728 gen 23883 top level 5 path virtualbox/machines/winsrv-01 ID 1936 gen 220833 top level 5 path lxc/containers/strongswan01 ID 1941 gen 20910 top level 5 path virtualbox/machines/osxsrv-01 ID 1942 gen 221481 top level 5 path distribution/updatecache ID 1945 gen 220834 top level 5 path lxc/containers/guacamole01 ID 1947 gen 222160 top level 5 path backup/clonezilla ID 1948 gen 220835 top level 5 path lxc/containers/oldshare01 ID 1949 gen 220832 top level 5 path lxc/containers/share01/.snapshots/samba-4.0.0-rc4 ID 1951 gen 220836 top level 5 path lxc/containers/share02 ID 1952 gen 220836 top level 5 path lxc/containers/timemachine01 ID 1954 gen 30895 top level 5 path db ID 1955 gen 145587 top level 5 path db/mysql ID 1956 gen 222233 top level 5 path db/psql ID 1957 gen 220842 top level 5 path portagetmp ID 1959 gen 220837 top level 5 path lxc/containers/nagios01 ID 1967 gen 36350 top level 5 path distribution/gentoo/stages ID 1968 gen 38795 top level 5 path pentium-m ID 1969 gen 220844 top level 5 path pentium-m/system ID 1970 gen 220846 top level 5 path pentium-m/lxc ID 1971 gen 220846 top level 5 path pentium-m/smb4ad02 ID 1972 gen 201734 top level 5 path share/music/frommacbook ID 1973 gen 220837 top level 5 path lxc/containers/havp01 ID 1974 gen 220848 top level 5 path openwrt ID 1975 gen 220810 top level 5 path distribution/openwrt ID 1990 gen 220848 top level 5 path base ID 1991 gen 83753 top level 5 path containers/athlonxp ID 1993 gen 80489 top level 5 path containers ID 1994 gen 66026 top level 5 path containers/pentium4 ID 1995 gen 220850 top level 5 path containers/pentium4/base ID 1996 gen 92900 top level 5 path containers/core2 ID 1998 gen 220837 top level 5 path lxc/containers/crashplan01 ID 1999 gen 222233 top level 5 path backup/crashplan ID 2001 gen 220838 top level 5 path lxc/containers/phpvirtualbox01 ID 2005 gen 97049 top level 5 path containers/prescott ID 2006 gen 222166 top level 5 path containers/prescott/samba4 ID 2019 gen 97033 top level 5 path containers/pentiumm ID 2022 gen 220851 top level 5 path containers/core2/samba4 ID 2023 gen 220854 top level 5 path containers/pentiumm/base ID 2025 gen 220854 top level 5 path containers/pentiumm/samba4 ID 2031 gen 220869 top level 5 path gentoosrc ID 2032 gen 220850 top level 5 path containers/athlonxp/base ID 2033 gen 220838 top level 5 path lxc/containers/smb4ad01 ID 2034 gen 220841 top level 5 path lxc/containers/xbmc01 ID 2041 gen 220852 top level 5 path containers/prescott/xbmc ID 2042 gen 220853 top level 5 path containers/prescott/p2p02 ID 2045 gen 222233 top level 5 path lxc/containers/smb4fs01 ID 2046 gen 220842 top level 5 path lxc/containers/smb4vpn01 ID 2047 gen 220853 top level 5 path containers/prescott/db ID 2048 gen 220855 top level 5 path containers/pentiumm/db ID 2049 gen 220855 top level 5 path containers/pentiumm/tor ID 2050 gen 220853 top level 5 path containers/prescott/tor ID 2051 gen 220869 top level 5 path oldgentoo ID 2066 gen 220870 top level 5 path wrtbuild01 ID 2081 gen 222160 top level 5 path owncloud01 ID 2082 gen 183950 top level 5 path dropbox ID 2083 gen 222211 top level 5 path backup/timemachine/mieszko ID 2084 gen 199001 top level 5 path distribution/android ID 2085 gen 222165 top level 5 path backup/backuppc01
With -p
, sorry.
The paths are internal, /strg01
is external.
Ah, with containers and containers/athlonxp I get parent_root_id > root_id, got it.
What should i start with '-p'? If it's easier, I'm on #btrfs as spitf1r3 now
Nah, I just found the assumption that was broken. When volumes are moved it's possible that parent_root_id > root_id. I'll fix the code.
Okay, let me know when it's ready to test:)
Here, I've pushed a fix.
Seems to work fine (doesn't crash at the start), let me wait for the result.
Do let me know, but I prefer to keep this to one problem per issue either way.
I have 2 btrfs volumes (one mounted at /, another one a /strg01). They both have sub volumes. / consists of one drive /strg01 is a raid 10 volume using 4 drives. 'bedup dedup /' runs just fine. But whenever I try to use something else, than '/' as a target (ie. '/strg01' or just run 'bedup dedup' as suggested to dedup all drives) I get an error. See commands I ran to reproduce it.