OverflowError: long too big to convert

[Burnfaker]

git)-[master] # ~/bin/bedup dedup /dev/mapper/XYZXYZ--extern-repos --size-cutoff 4096

Scanning volume /home/XYZXYZ/repos generations from 0 to 13899, with size cutoff 4096 00.00 Scanned retained 0/usr/local/lib/python2.7/dist-packages/cffi/vengine_cpy.py:177: UserWarning: reimporting '_cffix45ca2a7bxd5114704' might overwrite older definitions % (self.verifier.get_module_name())) 05:53.0 Scanned 3137686 retained 163440 Traceback (most recent call last): File "/root/bin/bedup", line 9, in load_entry_point('bedup==0.9.0', 'console_scripts', 'bedup')() File "/root/.local/lib/python2.7/site-packages/bedup-0.9.0-py2.7-linux-x86_64.egg/bedup/main__.py", line 487, in script_main sys.exit(main(sys.argv)) File "/root/.local/lib/python2.7/site-packages/bedup-0.9.0-py2.7-linux-x86_64.egg/bedup/main.py", line 476, in main return args.action(args) File "/root/.local/lib/python2.7/site-packages/bedup-0.9.0-py2.7-linux-x86_64.egg/bedup/main.py", line 184, in vol_cmd track_updated_files(sess, vol, tt) File "/root/.local/lib/python2.7/site-packages/bedup-0.9.0-py2.7-linux-x86_64.egg/bedup/tracking.py", line 206, in track_updated_files sess, Inode, vol=vol.impl, ino=ino) File "/root/.local/lib/python2.7/site-packages/bedup-0.9.0-py2.7-linux-x86_64.egg/bedup/model.py", line 71, in get_or_create return sess.query(model).filter_by(kwargs).one(), False File "build/bdist.linux-x86_64/egg/sqlalchemy/orm/query.py", line 2355, in one File "build/bdist.linux-x86_64/egg/sqlalchemy/orm/query.py", line 2398, in iter** File "build/bdist.linux-x86_64/egg/sqlalchemy/orm/query.py", line 2413, in _execute_and_instances File "build/bdist.linux-x86_64/egg/sqlalchemy/engine/base.py", line 717, in execute File "build/bdist.linux-x86_64/egg/sqlalchemy/sql/elements.py", line 317, in _execute_on_connection File "build/bdist.linux-x86_64/egg/sqlalchemy/engine/base.py", line 814, in _execute_clauseelement File "build/bdist.linux-x86_64/egg/sqlalchemy/engine/base.py", line 927, in _execute_context File "build/bdist.linux-x86_64/egg/sqlalchemy/engine/base.py", line 1079, in _handle_dbapi_exception File "build/bdist.linux-x86_64/egg/sqlalchemy/engine/base.py", line 920, in _execute_context File "build/bdist.linux-x86_64/egg/sqlalchemy/engine/default.py", line 425, in do_execute OverflowError: long too big to convert ~/bin/bedup dedup /dev/mapper/XYZXYZ--extern-repos --size-cutoff 4096 347,76s user 3,70s system 99% cpu 5:53,46 total

[Burnfaker]

Have any one solution here?

[mac-linux-free]

I think the solution is NOT to use skinny extents.

[mac-linux-free]

Ok it is all fixed with kernel 3.18.1 and btrfs-progs 3.18

[Burnfaker]

No it is not :)

:: Scanning volume /home/XXXX/repos generations from 0 to 25845, with size cutoff 4096 00.00 Scanned retained 0/root/.local/lib/python2.7/site-packages/cffi/vengine_cpy.py:177: UserWarning: reimporting '_cffixdf9034e3x2d9bc207' might overwrite older definitions % (self.verifier.get_module_name())) 09:10.7 Scanned 3523589 retained 124374 Traceback (most recent call last): File "/root/.local/bin/bedup", line 9, in load_entry_point('bedup==0.9.0', 'console_scripts', 'bedup')() File "/root/.local/lib/python2.7/site-packages/bedup-0.9.0-py2.7-linux-x86_64.egg/bedup/main__.py", line 487, in script_main sys.exit(main(sys.argv)) File "/root/.local/lib/python2.7/site-packages/bedup-0.9.0-py2.7-linux-x86_64.egg/bedup/main.py", line 476, in main return args.action(args) File "/root/.local/lib/python2.7/site-packages/bedup-0.9.0-py2.7-linux-x86_64.egg/bedup/main.py", line 184, in vol_cmd track_updated_files(sess, vol, tt) File "/root/.local/lib/python2.7/site-packages/bedup-0.9.0-py2.7-linux-x86_64.egg/bedup/tracking.py", line 206, in track_updated_files sess, Inode, vol=vol.impl, ino=ino) File "/root/.local/lib/python2.7/site-packages/bedup-0.9.0-py2.7-linux-x86_64.egg/bedup/model.py", line 71, in get_or_create return sess.query(model).filter_by(*_kwargs).one(), False File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2395, in one ret = list(self) File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2438, in iter return self._execute_and_instances(context) File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2453, in _execute_and_instances result = conn.execute(querycontext.statement, self._params) File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 729, in execute return meth(self, multiparams, params) File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 322, in _execute_on_connection return connection._execute_clauseelement(self, multiparams, params) File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 826, in _execute_clauseelement compiled_sql, distilled_params File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 958, in _execute_context context) File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1162, in _handle_dbapi_exception util.reraise(_exc_info) File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 951, in _execute_context context) File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 436, in do_execute cursor.execute(statement, parameters) OverflowError: Python int too large to convert to SQLite INTEGER ~/.local/bin/bedup dedup --flush --defrag --size-cutoff 4096 398,96s user 6,83s system 73% cpu 9:12,20 total

uname -a :( Linux XXXXX 3.19.0-0.rc3.git1.3.fc22.x86_64 #1 SMP Fri Jan 9 16:27:01 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

cat /etc/issue Fedora release 21 (Twenty One) Kernel \r on an \m (\l)

btrfs-progs-3.18-1.fc21.x86_64

[mac-linux-free]

for me it works...before linux 3.18.x and btrfs-progs 3.18.x it worked only without skinny-extents...after updating the kernel and progs it works also with skinny-extents (about 60TB undeduped)....did you apply this patch (https://github.com/g2p/bedup/blob/master/bedup/migrations.py#L52) delete the last line befor compliling?

I´m using debian 7 with mainline kernel from ubuntu and everything else complied from git

[Burnfaker]

Yes i've removed the last line in bedup/migration.py

If i try "~/.local/bin/bedup dedup --flush --defrag --size-cutoff 49200" works fine and if i try with "--size-cutoff 4096" i get an error

"OverflowError: Python int too large to convert to SQLite INTEGER"

[mac-linux-free]

Ah ok, my default cron is /root/.local/bin/bedup dedup --size-cutoff=524288 /mnt/btrfs. What does flush and defrag do ?

[Burnfaker]

README.rst 172:Before Linux 3.9, defragmentation could break copy-on-write sharing, 174:Btrfs defragmentation has to be explicitly requested (or background 175:defragmentation enabled), so this generally shouldn't be a problem for 178:Users of Linux 3.9 or newer can safely pass the --defrag option to 179:bedup dedup, which will defragment files before deduplicating them.

Flush outstanding data using syncfs before scanning volumes

[Burnfaker]

Ok. I solved this problem.

I've created a new partition with the Parameters Skinny-Metadata and this should be works fine. With the old partition and enabling skinny-metadata this didn't work.

Thx pls mark as RESOLVED and change the ReadME