search

g2p / bedup

Btrfs deduplication
http://pypi.python.org/pypi/bedup
GNU General Public License v2.0
322 stars 50 forks source link

OverflowError: Python int too large to convert to SQLite INTEGER #79

Open elmystico opened 8 years ago

elmystico commented 8 years ago

Hi I'm using linux containers so deduplication is serious thing regarding ram usage- fixing this will change a lot. When changed --size-cutoff=1024 I got this:

Traceback (most recent call last): File "/usr/sbin/bedup", line 9, in load_entry_point('bedup==0.10.1', 'console_scripts', 'bedup')() File "/root/.local/lib/python3.5/site-packages/bedup-0.10.1-py3.5-linux-x86_64.egg/bedup/main.py", line 497, in script_main sys.exit(main(sys.argv)) File "/root/.local/lib/python3.5/site-packages/bedup-0.10.1-py3.5-linux-x86_64.egg/bedup/main.py", line 486, in main return args.action(args) File "/root/.local/lib/python3.5/site-packages/bedup-0.10.1-py3.5-linux-x86_64.egg/bedup/main.py", line 185, in vol_cmd track_updated_files(sess, vol, tt) File "/root/.local/lib/python3.5/site-packages/bedup-0.10.1-py3.5-linux-x86_64.egg/bedup/tracking.py", line 203, in track_updated_files sess, Inode, vol=vol.impl, ino=ino) File "/root/.local/lib/python3.5/site-packages/bedup-0.10.1-py3.5-linux-x86_64.egg/bedup/model.py", line 71, in get_or_create return sess.query(model).filter_by(*_kwargs).one(), False File "/root/.local/lib/python3.5/site-packages/SQLAlchemy-1.1.0b2-py3.5-linux-x86_64.egg/sqlalchemy/orm/query.py", line 2736, in one ret = self.one_or_none() File "/root/.local/lib/python3.5/site-packages/SQLAlchemy-1.1.0b2-py3.5-linux-x86_64.egg/sqlalchemy/orm/query.py", line 2706, in one_or_none ret = list(self) File "/root/.local/lib/python3.5/site-packages/SQLAlchemy-1.1.0b2-py3.5-linux-x86_64.egg/sqlalchemy/orm/query.py", line 2777, in iter return self._execute_and_instances(context) File "/root/.local/lib/python3.5/site-packages/SQLAlchemy-1.1.0b2-py3.5-linux-x86_64.egg/sqlalchemy/orm/query.py", line 2800, in _execute_and_instances result = conn.execute(querycontext.statement, self._params) File "/root/.local/lib/python3.5/site-packages/SQLAlchemy-1.1.0b2-py3.5-linux-x86_64.egg/sqlalchemy/engine/base.py", line 947, in execute return meth(self, multiparams, params) File "/root/.local/lib/python3.5/site-packages/SQLAlchemy-1.1.0b2-py3.5-linux-x86_64.egg/sqlalchemy/sql/elements.py", line 262, in _execute_on_connection return connection._execute_clauseelement(self, multiparams, params) File "/root/.local/lib/python3.5/site-packages/SQLAlchemy-1.1.0b2-py3.5-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1055, in _execute_clauseelement compiled_sql, distilled_params File "/root/.local/lib/python3.5/site-packages/SQLAlchemy-1.1.0b2-py3.5-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1191, in _execute_context context) File "/root/.local/lib/python3.5/site-packages/SQLAlchemy-1.1.0b2-py3.5-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1389, in _handle_dbapi_exception util.reraise(_exc_info) File "/root/.local/lib/python3.5/site-packages/SQLAlchemy-1.1.0b2-py3.5-linux-x86_64.egg/sqlalchemy/util/compat.py", line 186, in reraise raise value File "/root/.local/lib/python3.5/site-packages/SQLAlchemy-1.1.0b2-py3.5-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1184, in _execute_context context) File "/root/.local/lib/python3.5/site-packages/SQLAlchemy-1.1.0b2-py3.5-linux-x86_64.egg/sqlalchemy/engine/default.py", line 462, in do_execute cursor.execute(statement, parameters) OverflowError: Python int too large to convert to SQLite INTEGER

elmystico commented 6 years ago

@g2p Hello Gabriel

Have you seen this? I'm not python fluent so much I'm able to fix your problem. People here and there advise to store the value as TEXT not INTEGER in sqlite but I'm not sure it will work in your particular implementation.

Could you give it little time and fix please?

elmystico commented 6 years ago

Further descrition: When --size-cutoff= is set somehow bigger the error is no more, so I guess some index gets overflowed. inode count?