search

jborg / attic

Deduplicating backup program
Other
1.11k stars 104 forks source link

OverflowError when attempting to create a backup #354

Open jscinoz opened 9 years ago

jscinoz commented 9 years ago

This seems to be the same as #108, which was closed due to being unreproducible against 0.16. When attempting to create a new backup, during the cache construction the following error is encountered (again, this is with Attic 0.16):

Traceback (most recent call last):
  File "/usr/lib64/python3.4/site-packages/attic/remote.py", line 298, in get_many
    yield self.load_object(*self.index[key])
  File "hashindex.pyx", line 95, in attic.hashindex.NSIndex.__getitem__ (attic/hashindex.c:2315)
KeyError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/attic", line 3, in <module>
    main()
  File "/usr/lib64/python3.4/site-packages/attic/archiver.py", line 715, in main
    exit_code = archiver.run(sys.argv[1:])
  File "/usr/lib64/python3.4/site-packages/attic/archiver.py", line 705, in run
    return args.func(args)
  File "/usr/lib64/python3.4/site-packages/attic/archiver.py", line 100, in do_create
    cache = Cache(repository, key, manifest)
  File "/usr/lib64/python3.4/site-packages/attic/cache.py", line 33, in __init__
    self.sync()
  File "/usr/lib64/python3.4/site-packages/attic/cache.py", line 163, in sync
    for key, chunk in zip(archive[b'items'], repository.get_many(archive[b'items'])):
  File "/usr/lib64/python3.4/site-packages/attic/remote.py", line 302, in get_many
    self.store_object(key, data)
  File "/usr/lib64/python3.4/site-packages/attic/remote.py", line 288, in store_object
    self.index[key] = offset - len(data), len(data)
  File "hashindex.pyx", line 101, in attic.hashindex.NSIndex.__setitem__ (attic/hashindex.c:2424)
OverflowError: value too large to convert to int

The error reoccurs whenever I attempt to create a backup, but it's not consistently failing on analysing any specific archive - it fails on a different archive each time I run it.

ThomasWaldmann commented 9 years ago

This is because the RepoCache which is used for remote repos can only cache 2GB. I've fixed that in my repo.

jscinoz commented 9 years ago

@ThomasWaldmann Thanks for the information - is there any migration path for existing repositories from attic -> borg?

ThomasWaldmann commented 9 years ago

Yes, the current git code (and the next release) has migration code.

jscinoz commented 9 years ago

Ah, great, thank you for that. I'll give it a try with borg.

jscinoz commented 8 years ago

@ThomasWaldmann Done: borgbackup/borg#269