Fast check if repo is ok

knutov commented 5 years ago

Connected with #4845

When I'm trying to add new snapshot to some broken repo I've mostly immediately got error.

Is there some way to fast check if repo is supposed to be ok or not rather than try to borg create ?

The case:

60 Tb of borg repos Full borg check takes about 3-6 hours per 200 Gb of data The idea is to fast find definitely broken repo and borg check only them.

ThomasWaldmann commented 5 years ago

what error do you get in such a case?

knutov commented 5 years ago

Unfortunately was able to find only one saved log, it's from the same repo as #4845:

Creating archive at "user@my.server:borg/repo::repo-2019-11-19-162034"
Initializing cache transaction: Reading config
Initializing cache transaction: Reading chunks
Initializing cache transaction: Reading files

Synchronizing chunks cache...
Archives: 120, w/ cached Idx: 119, w/ outdated Idx: 0, w/o cached Idx: 1.
  0% Syncing chunks cache. Processing archive repo-2019-08-15-034633
  Reading cached archive chunk index for repo-2019-08-15-034633 ...
Merging into master chunks index ...
  1% Syncing chunks cache. Processing archive repo-2019-09-26-024634
  Reading cached archive chunk index for repo-2019-09-26-024634 ...
Merging into master chunks index ...
  2% Syncing chunks cache. Processing archive repo-2019-09-04-024647
  Reading cached archive chunk index for repo-2019-09-04-024647 ...
Merging into master chunks index ...
[...]
  21% Syncing chunks cache. Processing archive repo-2019-10-01-024527
  Reading cached archive chunk index for repo-2019-10-01-024527 ...
Merging into master chunks index ...
 22% Syncing chunks cache. Processing archive repo-2019-07-15-024527
 Fetching and building archive index for repo-2019-07-15-024527 ...
Local Exception
Traceback (most recent call last):
  File "borg/archiver.py", line 4455, in main
  File "borg/archiver.py", line 4387, in run
  File "borg/archiver.py", line 154, in wrapper
  File "borg/archiver.py", line 550, in do_create
  File "borg/cache.py", line 380, in __new__
  File "borg/cache.py", line 374, in local
  File "borg/cache.py", line 467, in __init__
  File "borg/cache.py", line 851, in sync
  File "borg/cache.py", line 805, in create_master_idx
  File "borg/cache.py", line 714, in fetch_and_build_idx
  File "src/borg/hashindex.pyx", line 530, in borg.hashindex.CacheSynchronizer.feed
ValueError: cache_sync_feed failed: Unexpected object: integer

ThomasWaldmann commented 5 years ago

Well, finding that repo issue was in a rather slow operation (syncing the local chunks index by processing all archives' metadata).

knutov commented 5 years ago

It still MUCH faster than borg check..

ThomasWaldmann commented 5 years ago

Even when comparing to borg check --archives-only ...?

knutov commented 5 years ago

yes.

borg check --archives-only looks cpu consuming operation and lasts for about 30 minutes for this repo running locally, while trying to borg create takes about 5 minutes or less, even over network with high latency

borgbackup / borg

Fast check if repo is ok #4851