Tool to fix problems with siva files

[jfontan]

Problems

• Gluster brick had malfunction and we don't know if we have all the files.
• Bucket configuration was incorrect and siva files were downloaded to the incorrect directory. There are some siva files that were already downloaded to the correct directory. This makes these files to contain incomplete information.

Possible solutions

List of proposed commands and which ones are needed now

Lost gluster brick

Accessing gluster filesystem to check the presence of a file can be very expensive. Instead of using gluster we can take the list of files directly from the bricks (native FS) and have this list loaded in memory to find missing siva files:

• Create a curated list of hashes of siva files available in the bricks. This list can be retrieved with find in each host and then merged and take out duplicates.
• The list is loaded in memory in a map for faster query.
• Option A (slow):
  • Get the list of pairs of repositories - init commits from the database:

    select repository_id, init from repository_references
    group by repository_id, init

  • Iterate them and if the init is not in the list delete the reference and queue the repository.
• Option B:
  • As Index in References is already indexed we can get the list of distinct Index and find which ones are not in the map.
  • Delete all references with the missing Init and queue its repositories.

Notes

• Make sure that each repo is only queued once.
• To make it faster we can use several threads in a worker pool fashion. If it's done like this there should be a lock per repository or even better make each repository a single job.
• There can be some improvements in the algorithm like deleting all references from all repos to missing siva file and queuing them.
• Borges should be stopped. ??
• There is a naive initial implementation that still does not meet all the goals but could be a starting point:
  • https://github.com/jfontan/requeue

siva files not it buckets

A similar list can be done the same way as with lost gluster brick problem with the bucketed siva files and another one with the ones incorrectly placed. The siva files that are in both lists must be deleted and repositories requeued. The list of siva files that are only in the incorrect path must be moved to its proper bucket dir.

• Go through the list of all siva hashes (siva file name without .siva) that appear in the two lists.
  • Delete all references from that siva hash.
  • Queue repositories in that list.
• Go trough the list of siva hashes placed in the incorrect path but only appear once.
  • Move to the correct bucket directory.

Notes

• Make sure that each repo is only queued once.
• Check that gluster mv is only modifying metadata and not copying the file.
• Fuse can be very slow. Check @kuba-- gluster utils and bindings for libgfs:
  • https://github.com/kuba--/gfs
  • https://github.com/gluster/gogfapi

General notes

• It would be nice that this is the basis for a Swiss army knife to fix problems with borges, not only specific to these.
• If these two problems have to be fixed it can happen that a repository is queued twice. We may want to change the state to pending when queuing and do not queue again if is already in that state. I believe this could be enough.
• Nice to have features:
  • This process can be long so it would be good saving a checkpoint from time to time. This will make possible continuing the work if it is interrupted. Interesting things to save could be the siva and repositories processed but also the computed lists.
  • A --dry option that does not make any changes but tells what are the actions to be taken would be nice as errors here can produce data loss. It could also save the actions to be taken in a replayable file that can be executed after reviewing them. This will save doing some computation twice.
  • An option to quarantine files to another directory instead of deleting them. Just in case something goes wrong and we want to recover siva files.

[jfontan]

DISTINCT should not be used to get the init commits as is way slower than getting all:

testing=# select count(init) from repository_references;
   count   
-----------
 233888927
(1 row)

Time: 53338.143 ms
testing=# select count(distinct(init)) from repository_references;
  count   
----------
 11849984
(1 row)

Time: 1209028.016 ms

[jfontan]

To make it more flexible and easier to extend for unforeseen cases there can be commands that create lists and others that do some actions over them. Kind of unix piping style. Most of the list and filter commands can be done with standard unix commands so they may not need to be implemented.

Some examples of commands could be:

• get list of all init files (needed now)

tool siva database [<db connection string>]

• get list from disk

tool siva disk <directory>

• separate by bucket size

tool siva filter by-bucket <root dir> <list>

• get list of repeated sivas

tool siva filter repeated <list1> <list2>

• get list of inits only in one side

tool siva filter only-first <list1> <list2>

• change bucket size (move files from root to size 2 bucket dir) (needed now)

tool siva rebucket [--path <rooted-repos>] 0 2 <list>

• delete references with specific siva files and requeue (needed now)

tool siva delete [--path <rooted-repos>] [--bucket 2] <list>

• get list of repositories using a list of siva files

tool siva repos <list>

• get list of siva files from a list of repository endpoints

tool repos siva <list>

[jfontan]

Issues related to implement the tool:

• [x] Create go-billy-gluster #368
• [ ] Create tool to fix borges problems #369