Feature: snapper for snapshot management

[sfranzen]

Forum user flubber would like to see improved snapshot management using the snapper management tool:

Desired feautures:

• Timeline
• cleanup: delete snapshots without any difference to previous snapshots
• see differences between snapshots
• optional: possibility to rollback selected files, not the whole snapshot

  About Snapper

Snapper is a smart management tool for filesystem snapshots. Instead of manually configuring cron tasks, you configure a set of options independently for each btrfs subvolume. They are listed here, but the most important ones are the Timeline options. An important one missing from that list is SPACE_LIMIT, which allows specifying the maximum fraction of filesystem space available for snapshots. Snapper runs a daemon that interprets your configuration and automatically manages its cron tasks, allowing a very fine-grained and thought-free control over your snapshots.

Task list

Translation of desired features into application components/tasks:

• [x] Add dependency on snapper rpm and provide any default configuration needed;
• [x] Set up yum to take pre/post snapshots upon changing packages.
• [ ] Implement a web API at api/snapper/:
  • api/snapper:
  • [x] GET available configs
  • [x] POST create new config
  • api/snapper/:name:
  • [x] GET particular config
  • [x] PUT modify config
  • [x] DELETE
  • api/snapper/:name/snapshots:
  • [x] GET snapshot list for config (or a subset),
  • [x] POST create new snapshot
  • api/snapper/:name/snapshots/:id:
  • [x] GET particular snapshot details
  • [x] PUT modify details
  • [x] DELETE
  • api/snapper/:name/snapshots/:id/diffs:
  • [ ] POST request a comparison (file list + diffs) between snapshots
  • api/snapper/:name/snapshots/:id/diffs/:diff_id:
  • [ ] GET the diff between the snapshots id and diff_id
• [ ] Miscellaneous helpers (api/commands/?);
• [ ] Write required views:
  • [x] Overview: list of snapshot for given volume, actions to create/modify configs and snapshots,
  • [ ] Detail view: details about snapshots and diffs, actions to restore a snapshot,
  • [x] Config view: form for creating or editing snapper configs
• [ ] Figure out how to transition currently configured snapshots to snapper.

  Visuals

For a visual impression, this is what the snapper GUI looks like on my openSUSE system:

[williamstlr]

I used snapper before using Rockstor and I thought it was excellent for snapshot management, exactly for the reason you have listed. Would it be possible to move over current snapshots into snapper's layout?

[sfranzen]

@williamstlr We are currently using the native btrfs tools to create and delete snapshots, but with snapper installed it should certainly be possible to create a nicer interface.

[sfranzen]

Short progress summary: my API implementation so far communicates directly with the snapper daemon through its DBus interface, without the need for direct file manipulation or Django models. This works pretty nicely and avoids duplicating the entire configuration to the database. Backbone also does not care about this as any Python dict (-> JS object) is accepted as a model and an array of those as a collection.

One unsolved problem though is the fact that snapper also wants to use the .snapshots/ subvolumes. It will complain and refuse to create a configuration if this location already exists. I'm not yet sure about the best way to solve this, but my idea was to trap the exception thrown by snapper in that case and respond by presenting the user with a special view linked to an API command. However, I don't think it's possible to make snapper recognise or import the "standard" btrfs snapshots we currently use; the only way out may be to force deletion of the existing snapshots before configuring snapper.

Also, it might be more RESTful to structure the urls like api/snapper/{name}/snapshots/{number} or api/snapper-configs/{name}/snapshots/{number}. Please let me know what you think.

[sfranzen]

Another interesting case is comparing snapshots. This generates new information not stored anywhere by snapper, so for this part of the API I think I will use the database to cache the generated information instead of blindly requesting a new comparison each time.

Edit: It turns out that snapper does cache this information, within the snapshot directory. This makes things nice and easy as it ensures the cache is automatically removed along with the snapshot.

[MFlyer]

To @sfranzen : i was getting crazy looking for a good samba vfs objects reference and...et voila, a little think you can play with :grimacing: https://fossies.org/linux/samba/docs/manpages/vfs_snapper.8

[sfranzen]

@MFlyer Let's get this working first though :tongue: It's quite a lot of thinking and testing for relatively little code... :thinking:

[sfranzen]

Hrm, the snapper diff information will have to be a hybrid solution. I found out that while snapper stores some information after comparing snapshots, there is no way to check for which pairs this has been done. It can be quite an intensive operation when a large number of files have been modified, so some or all of the information will have to go in the database and be erased when either of the related snapshots is deleted.