httm on mounted restic repository?

restic is a backup program that is fast, efficient and secure.

Is it possible with some trickery to use httm on mounted restic repository? Restore using mount

The structure of mounted restic repository is similar to zfs snaphots directory.

Is it possible with some trickery to use httm on mounted restic repository? Restore using mount

Yes, it's possible for restic and borg and kopia.

The structure of mounted restic repository is similar to zfs snaphots directory.

One issue I see/one question I have is: How does one relate the live filesystem to the restic repo? If it's as simple as, there is a single machine/namespace/top level at the repo (as opposed to a hierarchy of many filesystems like btrfs/ZFS), then that is very much like a Time Machine backup (something we already do), and adding support would be similar to the Time Machine method.

What I don't want is to install restic and find out it has nothing like the ZFS directory structure, or there is some other impediment to use. btrfs support was a nightmare, whereas nilfs2 was a breeze, and the reason why is even btrfs users could not explain to me how it worked, and left me to figure out on my own.

So re: your support request -- if you or others want support, one could provide, 1) the FUSE mount info for a restic mount, and 2) a precise explanation as to how to find the snapshots from the base mount point (here /srv/timemachine).

Like so --

~ mount | grep zfs | grep timemachine
rpool/timemachine on /srv/timemachine type zfs (rw,noatime,xattr,posixacl,casesensitive)

~ cd /srv/timemachine
~ ls -1 .zfs/snapshots
...
snap_2024-07-05-04:54:16_postmountSnap
snap_2024-07-05-04:54:17_premountSnap
snap_2024-07-05-05:36:45_postmountSnap
snap_2024-07-05-06:39:17_premountSnap
snap_2024-07-05-06:47:22_postmountSnap
snap_2024-07-05-07:40:07_premountSnap
snap_2024-07-05-07:47:14_postmountSnap
snap_2024-07-05-08:40:19_premountSnap
snap_2024-07-05-08:47:31_postmountSnap
snap_2024-07-05-09:41:20_premountSnap
snap_2024-07-05-10:17:21_postmountSnap
syncoid_montrose_2024-07-05:00:02:16-GMT-05:00

~ mount | grep '^restic'
restic on /tmp/05-restic/mnt (fusefs, read-only)

~ cd /tmp/05-restic/mnt
~ ls -1 snapshots/
2024-07-05T23:17:59+03:00
2024-07-05T23:18:33+03:00
2024-07-05T23:18:52+03:00
latest

You can try a dev branch by installing via: cargo install --locked --git https://github.com/kimono-koans/httm.git --branch restic_support

One impediment to use right now is restic support will fail with a warning, if there is already a supported root mount (like a btrfs or ZFS mount with snapshots). This works like the Time Machine support. I understand this is non-ideal and the solution may be to use the --map-alias support, or to provide a toggle. I haven't decided yet.

Feel free to try. If it doesn't work for you re: a basic command, please feel free to let me know here and provide the --debug output.

Sorry for the late reply.

I have not had any luck so far with your dev branch, which has been installed in the FreeBSD jail.

From the debug output, it seems that httm cannot find the mounted restic snapshots:

~ .cargo/bin/httm --debug wrk/hello1.txt
Config {
    paths: [
        PathData {
            path_buf: "/usr/home/user/wrk/hello1.txt",
            metadata: Some(
                PathMetadata {
                    size: 23,
                    modify_time: SystemTime {
                        tv_sec: 1720502863,
                        tv_nsec: 221548000,
                    },
                },
            ),
        },
    ],
    opt_recursive: false,
    opt_exact: false,
    opt_no_filter: false,
    opt_debug: true,
    opt_no_traverse: false,
    opt_omit_ditto: false,
    opt_no_hidden: false,
    opt_json: false,
    opt_one_filesystem: false,
    opt_no_clones: false,
    uniqueness: UniqueMetadata,
    opt_bulk_exclusion: None,
    opt_last_snap: None,
    opt_preview: None,
    opt_deleted_mode: None,
    opt_requested_dir: None,
    requested_utc_offset: +03:00:00,
    exec_mode: BasicDisplay,
    print_mode: FormattedDefault,
    dataset_collection: FilesystemInfo {
        map_of_datasets: MapOfDatasets {
            inner: {
                "/": DatasetMetadata {
                    source: "zroot/bastille/jails/httm/root",
                    fs_type: Zfs,
                },
                "/.bastille": DatasetMetadata {
                    source: "/usr/local/bastille/releases/14.0-RELEASE",
                    fs_type: Zfs,
                },
            },
        },
        map_of_snaps: MapOfSnaps {
            inner: {
                "/": [
                    "/.zfs/snapshot/01",
                ],
                "/.bastille": [],
            },
        },
        filter_dirs: FilterDirs {
            inner: {
                "/dev",
                "/usr/home/user/mnt",
            },
        },
        opt_map_of_alts: None,
        opt_map_of_aliases: None,
        opt_common_snap_dir: None,
    },
    pwd: "/usr/home/user",
}
────────────────────────────────────────────────────────────────────────────────────
Tue Jul 09 08:27:43 2024  23 bytes  "/.zfs/snapshot/01/usr/home/user/wrk/hello1.txt"
────────────────────────────────────────────────────────────────────────────────────
Tue Jul 09 08:27:43 2024  23 bytes  "/usr/home/user/wrk/hello1.txt"
────────────────────────────────────────────────────────────────────────────────────

There is no warning.

Also no luck with --map-aliases:

~ .cargo/bin/httm --debug --map-aliases /usr/home/user/wrk:/usr/home/user/mnt wrk/hello1.txt
thread 'main' panicked at /home/user/.cargo/registry/src/index.crates.io-6f17d22bba15001f/clap_builder-4.5.2/src/parser/error.rs:32:9:
Mismatch between definition and access of `MAP_ALIASES`. Could not downcast to TypeId { t: (13369810092395217272, 7199936582794304877) }, need to downcast to TypeId { t: (1868318267586526722, 11993923412381082341) }

stack backtrace:
   0:     0x3317082ed3d0 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h0d22df41a72f543b
   1:     0x33170831b8cb - core::fmt::write::hf2dc21fb3bf42417
   2:     0x3317082f038f - std::io::Write::write_fmt::h5f62cfea9e36bf6d
   3:     0x3317082ed19e - std::sys_common::backtrace::print::hd8888c2893c6effb
   4:     0x3317082f3f0a - std::panicking::default_hook::{{closure}}::h18c73f323706cc09
   5:     0x3317082f3bfd - std::panicking::default_hook::h81da2ddd2a0d9c9c
   6:     0x3317082f43e9 - std::panicking::rust_panic_with_hook::h005f80f82882b02a
   7:     0x3317082ed7a4 - std::panicking::begin_panic_handler::{{closure}}::hc308c169babd81fb
   8:     0x3317082ed5e9 - std::sys_common::backtrace::__rust_end_short_backtrace::hd1f19fa3642d785e
   9:     0x3317082f40e7 - rust_begin_unwind
  10:     0x331708326143 - core::panicking::panic_fmt::hf140f60dfedb8b15
  11:     0x33170807ed7f - httm::config::generate::Config::from_matches::h72e6c7159dfad0d8
  12:     0x331708069a24 - core::ops::function::FnOnce::call_once::h277778b80a822778
  13:     0x33170806e638 - once_cell::imp::OnceCell<T>::initialize::{{closure}}::ha460fe14211193d6
  14:     0x331708278c9c - once_cell::imp::initialize_or_wait::h5771a504e0286d58
  15:     0x33170806e0f5 - once_cell::imp::OnceCell<T>::initialize::h24251c25dbcabf88
  16:     0x33170808171d - httm::main::hcfa3f2cb03b83309
  17:     0x3317080475c3 - std::sys_common::backtrace::__rust_begin_short_backtrace::hcc8b97cdbc0c9516
  18:     0x3317080a98d9 - std::rt::lang_start::{{closure}}::hf7d4ce6af18d60b8
  19:     0x3317082f3f87 - std::panicking::try::hbf82ab7fea049861
  20:     0x3317082f73c8 - std::rt::lang_start_internal::h1da71261e5f3a6ce
  21:     0x331708082905 - main
  22:     0x331f2b079afa - __libc_start1
  23:     0x331707fd219d - _start

~ mount | grep '^restic'
restic on /usr/home/user/mnt (fusefs, read-only)

~ cd /usr/home/user/mnt
~ ls -1 snapshots
2024-07-09T08:26:59+03:00
2024-07-09T08:27:22+03:00
2024-07-09T08:27:49+03:00
latest
~ tree snapshots/2024-07-09T08:26:59+03:00/
snapshots/2024-07-09T08:26:59+03:00/
└── home
    └── user
        └── wrk
            └── hello1.txt

4 directories, 1 file

Sorry for the late reply.

I have not had any luck so far with your dev branch, which has been installed in the FreeBSD jail.

I'm sorry. Support has only been added for Linux, though I previously thought I added support for all! I now understand FreeBSD support will be a little trickier. I'm rethinking the way I've implemented so far. I'm going to think a little more deeply about how to do this best. Maybe I need to add a new flag. I will let you know once I nail it down.

From the debug output, it seems that httm cannot find the mounted restic snapshots:

There is no warning.

Also no luck with --map-aliases:

The failure you experienced was another bug related to a dependency that had to be upgraded called clap. This is fixed at: https://github.com/kimono-koans/httm/commit/cef0590764b0641d873213751f29b212d8debbad

If you like, you can try the new dev branch by installing via: cargo install --locked --git https://github.com/kimono-koans/httm.git --branch restic2. But I would understand if you are once burnt, twice shy.

My test on Linux:

➜  httm git:(restic2) ✗ sudo httm --debug --backup=restic httm.1
WARN: httm has disabled any MAP_ALIASES in preference to an ALT_BACKUP specified.
Config {
    paths: [
        PathData {
            path_buf: "/srv/program/httm/httm.1",
            metadata: Some(
                PathMetadata {
                    size: 13684,
                    modify_time: SystemTime {
                        tv_sec: 1721416439,
                        tv_nsec: 436141043,
                    },
                },
            ),
        },
    ],
    opt_recursive: false,
    opt_exact: false,
    opt_no_filter: false,
    opt_debug: true,
    opt_no_traverse: false,
    opt_omit_ditto: false,
    opt_no_hidden: false,
    opt_json: false,
    opt_one_filesystem: false,
    opt_no_clones: false,
    uniqueness: UniqueMetadata,
    opt_bulk_exclusion: None,
    opt_last_snap: None,
    opt_preview: None,
    opt_deleted_mode: None,
    opt_requested_dir: None,
    requested_utc_offset: -05:00:00,
    exec_mode: BasicDisplay,
    print_mode: FormattedDefault,
    dataset_collection: FilesystemInfo {
        map_of_datasets: MapOfDatasets {
            inner: {
                "/": DatasetMetadata {
                    source: "restic",
                    fs_type: Restic(
                        Some(
                            [
                                "/mnt",
                            ],
                        ),
                    ),
                },
            },
        },
        map_of_snaps: MapOfSnaps {
            inner: {
                "/": [
                    "/mnt/snapshots/latest",
                    "/mnt/snapshots/2024-07-19T21:21:28-05:00",
                ],
            },
        },
        filter_dirs: FilterDirs {
            inner: {
                "/snap/snapd/21465",
                "/snap/core20/2264",
                "/snap/snapd/21759",
                "/dev/mqueue",
                "/sys/kernel/debug",
                "/boot/grub",
                "/sys/firmware/efi/efivars",
                "/run/lock",
                "/tmp",
                "/proc/sys/fs/binfmt_misc",
                "/dev",
                "/proc",
                "/dev/shm",
                "/run/snapd/ns/lxd.mnt",
                "/run/user/1000",
                "/sys/fs/cgroup",
                "/snap/bare/5",
                "/sys/fs/pstore",
                "/snap/core22/1122",
                "/var/snap/lxd/common/ns/shmounts",
                "/snap/core20/2318",
                "/dev/hugepages",
                "/sys/kernel/security",
                "/sys/kernel/tracing",
                "/var/snap/lxd/common/ns",
                "/run",
                "/snap/core22/1380",
                "/run/snapd/ns",
                "/boot/efi",
                "/dev/pts",
                "/sys/fs/fuse/connections",
                "/sys/fs/bpf",
                "/snap/lxd/29469",
                "/sys/kernel/config",
                "/snap/lxd/29398",
                "/sys",
                "/var/snap/lxd/common/ns/mntns",
            },
        },
        opt_map_of_alts: None,
        opt_map_of_aliases: None,
        opt_common_snap_dir: None,
    },
    pwd: "/srv/program/httm",
}
───────────────────────────────────────────────────────────────────────────────────
Fri Jul 19 14:13:59 2024  13.4 KiB  "/mnt/snapshots/latest/srv/program/httm/httm.1"
───────────────────────────────────────────────────────────────────────────────────
Fri Jul 19 14:13:59 2024  13.4 KiB  "/srv/program/httm/httm.1"
───────────────────────────────────────────────────────────────────────────────────

You can even have multiple repos mounted.

This should work on Linux, MacOS and BSD, but I haven't tested in BSD.

The new dev branch restic2 works also on FreeBSD. Kudos!

~ httm --debug --backup=restic hello1.txt 
WARN: httm has disabled any MAP_ALIASES in preference to an ALT_BACKUP specified.
Config {
    paths: [
        PathData {
            path_buf: "/home/user/wrk/hello1.txt",
            metadata: Some(
                PathMetadata {
                    size: 60,
                    modify_time: SystemTime {
                        tv_sec: 1721471714,
                        tv_nsec: 873409000,
                    },
                },
            ),
        },
    ],
    opt_recursive: false,
    opt_exact: false,
    opt_no_filter: false,
    opt_debug: true,
    opt_no_traverse: false,
    opt_omit_ditto: false,
    opt_no_hidden: false,
    opt_json: false,
    opt_one_filesystem: false,
    opt_no_clones: false,
    uniqueness: UniqueMetadata,
    opt_bulk_exclusion: None,
    opt_last_snap: None,
    opt_preview: None,
    opt_deleted_mode: None,
    opt_requested_dir: None,
    requested_utc_offset: +03:00:00,
    exec_mode: BasicDisplay,
    print_mode: FormattedDefault,
    dataset_collection: FilesystemInfo {
        map_of_datasets: MapOfDatasets {
            inner: {
                "/": DatasetMetadata {
                    source: "restic",
                    fs_type: Restic(
                        Some(
                            [
                                "/home/user/mnt",
                            ],
                        ),
                    ),
                },
            },
        },
        map_of_snaps: MapOfSnaps {
            inner: {
                "/": [
                    "/home/user/mnt/snapshots/2024-07-20T13:34:57+03:00",
                    "/home/user/mnt/snapshots/latest",
                    "/home/user/mnt/snapshots/2024-07-20T13:35:33+03:00",
                    "/home/user/mnt/snapshots/2024-07-20T13:35:58+03:00",
                ],
            },
        },
        filter_dirs: FilterDirs {
            inner: {
                "/dev",
            },
        },
        opt_map_of_alts: None,
        opt_map_of_aliases: None,
        opt_common_snap_dir: None,
    },
    pwd: "/home/user/wrk",
}
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Sat Jul 20 13:33:58 2024  30 bytes  "/home/user/mnt/snapshots/2024-07-20T13:34:57+03:00/home/user/wrk/hello1.txt"
Sat Jul 20 13:35:14 2024  60 bytes  "/home/user/mnt/snapshots/latest/home/user/wrk/hello1.txt"
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Sat Jul 20 13:35:14 2024  60 bytes  "/home/user/wrk/hello1.txt"
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Prior to FreeBSD 14.0, the default symbolic link for /home, referencing /usr/home was created. bbb2d2ce4220

When home directories are in /usr/home, then new dev branch restic2 does not work with restic default mount, as it seems, that restic saves paths in snapshots as starting with home. Then restic should be mounted with modified path template:

~ restic --repo /home/user/archive mount --path-template "snapshots/%T/usr" /home/user/mnt

kimono-koans / httm

httm on mounted restic repository? #124