Offer flatpak and snap version

[rugk]

Currently you offer a binary for easy usage, but with flatpak and snap becoming a (big?) alternative in the future, it would also be nice if you offer a flatpak and snap version of borg.

[lightonflux]

Quick and dirty snap that does not duplicate dependency and build setup debugging (because it uses the binary releases):

Project structure with rights

root@snap:~/borg-snap# tree -p
.
|-- [-rw-r--r--]  snapcraft.yaml
`-- [drwxr-xr-x]  source
    |-- [-rwxr-xr-x]  arch-switch.sh
    |-- [-rwxr-xr-x]  borg-linux32
    `-- [-rwxr-xr-x]  borg-linux64

1 directory, 4 files

Script for "multiarch"

root@snap:~/borg-snap# cat source/arch-switch.sh
#!/bin/sh

case "$SNAP_ARCH" in
    "amd64") $SNAP/bin/borg-linux64 "$@"
    ;;
    "i386") $SNAP/bin/borg-linux32 "$@"
    ;;
    *)
        echo "Unsupported architecture for this borg snap"
        exit 1
    ;;
esac

snapcraft file

root@snap:~/borg-snap# cat snapcraft.yaml
name: borg
version: '1'
summary: Borg is a backup program with many features
description: |
  The main goal of Borg is to provide an efficient and secure way to backup data. The data deduplication technique usedmakes Borg suitable for daily backups since only changes are stored. The authenticated encryption technique makes it suitable for backups to not fully trusted targets.

grade: devel # must be 'stable' to release into candidate/stable channels
confinement: devmode # use 'strict' once you have the right plugs and slots
architectures: ["amd64", "i386"]

apps:
  borg:
    command: bin/borg-launch
    plugs: ["home", "removable-media"]

parts:
  borg:
    source: source/
    plugin: dump
    organize:
      arch-switch.sh: bin/borg-launch
      borg-linux64: bin/borg-linux64
      borg-linux32: bin/borg-linux32

Tested, works on 32 and 64 bit systems.

Was it really quick? No, because i wanted to include both arch in one package. And i had to setup a snapcraft environment which meant organising an ubuntu installation.

Is it really dirty? Yes. It is larger because the binaries do not depend on the snap "runtime" (ubuntu core that is) and because snapcraft included unnecessary libs (114KB):

|-- [drwxr-xr-x 4.0K]  prime
|   |-- [drwxr-xr-x 4.0K]  bin
|   |   |-- [-rwxr-xr-x  220]  borg-launch
|   |   |-- [-rwxr-xr-x  12M]  borg-linux32
|   |   `-- [-rwxr-xr-x  13M]  borg-linux64
|   |-- [-rwxr-xr-x  321]  command-borg.wrapper
|   |-- [drwxr-xr-x 4.0K]  lib
|   |   `-- [drwxr-xr-x 4.0K]  x86_64-linux-gnu
|   |       |-- [-rw-r--r--  14K]  libdl.so.2
|   |       `-- [-rw-r--r-- 110K]  libz.so.1
|   `-- [drwxr-xr-x 4.0K]  meta
|       |-- [drwxr-xr-x 4.0K]  gui
|       `-- [-rw-r--r--  500]  snap.yaml

I don't think that it is beneficial to compile borg with snapcraft, as this means some hours setup that could be spend improving the normal build system. But it probably makes sense to make two or three snaps for each architecture, because it saves the user 10-20MB disk space.

Another problem is that if you want to make the snap available for all users you need to change from confinement: devmode to confinement: strict. Then borg can only access the users home (not true for /root) and removable media under /media and /run. Afaik there isn't a "plug" for the whole file system.

[rugk]

Hmm, interesting. One snap for each architecture is okay IMHO.

So what about flatpak?

[lightonflux]

Haven't created flatpaks. But its more for GUI apps and desktops than CLI tools.

[rugk]

Okay, maybe, but cannot you also use it for CLI tools? I mean the sandboxing and easy use it also useful for CLI tools.

[Unrud]

So what about flatpak?

Create the file com.gihub.borgbackup.borg.json:

{
  "app-id": "com.github.borgbackup.borg",
  "runtime": "org.freedesktop.Platform",
  "runtime-version": "1.6",
  "sdk": "org.freedesktop.Sdk",
  "command": "borg",
  "finish-args": ["--filesystem=host", "--share=network"],
  "modules": [
    {
      "name": "borg",
      "sources": [
        {
          "type": "file",
          "url": "https://github.com/borgbackup/borg/releases/download/1.0.10/borg-linux32",
          "sha256": "1b9f739bfb71f9c82b1ac4af365e3c54c3f2b16a44efe5284e582f49b4bddac9"
        },
        {
          "type": "file",
          "url": "https://github.com/borgbackup/borg/releases/download/1.0.10/borg-linux64",
          "sha256": "99f889e57630e64a67d0c1a54c056f8d82aac1ccc2cc56c9fb5f5f2710f29950"
        }
      ],
      "buildsystem": "simple",
      "build-commands": ["install -d ${FLATPAK_DEST}/bin", "install -m 0755 -T borg-${VARIANT} ${FLATPAK_DEST}/bin/borg"],
      "build-options": {
        "arch": {
          "i386": {"env": {"VARIANT": "linux32"}},
          "x86_64": {"env": {"VARIANT": "linux64"}}
        }
      }
    }
  ]
}

Build the flatpak file with the following commands:

# Add the repository that contains the runtime and SDK
flatpak remote-add --if-not-exists gnome https://sdk.gnome.org/gnome.flatpakrepo
# Install the SDK
flatpak install gnome org.freedesktop.Platform/x86_64/1.6
flatpak install gnome org.freedesktop.Sdk/x86_64/1.6
# Build and export it into the repo directory
flatpak-builder --arch=x86_64 --repo=repo --force-clean build com.github.borgbackup.borg.json
# Export flatpak file from the repo
flatpak build-bundle --arch=x86_64 repo borg-x86_64.flatpak com.github.borgbackup.borg

Replace x86_64 by i386 to build the 32 bit version. Please note that distributing the flatpak file directly has some drawback , because you have to add the runtime repo manually, install the flatpak file for the correct architecture and don't get automatic updates. You should sign the repo, put it on a web server and create a flatpakref file (that contains the URL of the repo, name of the package, the public key and the URL of the runtime repo).

[ThomasWaldmann]

Hmm, what do we win by just adding another (snap or flatpak) packaging layer above the released single-file binary?

[RonnyPfannschmidt]

@ThomasWaldmann i believe those offer a better controlled and self-contained upgrade experience disconnected from the system

at the current point in time its a really nice convenience for desktop users

[lightonflux]

Also integration into the linux Software Center. But at this point both snap and flatpak are still being improved and the repo setup process for flatpak is not 100% polished.

[RAOF]

The other nice thing about a snap is you can make a self-contained service out of it. I've started working up a borg-server snap borg borg and a listening ssh server (on configurable port) that accepts (configurable) ssh keys and only runs borg serve. I think the snap configuration interface is sufficiently expressive to make this possible now.

This would be particularly if and when the Ubuntu-Core-on-home-routers/NAS promises eventuate.

[thefenriswolf]

@ThomasWaldmann I'd like to revive this topic as I've made a snap package for Borg and published it (with proper name spacing of course). To devs it probably looks grossly duct-taped together, but I'd like to encourage the maintainers to take over from here. Feel free to copy my snapcraft.yaml and modify it to your liking!

[ThomasWaldmann]

@thefenriswolf having not used snaps yet, I am still searching for some kind of rectification of the additional work needed to offer them (e.g. compared to the binaries we already offer).

In your yaml file, you say fuse is broken upstream? AFAIK fuse stuff works, so, can you give details?

[RAOF]

Pretty much just add the snapcraft.yaml to the repository and set up https://build.snapcraft.io/, at least for autobuilds.

@thefenriswolf 's snapcraft.yaml isn't appropriate for this, as it just packages up the existing PyPi binaries, but one that built from source would be.

The attached snapcraft.yaml would be appropriate for a simple manually uploaded release into the stable channel. The work involved there would be:

Once per release: 1) Wait until the binaries are on PyPi 2) Update the ‘version’ field of the snapcraft.yaml to match the new release 3) Run snapcraft snap 4) Run snapcraft push borg_$THE_NEW_VERSION.snap --release stable

This workflow would change if and when the snapcraft.yaml was changed to autobuild from source in build.snapcraft.io. The autobuild would publish into the edge channel; at release time you'd press the button to promote the relevant autobuild into one-or-more of the higher channels - beta/candidate/stable.

If you wanted to be fancy you could promote RCs into the candidate channel, and earlier snapshots into the beta channel, or autobuild a stable branch into candidate and autobuild master into edge but the simplest thing is to autobuild into edge and then promote the release into stable.

[thefenriswolf]

@ThomasWaldmann

In your yaml file, you say fuse is broken upstream? AFAIK fuse stuff works, so, can you give details?

My daily driver is SolusOS and python-llfuse seems to be not in the repo, so every time I try to use borg mount it gives me this error: borg mount not available: loading fuse support failed [ImportError: No module named 'llfuse']

snapcraft on the other hand refuses to build borg if I include llfuse as a dependency, which led me to the conclusion that fuse must be broken somewhere upstream?

Anyway, I totally agree with @RAOF https://build.snapcraft.io/ is an easy way to automate builds and have beta versions and rc automatically published to the appropriate channels for testing.

@lightonflux mentioned setting up snapcraft build environments being labor intensive. That is not true at all, as an ubuntu container [docker, systemd-nspawn, lxc ,...] can be set up in less than 5 minutes.

I also put together a snapcraft.yaml file for building borg from source. It throws a build error at the moment but the rest should be fine!

[bitinerant]

@thefenriswolf - your work on this is really appreciated!

What would be needed to make the "borg-fenriswolf" snap available on the armv7l architecture (Raspberry Pi 3)? I currently use the Borg apt package on two RPi3 boards and it works beautifully, but I would like to try it with Ubuntu Core 16. Or is there already a way to do this? I'm new to snaps.

[bitinerant]

A little off-topic, but on the Ubuntu Core system, I enabled classic Debian packages and ran "sudo apt install borgbackup". That succeeded but "borg --version" crashes with "pkg_resources.DistributionNotFound: The 'borgbackup==1.0.7' ..."

[RAOF]

FWIW, here's a snapcraft.yaml that successfully builds-from-source, and so could be used with autobuilding.

[ThomasWaldmann]

@RAOF not sure why you are building 1.1.3 after 1.1.4 is out with data corruption fix.

[RAOF]

Because 1.1.3 was most recent when I was writing the snapcraft.yaml of course.

I've only just got around to posting it.

[RAOF]

Is it obvious how to update the yaml to 1.1.4?

Whoever uses it should update it to whatever is current at the time!

[ThomasWaldmann]

@RAOF guess it is obvious how to update it to 1.1.4, but you know, people are lazy and some will just use it as is. So a link to an updated version would be appreciated.

[RAOF]

I've revised the gist; it now pulls 1.1.4.