Old-style single zfs-autobackup executable

[lciti]

Until recently, zfs-autobackup was a single python script that could be easily moved around, e.g. copied into /usr/local/bin and executed from there. The code has now been reorganised and split into several python files. This makes the code easier to understand, maintain, and reuse. The normal way to install the tool is now to use pip or easy_install, which install it in the standard python modules folders. However, in some cases one may prefer the old-style single-file approach. This can still be easily obtained thanks to the zipapp module. One only needs to cd into the root folder of this repository (either obtained using git or downloaded as a zip file) and run the following:

mkdir temp
cp -a zfs_autobackup temp
python3 -m zipapp temp -p /usr/bin/python3 -o zfs-autobackup -m zfs_autobackup:cli
rm -r temp

This produces a zfs-autobackup file that can be executed directly from the shell (as ./zfs-autobackup or as zfs-autobackup if installed in a PATH folder such as /usr/local/bin). If @psy0rz and the other users find this useful, it can be added to the script folder or to the documentation.

Edit Sept 2022: The zfs_autobackup code has been slight restructured since I wrote this post and the cli function is now under zfs_autobackup.ZfsAutobackup. See my comment below for updated instructions.

[psy0rz]

Thats indeed a nice tool. I'll try to figure out if i can generate it automatically for each release via github, so we can have a direct-download link in the github release-page.

[DvdGiessen]

Note: We've been using PyInstaller for similar reasons; it builds a single executable with no external dependencies as PyInstaller also packages the Python interpreter and all its libraries into a single file.

# pip install --upgrade pyinstaller zfs-autobackup
# pyinstaller --onefile $(which zfs-autobackup)
# ls -lah dist/zfs-autobackup
-rwxr-xr-x 1 root root 7.1M Sep 16 08:19 dist/zfs-autobackup

I'm running zfs-autobackup in a SmartOS global zone where Python is not available, thus having a single binary works great here: I build zfs-autobackup in a non-global SmartOS zone, and then copy the resulting binary into the global zone path.

Since SmartOS is listed in the README already and by default it doesn't have Python available in it's global zone (which is often where zfs-autobackup would be useful to have since the global zone has access to the entire ZFS pool), this might be an interesting option to add to documentation.

While it's possible to distribute these binaries, since these are platform-specific (different from the zipapp files) I think it's better to just document how to use it.

[psy0rz]

@DvdGiessen yes i think its a good idea to add documentation for that.

However this is different from the zipapp method mentioned above: pyinstaller will actually create a binary which can only be used on the platform it was created on. So indeed we can not generate that automaticly via a github action.

I myself actually went the lazy way and just installed python in the global zone:

[root@smartos3 ~]# uname -v
joyent_20210701T204427Z
[root@smartos3 ~]# python3 -V
Python 3.5.1

[psy0rz]

@lciti i get this:

psy@ws1 ~/zfs_autobackup % ./zfs-autobackup 
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "./zfs-autobackup/__main__.py", line 8, in <module>
ImportError: attempted relative import with no known parent package

[psy0rz]

If i change some stuff and remove all the "zfs_autobackup." prefixes from the imports it works, but then it wont function as a proper python module anymore..sigh..

If somebody knows how to do this properly let me know.

[psy0rz]

I think its this problem: https://bugs.python.org/issue26277

[psy0rz]

I wasted several hours on this. If someone wants to fix it, create a PR.

I cant get it so that zfs_autobackup is both a standaard python 2 and 3 module, AND functions in zipapp. (without ugly hacks like a search/replace over the sourcecode or something like that)

[lciti]

Hi psy0rz I am sorry you have spent a lot of time on this. I am not sure what is the problem you are having. I have tried again the four lines in my first post and it works for me (I have Python 3.8.10 in Ubuntu 20.04). Are you trying to make it work on your desktop or on the github automatic build system?

[psy0rz]

I tried your changes to no avail:

psy@ws1 ~/zfs_autobackup % python3 -m zipapp zfs_autobackup  -p "/usr/bin/env python3" -o zfs-autobackup  ; ./zfs-autobackup
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "./zfs-autobackup/__main__.py", line 8, in <module>
ImportError: attempted relative import with no known parent package
psy@ws1 ~/zfs_autobackup % python3 --version                                                                                
Python 3.8.10
psy@ws1 /tmp % lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.3 LTS
Release:    20.04
Codename:   focal

I'll try upgrading and on another system. Maybe its something local.

[psy0rz]

I've made the wrong assumption that the directory structure would be temp/.py instead it should temp/zfs_autobackup/.py and then its fine. (doh)

[jgod]

There's also Nuitka

[neodc]

Any news on this ? I don't have (and don't want to install) pip or easy_install on the host where I use zfs-autobackup (as it's a minimal OS with only LXD container doing the real jobs), for now I use the version 3.0 but I'd like to update at some point.

[psy0rz]

Sorry no news yet, first have to finish zfs-autoverify/zfs-check, after that i'll look into it.

@neodc what if you create a venv outside the container and use that inside the container?

[lciti]

@neodc Have you tried my instructions in the first post? It's manual process (rather than automatically triggered after each git update) but should do what you need and it's literally 4 instructions (after you have downloaded and unzipped the repo).

[neodc]

First, sorry for the (huge) delay.

I'm starting to get warning and error since I updated to Ubuntu 22.04 so I tried to use your recommendations to update.

@psy0rz From what I can see the venv hard code the path of where it was generated, so that would not work I think.

@lciti The zipapp command give me a not functional bin, when I run it (the created bin) I get an error (see all the command I used bellow). I suppose the zfs_autobackup:cli need to be adapted but I didn't find what to use (I'm no python developer).

neodc@neodc-pc:~/tmp$ git clone 'https://github.com/psy0rz/zfs_autobackup.git'
Clonage dans 'zfs_autobackup'...
remote: Enumerating objects: 3012, done.
remote: Counting objects: 100% (66/66), done.
remote: Compressing objects: 100% (47/47), done.
remote: Total 3012 (delta 41), reused 36 (delta 19), pack-reused 2946
Réception d'objets: 100% (3012/3012), 736.72 Kio | 7.15 Mio/s, fait.
Résolution des deltas: 100% (2020/2020), fait.
neodc@neodc-pc:~/tmp$ cd zfs_autobackup/
neodc@neodc-pc:~/tmp/zfs_autobackup$ mkdir tmp
neodc@neodc-pc:~/tmp/zfs_autobackup$ cp -a zfs_autobackup tmp
neodc@neodc-pc:~/tmp/zfs_autobackup$ python3 -m zipapp tmp -p /usr/bin/python3 -o zfs-autobackup -m zfs_autobackup:cli
neodc@neodc-pc:~/tmp/zfs_autobackup$ ./zfs-autobackup --version
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "./zfs-autobackup/__main__.py", line 3, in <module>
AttributeError: module 'zfs_autobackup' has no attribute 'cli'

[lciti]

@neodc The code has been slightly restructured from the time when I wrote my initial post. The instructions required are now:

git clone 'https://github.com/psy0rz/zfs_autobackup.git'
cd zfs_autobackup/
mkdir temp
cp -a zfs_autobackup temp
python3 -m zipapp temp -p /usr/bin/python3 -o zfs-autobackup -m zfs_autobackup.ZfsAutobackup:cli
rm -r temp
./zfs-autobackup --version

[neodc]

Looks like it's working with this (at least I get the version string even after scp it to to destination server), but then I was on version 3.2-beta1, I don't like the idea of using beta version for my backup so I checkout the v3.1.3 tag and could "compile" using the original version.

Thanks for the help

[psy0rz]

3.2-beta1 will be released soon, its safe to use. :)

Edwin

[neodc]

I set it up this weekend on 3.1.3 and it work perfectly now, so I don't see any reason to rush to a beta version =)

[psy0rz]

yep, no if it works then keep it that way :)

[ghan1t]

@neodc The code has been slightly restructured from the time when I wrote my initial post. The instructions required are now:

git clone 'https://github.com/psy0rz/zfs_autobackup.git'
cd zfs_autobackup/
mkdir temp
cp -a zfs_autobackup temp
python3 -m zipapp temp -p /usr/bin/python3 -o zfs-autobackup -m zfs_autobackup.ZfsAutobackup:cli
rm -r temp
./zfs-autobackup --version

Please be aware that the main branch can be broken at times. It is safer to checkout the newest tag with:

git clone --branch <tag_name> 'https://github.com/psy0rz/zfs_autobackup.git'