Feature: permit running anything-sync-daemon as an unprivileged user

[tomeon]

Plus a bunch of other fixes and features. Note that this changeset includes major refactorings; I offer this PR with the expectation that its scope is too large to effectively review and therefore not reasonable to merge. However, I've done my best to test and document the changes, and I think that they are genuinely useful, so if you've got the inclination to review this -- many thanks in advance!

Summary of changes:

1. Permit running asd as an unprivileged user. This includes installing a systemd user unit, and loading asd.conf from a different location than /etc/asd.conf. I've also added the script asd-mount-helper, based on psd-overlay-helper, to facilitate mounting and unmounting as a non-root user.
2. Add an anything-sync-daemon Nix package (nix/packages.nix).
3. Add a NixOS module (nix/nixos-module.nix) for configuring and running system and user instances of asd.service, asd-resync.service, and asd-resync.timer.
4. Add NixOS-based systems tests (nix/checks.nix) that verify the correct operation of system and user asd services.
5. Add Nix-based development tooling (nix/devshell.nix), including the shellcheck shell linter and alejandra Nix expression language formatter.
6. Update the GitHub actions configuration to run the NixOS-based systems tests (and replace the shellcheck workflow, as an equivalent job is included in the Nix checks).
7. Translate doc/asd.1 to Markdown (in USAGE.md) and generate the manpage from that Markdown. Also expand the documentation to cover environment variables, and add some relevant links.
8. Factor out sync target iteration (repeated definitions of DIR, BACKUP, TMP, and so on).

I'm still ironing out the GitHub Actions workflow; currently it is quite slow (at least, the NixOS systems tests are). But you can see a recent run here.

Thanks again for your consideration!

[manorit2001]

Hi @tomeon, I really like the changes that you have been doing ( just a bird's eye review based on your commits ). Unfortunately , I haven't been able to spend much time to give a detailed review as you already know the changes are humongous and it's not exactly possible to review but I do see that you are active and doing some good improvements based on your commits. I will see if I can review few of the stuff but I can't commit at this stage.

Although, I would like to know a few things. Is there any breaking change in this full PR w.r.t the current asd version and something that you would like to summarise based on that? The debug capability handling along with fixes and added nix os support is always welcomed but the only thing that I am mostly worried about is the compatibility with the current asd version. I see that asd is a very critical piece of software and people have actually suffered data losses in the past due to the refactor and the syncing process, if you know anything that you have done to drastically alter that process ( along with changing any names for the backup folders or anything), I would recommend giving a summary of it and maybe explaining the changes here ( or if possible pointing to those specific commits along with the explanation ) as that'd help us see the scope of the upgradation process from current asd to the new one. Also, I see that most of your testing has been NixOS based, is there anything that you have done for arch or is there some difference in that? I haven't really ventured into NixOS so feel free to impart your knowledge

[tomeon]

Hello @manorit2001 -- thank you very much for your feedback.

Is there any breaking change in this full PR w.r.t the current asd version and something that you would like to summarise based on that?

If you subscribe to Hyrum's Law, then I'm afraid there are breaking changes in just about every commit :wink: . More seriously: a few of my changes are liable to break some setups, particularly the changes having to do with loading asd.conf and its snapshot. However: setups broken by these changes are probably doing something (a) naughty (like messing with anything-sync-daemon's internal state) or (b) inadvisable (like launching long-running processes). In case (a), users shouldn't do that, and in case (b) there's the workaround/escape hatch of specifying a longer timeout in ASDCONFTIMEOUT.

I see that asd is a very critical piece of software and people have actually suffered data losses in the past due to the refactor and the syncing process, if you know anything that you have done to drastically alter that process ( along with changing any names for the backup folders or anything), I would recommend giving a summary of it and maybe explaining the changes here ( or if possible pointing to those specific commits along with the explanation ) as that'd help us see the scope of the upgradation process from current asd to the new one.

I am working on a CHANGELOG.md that will cover all significant changes, highlighting especially any breaking changes. I have not changed the names of backup folders, and any path-related changes are strictly opt-in, either by (a) running anything-sync-daemon as an unprivileged user (so should not affect existing installations, though I cannot say this for certain) or (b) setting ASDNOV1PATHS=1 or similar, which is not the default.

I see that most of your testing has been NixOS based, is there anything that you have done for arch or is there some difference in that? I haven't really ventured into NixOS so feel free to impart your knowledge

I hadn't done anything for Arch Linux, though, inspired by your mention of it, I've added a Vagrant configuration that spins up an Arch Linux machine and deploys anything-sync-daemon. Found a few bugs in my other changes as a result, so thanks for that :slightly_smiling_face:.

Part of what I like about NixOS systems tests is how the testing framework is distributed with nixpkgs, so if you've got the Nix package manager and an internet connection, you're only a few steps from rockin' and rollin' with pkgs.testers.nixosTest. No need to explicitly install additional tooling, or even configure much of anything outside the actual stuff of interest (in this case, settings in the config.services.asd "namespace"), as nixpkgs handles all the QEMU/KVM/networking/etc. stuff out-of-the-box.

However, I am happy to look into rigorous testing on Arch Linux, too. Haven't found any analog of pkgs.testers.nixosTest for Arch, though perhaps Serverspec or InSpec or similar would do the trick. WDYT?

[tomeon]

I have added the changelog.

[tomeon]

Also, latest run is passing; see here.

[manorit2001]

However, I am happy to look into rigorous testing on Arch Linux, too. Haven't found any analog of pkgs.testers.nixosTest for Arch, though perhaps Serverspec or InSpec or similar would do the trick. WDYT?

There is one test that I am particularly worried about first of all with this PR, it is basically the upgrade process, if possible could you check if we use the current asd version, stop asd ( this will happen as the part of upgradation process ) and then update your asd version to the new one with same configs as it was previously, I want to know that nothing breaks as a part of that.

W.r.t inspec or serverspec, I don't have any strong preference as I haven't worked on them, I would recommend you to choose whatever feels scalable to you for the project, would see the changes based on that.