Introduce a `run-with` option

[damoclark]

Congrats on creating a really great companion for Restic. :)

As a suggested improvement, it would be useful to be able to configure the use of utility commands for executing the restic binary, either within a restic profile, or even globally, allowing more customisable behaviour for specific operating systems and/or environments.

For example:

Always run restic as another user

default:
  run-with: sudo -u restic

Which would generate commands like:

sudo -u restic restic backup ...
sudo -u restic restic forget ...
etc

or

Perform pull rather than push backups

backup:
  run-with: "ssh hostname"

Which would generate the command:

ssh hostname restic backup ...

or

For backups to local filesystem repository, set your primary unix group so all users of a group can read/write the repository files using restic

backup:
  run-with: "sg - backups"

Which would generate the command:

sg - backups restic backup ...

or

Use a distributed lock manager to execute the backups

backup:
  run-with: "flom --lock-mode CW --"

Which would generate the command:

flom --lock-mode CW -- restic backup ...

Parameters to the utility commands could be altered through resticprofile variables and utility commands could be chained together. For example:

Use a distributed lock manager to execute different restic commands depending on lock semantics

prune:
  run-with: "sudo -u {{ .sudo_user }} flom --resource-name '/{{ .Profile.Name }}' -l {{ .lock_mode }} --"

Which would generate the command:

sudo -u backups flom --resource-name '/myprofile' -l EX -- restic prune ...

Going a step further again, the full restic command itself, could be exposed as a built-in variable {{ .Command }}. .Command could be an object that generates shell-escaped or quoted version of the command.

backup:
  run-with: "/bin/sh {{ .Command.quoted }} | tee -a .local/log/{{ .Profile.Name }}.log"

Which would generate the command:

/bin/sh 'restic' 'backup' ... '/My Directory' | tee -a .local/log/myprofile.log

or

backup:
  run-with: "/bin/sh -c '(date ; {{ .Command.escaped }} ; date)' | tee -a .local/log/{{ .Profile.Name }}.log"

Which would generate the command:

/bin/sh -c '(date ; restic backup ... /My\ Directory ; date)' | tee -a .local/log/myprofile.log

[jkellerer]

Hi @damoclark,

Thanks a lot for the great feedback on possible use-cases when the restic commandline supported customisations. In fact we have a few cases (#153 - redirect to file/command & #69 - remote shell) that partially address your examples.

A custom shell environment declaration (e.g. with setup & teardown) and/or a commandline-template as you proposed might be quite useful and a lot more flexible.

E.g. something like this (just a very quick draft):

version: 2

environments:
  remote-shell:
    setup: 
      - "ssh {{ .Env.BACKUP_GATEWAY_HOST }}"
      - "ssh $TARGET_HOST"

  sudo-bash:
    shell: "bash"
    setup: 
      - "sudo -u {{ .Env.BACKUP_USER }} flom --resource-name '/{{ .Profile.Name }}' --"
    run: "(date ; $COMMNAND ; date)"

profiles:
  backup-remotes:
    backup:
      # default shell for backup
      shell:
        name: remote-shell
        foreach-vars:
          TARGET_HOST_FILE: "/hostlist.txt"

      run-before:

        - "echo running {{ .Profile.Name }} on $TARGET_HOST"

        - run: "echo running {{ .Profile.Name }} from $(hostname)"
          shell: "bash" # override shell to run locally

@creativeprojects , what do you think?

Looks like it could not only cover this request but implement #69 pretty closely with moderate efforts. We'd need to check that most is covered and that it is maintainable.

The way how variables can be used (including list [table?, matrix?]) will need a bit more planning.

E.g. looping is something not directly related to this topic but useful. It may also be good to support it in the profile with environment variables rather than tying it to the shell:

profiles:
  backup-remotes:
    run-foreach:
      name: "TARGET_HOST"
      delimiter: "\n"
      from-file: "/hostlist.txt"

[jkellerer]

Note: Shell env with setup & teardown may be difficult to implement (when existing behaviour must remain the same): Either needs an interactive run mode, a script or setup & teardown for every command (restic & hooks) which is not good for use cases like custom locking and also not very efficient but easy to implement in the current infrastructure.

[creativeprojects]

Yeah that's an excellent idea. I was thinking about doing something similar some time ago but I wasn't coming up with an easy way of configuring it.

I'm just back from holiday: I'll think about it this week.

[jkellerer]

Btw. Maybe we split it into 3 parts:

• Support for custom shell env without setup (just allow to set command line templates for restic and hooks, like described in this case)
• Support for setup in shell env (likely requires some sort of host process to communicate with, e.g. resticprofile itself)
• Support for looping on variables.

[damoclark]

@jkellerer I was going to suggest some caution in relation to over-engineering the configuration in response to my initial suggestion. I wonder whether a good guide for what to introduce into resticprofile and perhaps what to leave out might be in how difficult or even possible it is to achieve something in other ways.

The primary motivator for me in submitting this suggestion was to introduce distributed locking into my distributed backup strategy across all my hosts. Resticprofile already includes host-based locking which is great. But I have a fleet of hosts that backup to a small number of repositories and I wanted to be able to easily co-ordinate those backups without I/O contention, along with routine maintenance tasks, such as forgets and prunes which require exclusive access to the repositories - all without guessing timeslot windows. Flom is something I've used before and worked very well. But integrating it with Resticprofile is not simple to do (i.e. having to preface every invocation of resticprofile with the flom command and arguments), and if I want to use Resticprofile's scheduling ability, it is essentially impossible as it stands.

But rather than be specific, my suggestion was to address the shortcoming that would allow the use of any utility commands, such like flom, to be incorporated into Resticprofile configuration through a templated command line.

So I endorse the first of your 3-part suggestion above on this basis. I tried to propose a syntax that was both intuitive and simple, yet flexible enough for most cases.

With your suggestion for a custom shell environment, I was initially a little skeptical. However, I can see the benefits of being able to declaratively configure the shell environment (i.e. env vars) as part of Restic profile. This again is not something easily done outside of Resticprofile, or possible with the scheduling. So I agree it would be worthy inclusion.

The only one I'm not quite sure about the looping. Although quite likely there are instances that have not occurred to me. :)

Just another consideration with regard to locking...

Given the affordances of Restic, I do wonder whether my use case could be quite common. And unfortunately, Restic itself doesn't really include synchronisation between different distributed tasks - if the repository is locked, the restic command just bails rather than waits its turn. This is a nice gap that Resticprofile could also fulfil out-of-the box. Flom does have an external C/C++ library that you can link against. My knowledge of go is quite low, but I seem to recall the ability to statically link with external C libraries. Thus, an attractive feature of Resticprofile might be to introduce distributed locking directly within its declarative syntax, leveraging the capabilities of Flom.

If it is something you might like to explore, I can open a separate issue request if you like.

Thanks again for engaging with my suggestions.

Regards, Damo.

[creativeprojects]

I was going to suggest some caution in relation to over-engineering the configuration in response to my initial suggestion.

I definitely share your concerns here: as a matter of fact this kind of configuration has been proposed before. The idea was to call restic via caffeinate to avoid a Mac from sleeping during long backups.

A prevent-sleep option was introduced instead.

Now about Flom itself: calling a C library from Go is not actually straightforward, and in some cases you can lose the cross-compilation ability of the go builder. Which means I'd rather avoid using it.

There might be some other libraries available for distributed locking though.

But the thing is, what is going to be the next requirement for wrapping restic into another process? 😆

[damoclark]

@jkellerer said

Thanks a lot for the great feedback on possible use-cases when the restic commandline supported customisations. In fact we have a few cases (https://github.com/creativeprojects/resticprofile/issues/153 - redirect to file/command & https://github.com/creativeprojects/resticprofile/issues/69 - remote shell) that partially address your examples.

It's no accident I chose those examples for the command line template. ;)

@creativeprojects said:

Now about Flom itself: calling a C library from Go is not actually straightforward, and in some cases you can lose the cross-compilation ability of the go builder. Which means I'd rather avoid using it.

I did wonder about that. I did a little reading on cgo, and I see what you mean and understand completely.

I definitely share your concerns here: as a matter of fact this kind of configuration has been proposed before. The idea was to call restic via caffeinate to avoid a Mac from sleeping during long backups. A prevent-sleep option was introduced instead.

Perhaps it is worthwhile to introduce a specific config option in that instance. Preventing sleep is a common problem for desktop users, and presumably, you created a cross-platform solution, that is easily found in your documentation.

Broadly, a philosophy I often advocate as a guide and borrowed from far smarter people than I such as Larry Wall, is to design to make the simple things easy, and the difficult things possible.

The prevent-sleep option offers an easy solution to the simple problem of preventing sleep.

But the thing is, what is going to be the next requirement for wrapping restic into another process? 😆

Yes, you would want to be careful not to bloat the configuration dictionary with too many options (i.e. over-engineer). Thus, the command line template approach might contribute to making difficult things possible. Essentially a broader catch-all.

If I may, a couple of things you and @jkellerer might like to consider with Resticprofile going forward. The documentation that you have written for RP is quite good. However, one gap that I observed when I was learning how to use it was the limited examples. When learning something new, most people find it easier to learn through concrete examples, linked back to abstract concepts. Rather than the other way round. When I read some of the config options, it shows what type it expects, but it isn't always clear what it means. For example:

stdin-command: string OR list of strings

What does it mean to provide a list of strings for a Restic option that normally only takes one? An example of how you would use it this way would help to understand this abstract concept.

I am wondering if you might consider the following two suggestions:

1. Prepare some more examples, especially for those options of Resticprofile that aren't documented as part of Restic itself
2. More importantly, invite your user-base, either on the Resticprofile site, but also the Restic discussion forums to contribute some of their redacted resticprofile configurations.

By implementing suggestion 2, you will get a clearer sense of how people are using your tool, but also some great configuration examples you can share with the Resticprofile community, making it even easier to get started with Resticprofile.

And if you were to implement command-line templates into Resticprofile, you might get to see how (if much at all), people make the difficult possible. This means that if usage patterns emerge that indicates a meaningful gap in the configuration options, you could then add it in future releases. So it could also help guide your development priorities, like for #153 or #69 .

Finally, @creativeprojects and @jkellerer what are your thoughts on the command template syntax I initially proposed? Are their problems that I overlooked, or is there a simpler approach to the syntax? Shell escaping is always a challenge, even for seasoned UNIX users. @jkellerer I can see how configuring the shell environment ahead of execution of hooks and restic itself would be quite useful. Maybe that could be a second phase implementation, depending on how the command template is received and used?

[jkellerer]

@damoclark , thanks a lot for taking the time and sharing your feedback.

Guess we all see the risk of over-engineering, its always good to be reminded to stay focussed 😀

On the other hand backup can require many things to be done depending on what to achieve, resticprofile should be flexible enough to allow most of these cases and support common things directly and if possible in a simple way. This request enables things that existing shell hooks can't do in the same way.

With regards to locking we do have look-wait support, that includes retrying restic commands when locking fails on the repo.

The missing part with regards to locking really is distributed load management. Eventually it would be reasonable to implement load management directly. However, shell hooks and a customisable command line can also do the trick and as @creativeprojects mentioned, you never know what else is required.

Note: Besides being able to manage backup centrally, one motivation of pull based backup (#69) is to have controlled load. Still load management is likely easier to achieve without it.

---

With regards to cgo, I think it would be better to avoid it. resticprofile should remain a single standalone binary that is easy to compile and has little dependencies to the host env.

[jkellerer]

Basically I think step 1 (custom command line) should be added and step 2 (custom environment setup) likely as well but the focus should stay on step 1. How it is implemented exactly is something to be decided but it should also fit in the general concept and possible future.

Escaping can be challenging, so the exact syntax might differ from your proposal. In the end it needs to work properly with other features.

Custom setup (step 2) is not trivial so it may not come soon, but I think for more complex tasks it is required.

Looping already exists, we call it groups. So nothing really needs to be done here when thinking about it again (besides, maybe wildcard support in the group definition).

---

Regarding documentation: It is still rather new and progressing. It also takes lots of efforts to get it right, so feedback and contributions, e.g. in form of good examples, better wordings or just by mentioning what is unclear is very helpful.

What we probably can't change is adding too much details to the config reference as it would also make it hard to lookup information when you already know what you want to achieve. But we definitely need more pages with examples on specific topics.

[creativeprojects]

Thank you for your feedback 👍🏻

I agree we can go ahead with step 1 which shouldn't be too complicated.

I came across this project this week, I wonder if that could help us with step 1 and/or 2: https://github.com/mvdan/sh I haven't looked at it yet though.

After that we might want to review distributed locking later maybe, but step 1 should cover it for the time being.

Now, about the documentation. That was kind of an afterthought actually. I started this project to learn Go, and for that matter that was a success. By that I mean I'm now using Go in a new job 😆

Then people started using this project and I slowly started copying some of my configurations as a documentation. And here we are a few years later, where the newer features are actually better documented than the basics 😛

It's been a while that I want to review the documentation but there's always something else coming up

[jkellerer]

Self note: https://github.com/photostorm/pty

(This is the thing that would be needed to have a persistent terminal session)