One-time templates/"seed" files

xeruf commented 2 years ago

Is your feature request related to a problem? Please describe.

I have some config files which I want to setup initially on a new device (e.g. Musescore config) but then not track the resulting configuration changes, as these will be device-specific.

Describe the solution you'd like

A third option of alternative files/templates, a type of template maybe called "seed", which will simply copy over to the proper filename only if that file does not exist, so .config/doom/user.el##seed.esh would be processed into .config/doom/user.el upon updating only if the latter does not exist.

Whether the seeding uses template processors or alternative file conditions I don't really mind.

Describe alternatives you've considered

I currently use the following workaround in my bootstrap:

template() {
    local dir=$(dirname "$1")
    local file=${1/.template}
    test -d "$dir" && ! test -f "$file" &&
        cp -iv "$1" "$file"
}
template "$XDG_CONFIG_HOME/doom/user.template.el"
template "$XDG_CONFIG_HOME/MuseScore/MuseScore3.template.ini"
...

TheLocehiliosan commented 2 years ago

This is an interesting idea. 👍

masukomi commented 2 years ago

I've been dealing with this concept for years now in my dotfiles. the "right" solution is more complex than that.

I think "seed" files are the right desire,but wrong mental paradigm to address this problem. We can't think of this as as "one-time setup", which is what seed files are generally all about. You run them once and never again.

We need to think of this as idempotent bootstrap files that can be run as many times as needed, optionally pre/post-configure "new" things, and not screw up "old" things.

Issues to consider

Let's take it as a given for the following that we have a user with 2+ devices and they're keeping their configs in sync via yadm, and that we're just discussing bootstrap type operations that involve the setup of new tools.

Issues that must be addressed / considered

configs are constantly growing. The occasional removal of a package / tool is something we're going to ignore for now. For now, assume that a user will eventually add something to their dotfiles. Some new tool you've found that's useful and you want on both devices, and will use yadm to bring over.
installing some tools requires follow-up configuration.
file presence checking is an insufficient test. For example, on macOS intel vs m1 they've changed the install location of many things even in the same version of the OS. If we consider one machine being Linux and one mac the problem is even larger. I'm going to assume all devices in a person's yadm are using the same package managers (like Homebrew on the mac) but that could be configurable. E.g. apt-get on my linux box and homebrew on my mac. While important, I think that's a >= v2 problem.

File presence testing is fine for config files, but it's no good for executables installed by package managers or that may be present in one version of an os but not another.

If you want to test if a tool is installed you need to test the output of hash or command -v or which.
not all tools are installed the same way I'm a homebrew user, but there are a lot of great tools out there that aren't in homebrew. Ditto for every other package manager. So, flexibility of installation methods is important, but facilitating the use of your chosen package manager is too.
failing to install certain tools can mean STOP THE CAR!!!! for example, if during my bootstrap git has failed to install, continuing is only going to add chaos. Better to stop early, figure out the problem, try again.
dylibs suck & need to be shoved in env vars LDFLAGS, CPPFLAGS, & PKG_CONFIG_PATH need to be configured correctly to include all the relevant paths of all the things you've installed so that other things you want to install can be compiled.

So, with all that in mind, you need a mechanism by which you can add your new shiny package / tool / whatever to your yadm bootstrap functionality, and have it get installed via the correct means, and then optionally configured when you run it on the "other" computer.

I think the solution I've been using is a step in the right direction. I'm not proposing this is the solution yadm should use, but putting it out here as inspiration.

my bootstrap file contains 2 important functions
- is_installed tests if there is any executable on the PATH matching the supplied command.
- mayclone there are a bunch of git repos i want cloned to every machine. takes a directory where the thing would be installed and the clone paths. For example.
  
  tests if the directory exists, and if it it's a git repo, clones a repo there if it doesn't. Example.
```
mayclone ~/workspace "git@github.com:masukomi/git-rook.git"
```
  note: as i continued writing this i realized that i really ought to be treating git as if it were a package manager (see below)
I have a separate file for things installed via the package manager this is the more interesting one for this discussion. it's called early on by the bootstrap file to set up tools i rely on to complete the bootstrap process.
- maybrew same idea as maclone, but it's asking homebrew if it has installed the package, and then installing it.
- install_or_die as noted above some things are worth stopping the whole process if it doesn't work.
- is_installed used by maybrew. The key difference here is that tools that need followup configuration after install need a way to test if it's installed without installing in order to set up an if block. Maybe there's a pre-install step. Maybe post-install. Maybe both. Doesn't matter. providing a simple test for installation solves both.

If we go back to my original list this solves points 1 - 5. I have a solution for 6 but it's beyond the scope of anything I'm currently hoping for for yadm.

To summarize

For most config things, file presence tests aren't really relevant because yadm (git) already handles that. If a file isn't present it'll just come into existence when you next run yadm pull on the "other" machine. However, they are relevant if what you're testing is if a collection of things has been installed (cloned down for example).

The problem is to support the installation and optional configuration of executables. To do this really well we need to consider that there are installations via a package manager and manual installations.

I think we should have

a function that tests if an executable has been installed using means other than testing file locations.
a concept of separate install scripts for package managers. for each package manager there should be package specific helper functions
- test if x is installed via package manager
- helper function that tests & installs if not
- install-or-die type helper function

I'm thinking that this could be implemented by having a default ~/.config/yadm/bootstrap file (presumably written in bash) that provides the simple "is it installed?" type functions that are package manager independent. Then have it perform a series of if tests that check for ~/.config/yadm/<package_manager_name> and executes it if present. where package_manager_name is apt-get, homebrew, yum etc.. For each there would (eventually) be a default file with the aforementioned helper functions tailored to the package manager. I think git should also be considered a "package manager" for the purposes of yadm bootstrapping.

Users would be expected to use the helper functions and just append the things they need to install to the end of it. If they want to trash them, no big deal.

masukomi commented 2 years ago

note: my prior comment doesn't address @xeruf 's handling of $XDG_CONFIG_HOME being a ... variable variable. If you're using yadm on systems that define it differently then you can't just add the files to yadm and "let git handle it".

This presumes that there are some systems out there that actually put it somewhere other than ~/.config ? Because if every system puts it there then we can just add it to yadm (my current solution). $XDG_DATA_DIRS does seem like something more widely variable between systems.

Maybe, through the filter of my proposal above, XDG stuff could just be treated as if it were another "package manager" as, for the purposes of this, a "package manager" is just "a thing which installs stuff in a specific manner to a place of its choosing"

xeruf commented 2 years ago

The setting of variables like XDG_CONFIG_HOME is something that is part of my dotfiles and in sync across all systems and does not usually change, and I think that is typical, so it should not be much of a concern.

yadm-dev / yadm