search

twpayne / chezmoi

Manage your dotfiles across multiple diverse machines, securely.
https://www.chezmoi.io/
MIT License
13.24k stars 493 forks source link

Thoughts on collection of boilerplates for somewhat common variables #1474

Closed twpayne closed 2 years ago

twpayne commented 3 years ago

Discussed in https://github.com/twpayne/chezmoi/discussions/1439

Originally posted by **bradenhilton** September 20, 2021 Recently I noticed the section on determining the chassis type of the current machine was added to the how to, and it got me thinking about similar things that might be nice to have as variables. I was wondering what everyone thought about perhaps creating some kind of centralized repo containing boilerplates/snippets that can be used to achieve similar things. I'm mainly focusing on things that can be determined in an abstract way (if the variable itself was highly specific to your needs and your machine(s), there would be little point in adding it to a repo like this) and can also be done on all or most platforms supported by chezmoi. Some variables that would be nice to have for me personally that I can think of off the top of my head: - Whether the current machine is a laptop/desktop/mobile - already achieved with the chassis type snippet plus an additional check `if eq .chezmoi.os "android"`. - The CPU core count of the current machine - mpv assigns duplicate copies of the thumbnailer script that I use to additional threads in order to generate thumbnails faster. This can bog down an older machine if the number of thumbnailer threads is greater than ~1/2 of the core count and the CPU is also being used to play the video. - The number of displays connected to/active on the current machine - I typically use three displays with my desktop machine (but not always), and no external displays with my ThinkPad or MacBook Pro. Many applications allow settings to the effect of "Always open on display X" etc. It also occurred to me that some of these could be implemented in chezmoi itself, depending on demand/ease of implementation/scope of course. Feel free to offer your thoughts and also any ideas for useful variables you might have.
VorpalBlade commented 3 years ago

Side note with regards to the chassis snippet: unfortunately it requires a very recent systemd. The systemd version (245) on Ubuntu 20.04 LTS (current LTS version) does not support JSON output. Out of my systems, only my Arch desktop (systemd 249) has the json option.

Perhaps it should be updated to not require such modern features.

bradenhilton commented 3 years ago

@VorpalBlade do all of your Linux systems have /sys/devices/virtual/dmi/id/chassis_type?

@twpayne Have you given this any more thought? I have a few things I'm pretty much ready to share but I'm not sure where they should go or how they should be structured.

VorpalBlade commented 3 years ago

@bradenhilton My raspberry pis do not have this:

$ cat /sys/devices/virtual/dmi/id/chassis_type
cat: /sys/devices/virtual/dmi/id/chassis_type: No such file or directory
$ hostnamectl 
   Static hostname: rpi
         Icon name: computer
        Machine ID: XXXXXXXXXXXXXXXXXX
           Boot ID: XXXXXXXXXXXXXXXXXX
  Operating System: Raspbian GNU/Linux 10 (buster)
            Kernel: Linux 5.10.63-v7l+
      Architecture: arm

Nor do Android devices (they obviously don't have hostnamectl as well). All other Linux computers I have access to are x86-64 based and do have that file.

I would also assume that any Linux distros that don't use systemd (e.g. Gentoo) will not have hostnamectl. I'm also curious about the various BSDs, but I haven't used those for over a decade at this point.

bradenhilton commented 3 years ago

@VorpalBlade Ah got it! Seems like it's a better solution than hostnamectl, but it's not perfect - many tablets report a chassis_type of 3, which technically indicates a desktop.

Unfortunately I don't have a Raspberry Pi and wasn't aware of this. What does chezmoi data look like on one of your Pis? Anything that would indicate a Pi which can be checked for in a conditional?

Re Android - surely an if eq .chezmoi.os "android" check is sufficient in this case? Or do you have some bare metal Android installs on desktops/laptops or something like that?

If your particular use case is quite complicated it might be better to just write a custom script and use it in an output.

VorpalBlade commented 3 years ago

Raspberry Pi

Sure, here is an example of the data on a Raspberry Pi:

{
  "chezmoi": {
    "arch": "arm",
    "fqdnHostname": "rpi",
    "group": "arvid",
    "homeDir": "/home/arvid",
    "hostname": "rpi",
    "kernel": {
      "osrelease": "5.10.63-v7l+",
      "ostype": "Linux",
      "version": "#1457 SMP Tue Sep 28 11:26:14 BST 2021"
    },
    "os": "linux",
    "osRelease": {
      "bugReportURL": "http://www.raspbian.org/RaspbianBugs",
      "homeURL": "http://www.raspbian.org/",
      "id": "raspbian",
      "idLike": "debian",
      "name": "Raspbian GNU/Linux",
      "prettyName": "Raspbian GNU/Linux 10 (buster)",
      "supportURL": "http://www.raspbian.org/RaspbianForums",
      "version": "10 (buster)",
      "versionCodename": "buster",
      "versionID": "10"
    },
    "sourceDir": "/home/arvid/.local/share/chezmoi",
    "username": "arvid",
    "version": {
      "builtBy": "goreleaser",
      "commit": "8d9a89b0c131e69fc97ecbb2279558e2930ae9e1",
      "date": "2021-10-03T20:30:38Z",
      "version": "2.6.1"
    }
  }
}

As for checking if it was a pi (or other single board computer) I don't see anything in particular. Sure, I use raspbian, but there are in fact other distros available, including Ubuntu and Arch I believe. I don't know what they identify as.

You could presumably do some file system checks, at least on Raspbian, such as looking for a particular structure under /boot since the booting process on Raspberry Pi is rather peculiar, but I also believe the boot loader uboot has been ported, and in that case I assume the layout is different.

Personally all my PIs are headless, so I treat them the same as servers when it comes to config files. This will of course vary based on your use cases. I currently handle this based on hostname in .chezmoi.yaml.tmpl, but it would be nice to avoid that. Maybe something based on the existence of /usr/bin/Xorg could work (for my personal purposes, this is obviously not a general solution)? Or looking for wayland presumably as well in the future.

Android

When it comes to Android, I thought you were just interested in general of a broad spectrum of devices. I don't in fact use chezmoi on my phone (yet at least). I do have Termux installed, but only use it to ssh into the phone for rsync pretty much. Your suggestion based on chezmoi.os should presumably work. It has been a long time since I used LinageOS and similar, but I assume they also identify as Android.

Other than that: what about Sailfish and other marginally used alternative mobile OSes (if they are even still alive)? Or is that a case of "don't bother until someone reports a bug about it"?

Other things

I also just remembered that I have access to some routers, switches and access points that I can ssh in on (I don't use chezmoi on them personally, so this is just for completeness). None of them (OpenWRT on MIPS, AsusWRT-Merlin on ARM, Unifi on MIPS and Unifi on ARM) have either hostnamectl or DMI.

After doing some quick research I would assume all non-x86 systems are missing /sys/devices/virtual/dmi. According to the man page of dmidecode, DMI is tied to SMBIOS, which I seem to remember is x86-specific. Maybe UEFI-based ARM64 systems might have it, but I don't have access to any such systems.

bradenhilton commented 3 years ago

@VorpalBlade Perhaps something like if stat /sys/class/power_supply/battery/status might work better? Given that the current chassis type block assumes that any system that isn't a desktop is a laptop, it might be sufficient.

I suspect that in most applications, a Raspberry Pi is desktop-like, meaning that it's probably plugged into a constant power source with zero or more displays/peripherals. Perhaps it would be more appropriate to think of them as headless desktops and use a block to check for headless systems instead?

In Termux on my LineageOS device, chezmoi data shows android under chezmoi.os. I haven't tried PRoot, so I can only assume that chezmoi data from within a distro running in PRoot would show the distro itself and not android.

twpayne commented 3 years ago

Apologies for the late contribution, a few points of possible interest:

  1. Generally, identifying features of the local machine in a robust cross-platform way is a hard problem. Many years ago I used Puppet's Facter, but I am not up-to-date with its current status.
  2. chezmoi has supported Termux in the past but there are no tests on Termux and the support may well have bit-rotted.

Given the above, especially point 1, I've deliberately avoided including much system property detection code in chezmoi. What you get in the .chezmoi template variable is pretty minimal and is not easy to use. For example, the fields of .chezmoi.osRelease (ostensibly populated from /etc/os-release) may or may not exist so you have to wrap them in get template function calls if you want to use them.

That all said, there is clearly a huge need for detecting the properties of the machine so you can configure your dotfiles appropriately and I am fully supportive of @bradenhilton's proposal here.

VorpalBlade commented 3 years ago

The less I have to check based on hostname, and the more I can auto detect the better. For what I do I don't really make a difference between a laptop and desktop, but servers/headless vs GUI is the important distinction for me that could plausibly be auto detected. Detecting that is however not easy. Chassis type might be an option to detect proper servers. Looking for Xorg/Wayland is another idea. This will fail in some cases however:

bradenhilton commented 3 years ago

@VorpalBlade Could this be useful?

Perhaps you could do something like

if chezmoi.os = linux
    if chezmoi.osRelease.id = raspbian
        if output of tvservice -s | awk '{print $2}' = 0x40001
            headless
    else if output of xrandr --query | grep -c ' connected' = 0
        headless
    else if some kind of wayland check (possibly using wlr-randr?)
        headless
    else
        not headless

I understand you don't want to use hostname checks, but it may still be worth using them as a fallback. That way, if you're having an issue with a particular system, you can quickly add it to the hostname check to get up and running, then see if you can better implement an appropriate check for it later.

VorpalBlade commented 3 years ago

Something along those lines would work except that if output tries to run a non-existing command it will fail. So there has to be checks for that too apparently:

$ chezmoi execute-template '{{ output "xrandr" }}'
chezmoi: template: arg1:1:3: executing "arg1" at <output "xrandr">: error calling output: exec: "xrandr": executable file not found in $PATH

It also doesn't look like chezmoi does short circuit boolean evaluation which makes this annoying (plus there are distros that doesn't use the traditional paths to binaries, such as NixOS):

$ chezmoi execute-template '{{ and (stat "/usr/bin/xrandr") (output "xrandr") }}'
chezmoi: template: arg1:1:33: executing "arg1" at <output "xrandr">: error calling output: exec: "xrandr": executable file not found in $PATH

This could of course be handled by doing nested if statements or by putting the logic in the shell:

$ chezmoi execute-template '{{ output "bash" "-c" "type xrandr &>/dev/null && xrandr | grep -c \" connected\" || echo 0" }}'
0

This feels quite clumsy to me, so: @twpayne It would perhaps be useful to have a variant of the output template function that allowed for explicitly handling the return code and/or lack of the executable to deal with situations like this. Perhaps defining a nested template allows for encapsulating the complicated nature of this (I haven't really played around with that feature yet) but I don't see a way for them to actually define functions and return values.

Some ideas off the top of my head (take with a grain of salt, these are probably not entirely thought thru):

  1. A better way to handle complicated queries without the template code becoming super complex and heavily repeated. For example, a way to define a template helper function. That said, I understand that no one wants a turing complete template language.
  2. A output function that allows the user to deal with failure (i.e. what if the command wasn't found or had a non-zero exit code).
  3. Since dealing with these sort of complicated properties directly in chezmoi does indeed seem painful even across different Linux systems (plus there is MacOS, *BSD and even Windows), a snippet repository with user contributions might be the way to go, rather than trying to integrate this functionality into chezmoi itself. This would be especially powerful in combination with the user defined functions from point 1.
twpayne commented 3 years ago

There's a lookPath template function that will likely help.

VorpalBlade commented 3 years ago

I had missed lookPath. It certainly helps, but the lack of user defined functions as well as lack of short circuit boolean evaluation makes the example above more complex than I believe it should need to be, though it wasn't as bad as I expected!

I came up with this as the most compact thing I could think of to just test xrandr (more would be needed for wayland for example):

{{- $has_gui := false -}}
{{- if lookPath "xrandr" -}}
    {{- $has_gui = contains " connected" (output "xrandr") -}}
{{- end -}}
data:
    custom:
        has_gui: {{ $has_gui }}

I had thought it would require more than that! It would still be neat to be able to define helper functions to deal with repeated boilerplate in case you have many similar checks. If you only want to output this then defining a nested template works:

{{- define "program_check" -}}
    {{- $result := false -}}
    {{- if lookPath .program -}}
        {{- $result = contains .text (output .program) -}}
    {{- end -}}
    {{ .var }}: {{ $result }}
{{- end -}}

data:
    custom:
        {{template "program_check" dict "var" "has_x_gui" "program" "xrandr" "text" " connected" }}
        # Hopefully a wayland check would be done the same way here.
        # I don't have any wayland system to test with though.

But I can't get it to work with "global" scope variables. It seems the nested template has it's own namespace of variables and cannot access globally defined variables. That would make a combined has_gui = X or Wayland hard to do using generic helpers like this. It also makes the following fragment from my actual config impossible using helper templates like this:

has_gui: {{ not (or $is_root $is_headless) }}
felipecrs commented 2 years ago

I'm just thinking aloud:

We could have a repository called chezmoi/common-templates which we could store as much templates as we want, and users could use them with git submodules:

# .gitmodules
[submodule "common-templates"]
    path = ".chezmoitemplates/common-templates"
    url = https://github.com/chezmoi/common-templates
twpayne commented 2 years ago

On further reflection on this, I think that the right place for shared templates is documenting them in chezmoi's user guide so that users can copy them into their own dotfiles repo. Here's my reasoning:

Specific examples of this:

So, I'll close this issue with the resolution: "if you have useful templates to share, please submit a PR that adds them to chezmoi's user guide".

Note also: the eager vs. lazy evaluation of arguments to the and and or template functions comes from Go's text/template language. This has changed a little with the release of Go 1.18 (search for "template" to find the exact details). chezmoi will always use text/template.

VorpalBlade commented 2 years ago

@twpayne Where in the user guide would you prefer to add something like https://github.com/VorpalBlade/chezmoi_modify_manager? Would you prefer a new page for "third party addons" (thus making it clear you provide no support or guarantees about the code in question) or would it be better to integrate it into for example the modify_ section?

Perhaps both: third party addons page but a mention in the modify_ section that a script to handle KDE ini files can be found "on the third party addons page"? This would increase discoverability but also make it clear that users are on their own using it.

twpayne commented 2 years ago

Yes, both please. Specifically:

Add it to the user guide, maybe as part of this section.

Add it to the list of related software.