search

twpayne / chezmoi

Manage your dotfiles across multiple diverse machines, securely.
https://www.chezmoi.io/
MIT License
13.23k stars 493 forks source link

Recommendations for dealing with dynamically named directories #1226

Closed bradenhilton closed 2 years ago

bradenhilton commented 3 years ago

What exactly are you trying to do?

I intend to switch to chezmoi from Stow (partly because my install script is spiralling out of control as you describe in the comparison, and partly so I can include Windows/macOS/Termux) but before doing so I'd like to wrap my head around dealing with directories with dynamic names.

The most notable example of this is Firefox, which assigns random prefixes to profile directories by default e.g. (edited for brevity)

> tree -d ~/.mozilla/firefox/profiles
~/.mozilla/firefox/profiles
├── 1io09dal.default-release
├── 5adf213y.default-nightly
└── n32lkdfa.default

where 1io09dal, 5adf213y and n32lkdfa can be any random string of letters and digits as far as I'm aware.

I'm not entirely sure if there is an elegant way to handle this with chezmoi itself as I'm aware 1:1 mappings are preferred.

At the moment I'm thinking the best method to deal with this is something like the following:

  1. Map the files and templates to an intermediate directory, such as ~/.config/firefox/profiles/default-release etc.
  2. Run chezmoi apply
  3. Run a script after chezmoi apply to link or copy the files to the correct directory

I'm interested to see if you have any thoughts or recommendations regarding this.

What have you tried so far?

N/A

Where else have you checked for solutions?

Output of any commands you've tried with --verbose

N/A

Output of chezmoi doctor

N/A

Additional context

N/A

twpayne commented 3 years ago

This is a really interesting problem.

Presumably if Firefox is generating random directory names, it is because Firefox does not expect the contents of those directories to be portable between machines. So, my first question is: "is there another way to configure Firefox to get the behavior you want?"

That said, my initial thought was "well, chezmoi allows file contents and symlink targets to be templates, why not make directory names templates too?" This is certainly technically possible within chezmoi's current architecture, but will require careful thought.

How do you handle dynamically named directories in your current setup?

bradenhilton commented 3 years ago

I appreciate the timely response!

Presumably if Firefox is generating random directory names, it is because Firefox does not expect the contents of those directories to be portable between machines. So, my first question is: "is there another way to configure Firefox to get the behavior you want?"

More information on Firefox profiles, including the data contained in them:

Many of the files found within the profile directory aren't necessary to persist between machines, especially if you use a Firefox account to sync things like bookmarks/history etc. They can also pose security risks (passwords database etc.). Many of these files are either impossible to edit, or editing them outside of Firefox can cause problems. I would describe these as state or cache files and not necessarily configuration files.

Within a profile directory, I think chrome/ (Firefox UI customisations), as well as many of the *.json/*.js files are what most people choose to configure.

It's possible to create a profile with a custom name, and, going through the profile creation wizard on my laptop, it appears this only changes the suffix after the period by default e.g. g5bojqtf.Bob. It's possible to choose a directory with any name, which would certainly solve the issue regarding Firefox specifically. I don't think this solution is great though as it relies on all applications that exhibit this problem to also provide a workaround.

That said, my initial thought was "well, chezmoi allows file contents and symlink targets to be templates, why not make directory names templates too?" This is certainly technically possible within chezmoi's current architecture, but will require careful thought.

Given that the profile name is used as the suffix by default, perhaps a simple solution would be to simply ignore the prefix and map to any directory that contains the exact same suffix. default-release could map to 5oajd52l.default-release or 215lbnl6.default-release etc.

Perhaps a set of matchers could be defined, to be used as directives in the file/directory name to provide this functionality, something like dot_mozilla/firefox/profiles/partial_match_default-release, though this could cause some nasty side effects if overly vague, so perhaps not. Specificity could be increased with something like dot_mozilla/firefox/profiles/partial_match_suffix_dot_default-release. I think something along these lines is quite elegant but it would require extreme care.

Sometimes I want to use the same configuration on all of the versions I have installed, so it could be quite cumbersome to have several copies of the same file just to map them correctly.

Expanding on this, matchers could include:

How do you handle dynamically named directories in your current setup?

As I understand it, a freshly installed Firefox has no awareness of any profiles, and none are created by default. Thus it is necessary to either launch Firefox to create a default profile, or launch the Firefox Profile Manager to create one. This has the potential to impact chezmoi (when used to bootstrap a new machine) as simply creating a profile directory is not enough, Firefox must also be made aware of this profile.

Information about local Firefox profiles can be found inside the profiles.ini file inside the ~/.mozilla/firefox directory. It's possible to parse this file to get the name and path of each profile.

When I only had the Release version installed, I just used grep and sed on profiles.ini in my Bash script to find the path of the default profile and set that to the Stow target, but this isn't scalable, so I'm looking to move to a more robust manager.

bradenhilton commented 3 years ago

Edited the previous comment to add missing words, clarify Firefox workaround and why it's not ideal, and expanded on matcher idea.

twpayne commented 3 years ago

Thanks for all this detailed information. I'm not a Firefox user and so haven't tested any of the following ideas, but some possible pistes to explore are:

  1. Manage more of Firefox's config with chezmoi than just the individual profile. Is it possible to manage ~/.mozilla/firefox/profiles.ini with chezmoi as well, so you can make Firefox aware of a profile in ~/.mozilla/firefox/profiles without having to invoke Firefox Profile Manager?

  2. Replace the profile directory with a symlink to a directory partially managed by chezmoi.

  3. Use a run_before_ script to copy the profile into a known location before chezmoi updates the dotfiles, and then a run_after_ script to copy changes back.

  4. Use a tool other than chezmoi to manage Firefox profiles across multiple machines. Presumably Mozilla have some sort of existing infrastructure for this.

As you identify, the partial_match_ proposal would be very tricky to implement and add a lot of complexity and corner cases to chezmoi. Furthermore, partial matches do not completely define what the directory should be called if it does not already exist, and so is incompatible with setting up a system from scratch.

bradenhilton commented 3 years ago
  1. Manage more of Firefox's config with chezmoi than just the individual profile. Is it possible to manage ~/.mozilla/firefox/profiles.ini with chezmoi as well, so you can make Firefox aware of a profile in ~/.mozilla/firefox/profiles without having to invoke Firefox Profile Manager?

Based on my current understanding I don't think this is possible, but it looks like I have some testing to do!

  1. Replace the profile directory with a symlink to a directory partially managed by chezmoi.

  2. Use a run_before_ script to copy the profile into a known location before chezmoi updates the dotfiles, and then a run_after_ script to copy changes back.

I think "3." would work better out of these 2 suggestions. Given that the files are copied, is it possible to ensure that they don't become orphans if chezmoi is removed? It's not a big deal if not, it would be very convenient to automate is all.

  1. Use a tool other than chezmoi to manage Firefox profiles across multiple machines. Presumably Mozilla have some sort of existing infrastructure for this.

As far as I understand it, they want users to just copy the entire profile and then use the Profile Manager to restore it, or they want users to manually set up a new profile and copy over all necessary files.

As you identify, the partial_match_ proposal would be very tricky to implement and add a lot of complexity and corner cases to chezmoi. Furthermore, partial matches do not completely define what the directory should be called if it does not already exist, and so is incompatible with setting up a system from scratch.

Agreed! Hypothetically, if these matchers were to be implemented, what do you think the most appropriate course of action would be when facing this issue? Simply informing the user that the files in question cannot be applied, and the reason why, before moving on to other files?

Regarding Firefox specifically, I think this could be solved with run_before_ by checking if any profiles exist, and creating them if they don't. I agree this is starting to get a bit messy though.

Do you think it would be worth adding one or two of these suggestions to the How-To, specifically in the context of dealing with dynamic/random names?

As a side note, am I correct in assuming that run_before_ and run_after_ scripts can be written in Python? If so, how does chezmoi execute Python scripts on each platform? Just thinking about how I might make the scripts (either from "3". above or my example immediately above) cross-platform.

I'm happy to close for now if you don't think there's anything to explore further, I will return with any findings when I get around to testing various setups.

Thanks again!

twpayne commented 3 years ago

I think "3." would work better out of these 2 suggestions. Given that the files are copied, is it possible to ensure that they don't become orphans if chezmoi is removed? It's not a big deal if not, it would be very convenient to automate is all.

I would perhaps make the "copy back" script do a "move" instead of a copy, so that there are no orphans left behind.

One thing to watch out for is that chezmoi currently only actually runs scripts if you're running chezmoi apply. If you run chezmoi diff or chezmoi verify or chezmoi apply --dry-run then chezmoi will not run the scripts, which may, for example, cause chezmoi diff to not print the diff you expect. I'm considering adding run_always_{before_,,after_} scripts that are always run, even for chezmoi diff, to support your use case, but maybe they are not needed.

Hypothetically, if these matchers were to be implemented, what do you think the most appropriate course of action would be when facing this issue? Simply informing the user that the files in question cannot be applied, and the reason why, before moving on to other files?

To be honest, I don't think matchers are a good fit with chezmoi as they violate chezmoi's core assumption that everything except scripts is declarative and unambiguous.

Do you think it would be worth adding one or two of these suggestions to the How-To, specifically in the context of dealing with dynamic/random names?

I would like to see some real life examples of use before adding these suggestions to the how-to, as often there are subtle details that are only discovered during implementation.

As a side note, am I correct in assuming that run_before_ and run_after_ scripts can be written in Python?

Yes. The scripts are just execd by chezmoi (after interpreting them as templates if they have the template attribute).

I'm happy to close for now if you don't think there's anything to explore further, I will return with any findings when I get around to testing various setups.

Let's keep this issue open for now until it's resolved, either with changes to chezmoi or its documentation.

bradenhilton commented 3 years ago

Thought I'd provide a quick update.

  1. Manage more of Firefox's config with chezmoi than just the individual profile. Is it possible to manage ~/.mozilla/firefox/profiles.ini with chezmoi as well, so you can make Firefox aware of a profile in ~/.mozilla/firefox/profiles without having to invoke Firefox Profile Manager?

Based on my current understanding I don't think this is possible, but it looks like I have some testing to do!

It turns out this is very possible. The (seemingly random) IDs used to differentiate between installs in profiles.ini are actually uppercase hexadecimal representations of the install paths after they are hashed by Google's CityHash (version 1). (sources: GetInstallHash, city.cpp)

This means it's possible to set a custom profile (both name and path) as the default profile for a particular install if you know this ID, like so:

%AppData%\Mozilla\Firefox\profiles.ini or ~/.mozilla/firefox/profiles.ini:

[Profile0]
Name=default-release ; can be anything
IsRelative=0 ; indicates the path below is an absolute path instead of being relative to %AppData%\Mozilla\Firefox or ~/.mozilla/firefox
Path=<custom_release_profile_path> ; e.g. %USERPROFILE%\.config\firefox\profiles\default-release or ~/.config/firefox/profiles/default-release

[Profile1]
Name=default-nightly
IsRelative=0
Path=<custom_nightly_profile_path>

[General]
StartWithLastProfile=1
Version=2

[Install<hash_for_release_install>]
Default=<custom_release_profile_path> ; sets the default profile for this install
IsLocked=1

[Install<hash_for_nightly_install>]
Default=<custom_nightly_profile_path>
IsLocked=1

Knowing this, I tried the C++ version of CityHash (version 1) but I wasn't a fan of how it gave different results depending on the platform I ran it on (perhaps due to differences in size_t between platforms etc.), so I created a Go package mozillainstallhash.

Now that I can get the hashes I need to set the default profile for a given install, and I can create a custom profile with chezmoi, it's possible to do this:

.chezmoitemplates/Firefox/profiles.ini

{{- /* Profile paths for each installation version */ -}}
{{- $releasePath := joinPath .chezmoi.homeDir "/.config/firefox/profiles/default-release" -}}
{{- $nightlyPath := joinPath .chezmoi.homeDir "/.config/firefox/profiles/default-nightly" -}}

{{- /* IDs for each installation version */ -}}
{{- $releaseId := "0" -}}
{{- $nightlyId := "0" -}}
{{- if eq .chezmoi.os "windows" -}}
{{-   $releaseId = "308046B0AF4A39CB" -}}{{- /* "C:\Program Files\Mozilla Firefox" */ -}}
{{-   $nightlyId = "6F193CCC56814779" -}}{{- /* "C:\Program Files\Firefox Nightly" */ -}}
{{- else if eq .chezmoi.os "darwin" -}}
{{-   $releaseId = "2656FF1E876E9973" -}}{{- /* "/Applications/Firefox.app/Contents/MacOS" */ -}}
{{-   $nightlyId = "31210A081F86E80E" -}}{{- /* "/Applications/Firefox Nightly.app/Contents/MacOS" */ -}}
{{- else if eq .chezmoi.os "linux" -}}
{{-   $releaseId = "4F96D1932A9F858E" -}}{{- /* "/usr/lib/firefox" */ -}}
{{-   $nightlyId = "6BA5C87ECB35E12F" -}}{{- /* "/opt/firefox-nightly" */ -}}
{{- end -}}

[Profile0]
Name=default-release
IsRelative=0
Path={{- $releasePath }}

[Profile1]
Name=default-nightly
IsRelative=0
Path={{- $nightlyPath }}

[General]
StartWithLastProfile=1
Version=2

[Install{{- $releaseId -}}]
Default={{- $releasePath }}
Locked=1

[Install{{- $nightlyId -}}]
Default={{- $nightlyPath }}
Locked=1

Then on Windows, I can create AppData/Roaming/Mozilla/Firefox/profiles.ini.tmpl in the the chezmoi source directory:

{{- template "Firefox/profiles.ini" . -}}

And the generated file is %AppData%\Mozilla\Firefox\profiles.ini:

[Profile0]
Name=default-release
IsRelative=0
Path=C:\Users\Braden\.config\firefox\profiles\default-release

[Profile1]
Name=default-nightly
IsRelative=0
Path=C:\Users\Braden\.config\firefox\profiles\default-nightly

[General]
StartWithLastProfile=1
Version=2

[Install308046B0AF4A39CB]
Default=C:\Users\Braden\.config\firefox\profiles\default-release
Locked=1

[Install6F193CCC56814779]
Default=C:\Users\Braden\.config\firefox\profiles\default-nightly
Locked=1

Then all I need to do is have chezmoi create the profiles, which can also be comprised of various templates etc.

I think my next step would be to see if I can automate the hash generation of all currently installed Firefox versions (if any) and somehow pipe them to the profiles.ini template file to remove the if statement. Do you have any suggestions for this? I was thinking I could perhaps run the Go package I made like so: (not sure if the command is correct off the top of my head)

./get_mozilla_install_hash $(dirname $(which firefox))

then assign the output to a variable, then simply reference this variable in the profiles.ini template. Is that possible?

I am happy with this solution; however, this doesn't directly deal with the issue of randomly named directories. This is simply a workaround that is specific to Mozilla software, so for directories with truly random names one would likely have to deal with them as you describe in points 2. and 3. above.

twpayne commented 3 years ago

@bradenhilton you can make the variables releaseId and nightlyId normal chezmoi template variables (i.e. define them in the data section of chezmoi's config file).

bradenhilton commented 3 years ago

@twpayne I'm actually doing that for a few other things at the moment (whether I'm on a desktop or laptop, how many screens there are etc.) but I can't think of a nice way to 1) find the parent directory of the executable natively on all platforms and 2) pipe the location to my Go package in an output.

Regarding the first issue, there seems to be a hangup on Windows (as there often is). On a Unix-like platform you can probably use which and then get the directory of that path. On Windows, the default installation of Firefox does not add C:\Program Files\Mozilla Firefox to PATH, so using where or (Get-Command <command>).Path don't seem to work without providing the full path, so it's a bit of a Catch-22.

Regarding the second issue, this is probably because I'm trying to run the Go package like a script, and Go is not really a scripting language. I only landed on Go after exhausting my (convenient) options via Python and JavaScript, so perhaps I'll have to recreate the package there and try again.

Like I said previously, I'm pretty happy with the way I have it set up at the moment. After all, how many times does one not only reinstall Firefox, but every time they do, change the install location? I'm just trying to think of a way that would work for most people in the event that something like this is added to the documentation, but perhaps I'm overthinking it.

Regardless, I think I'll still work on it anyway to see if I can scratch that itch.

Also, for future reference, would you prefer if I used @ mentions in comments going forward? I elected not to previously as I didn't think it was absolutely imperative that you see the update ASAP, but either way is fine with me.

twpayne commented 3 years ago

1276 adds this functionality to as a template function, which should make your templates easier to manage. What are your thoughts on it?

Also, for future reference, would you prefer if I used @ mentions in comments going forward? I elected not to previously as I didn't think it was absolutely imperative that you see the update ASAP, but either way is fine with me.

As someone who maintains an open source project in my spare time with no compensation for that time and effort, I cannot imagine any situation where it would be absolutely imperative that I saw your update ASAP. Or am I missing something?

bradenhilton commented 3 years ago

1276 adds this functionality to as a template function, which should make your templates easier to manage. What are your thoughts on it?

I will check it out later today, but my immediate thought is I will leave that entirely up to you. You'd be adding two dependencies (one indirect) from me for something that is quite specialised, and I have no idea what I'm doing regarding Go modules. I don't currently have any plans to prioritise maintaining them, at least until I'm a lot more comfortable with the ecosystem, so they may break one day.

That said, if you can think of anything I'm doing that is obviously wrong, or would make it easier for you to manage them as dependencies, feel free to let me know or open a pull request.

I'm trying to familiarise myself with everything so I can push out stable releases (v1.0.0), in hopes that I won't have to mess with them too much, save for Mozilla completely changing the way it hashes install paths.

As someone who maintains an open source project in my spare time with no compensation for that time and effort, I cannot imagine any situation where it would be absolutely imperative that I saw your update ASAP. Or am I missing something?

Speaking of priorities, I worded that incorrectly. It was not my intention to suggest that. I would guess that some kind of hypothetical critical bug or vulnerability in chezmoi would be a higher priority for you, and my update was comparatively not even remotely a priority. I was trying to ask if you prefer if @ mentions are reserved for situations which might be considered higher priority is all, as they can be quite spammy at times. Apologies if I offended you.

twpayne commented 3 years ago

For the dependencies, the Go module system is pretty good at this. The modules are proxied by default so even if you end up deleting the repos chezmoi should continue to work. Also, the two modules are MIT licensed so I can bring them into the chezmoi repo if necessary. I had a look at the code in the two modules, and it's certainly good enough as-is. Firefox is sufficiently widely used that I don't mind introducing a template function to help cope with it. If they change the hashing function in the future then we can always add a new template function.

For the mentions, thanks for your words. In practice, in this case I see all notifications for everything in this repo, so a mention or not does not change anything in practices. For repos where I am not subscribed to all notifications, I appreciate notifications that either draw my attention to something I would not otherwise see, or, in a PR or issue in which I am already participating, to make it clear which parts of a comment refer to something I've said.

slurikadre commented 3 years ago

@bradenhilton I'd been doing something similar to this using an ugly bash script 😳.

The script reads a json object with jq iterates through a loop and writes the applicable user.js into the profile directory. I didn't have any smarts to figure out the profile name so I had to store that in the json object.... I'm not really happy with it though.

I wouldn't mind seeing what you've come up with.

bradenhilton commented 3 years ago

@slurikadre Sure! I'm not creating multiple files from/within the same template so I'm not sure I can help you there, but I'm happy to share my current Firefox setup.

Regarding Firefox specifically, some of my files are similar to what I shared in https://github.com/twpayne/chezmoi/issues/1226#issuecomment-867228095, however, they are incomplete so I will share them all here for completeness. I've placed the necessary files in a sample repo. I've added a note that highlights an issue pertaining to certain install methods in the README.

I haven't tested them on anything other than Windows at the moment so I can't guarantee they work.

I must say your use case is a lot more involved than mine, regardless I hope it can help in some way.

slurikadre commented 3 years ago

@slurikadre Sure! I'm not creating multiple files from/within the same template so I'm not sure I can help you there, but I'm happy to share my current Firefox setup.

Sure, that was kind of unrelated. I think I've come up with a solution to that though.

  1. Write a very simple helper script sh which simply duplicates the template.
  2. The helper script itself could be a template, *.tmpl that way it would know what to make the filenames of each template when it is run, this would come from chezmoidata. It would be run_before
  3. Each template for each account fills with data.

Regarding Firefox specifically, some of my files are similar to what I shared in #1226 (comment), however, they are incomplete so I will share them all here for completeness. I've placed the necessary files in a sample repo. I've added a note that highlights an issue pertaining to certain install methods in the README.

Nice.

I must say your use case is a lot more involved than mine, regardless I hope it can help in some way.

I actually don't think it is. Mine copies a userchrome.css and user.js. I decided I don't really need the git part as it is all in the same repo, with the same branch fx-desktop.

The .desktop file is also a template, for that I might do a similar thing to what I did with my mutt configs. Ie a script that is a template.

ayushnix commented 3 years ago

Hey, so I started using chezmoi a few days ago and I've stumbled upon the issue of managing my Firefox userChrome.css and user.js as well.

Is there a straightforward solution to managing these files if I don't care about the mozilla profile hash? All I want though is to have the changes made in ~/.mozilla/firefox/<hash>.release/user.js reflect automatically in chezmoi's source path and chezmoi status not reporting the user.js file as modified. I don't care if I later do chezmoi apply and the files are restored in some other path besides ~/.mozilla, such as $XDG_CONFIG_HOME/mozilla/firefox/release/user.js.

I tried using hard links and symlinks but chezmoi apply ends up replacing those. Will this method help? Or should I just use a hard link/symlink and add user.js to .chezmoiignore? Or is there something else I should do?

EDIT: I've started using the following method - create a hard link in chezmoi's source path and add the path in .chezmoiignore. chezmoi diff and chezmoi apply won't do anything but I can still manage those files once in a while when I do chezmoi cd and find out that they've been modified with git status. Perhaps not the best solution but works for now. Please let me know if there's anything better.

slurikadre commented 2 years ago

Regarding Firefox specifically, some of my files are similar to what I shared in #1226 (comment), however, they are incomplete so I will share them all here for completeness. I've placed the necessary files in a sample repo. I've added a note that highlights an issue pertaining to certain install methods in the README.

I've finally had time to take another look at this. My ugly script was getting to me, as it didn't work nicely and well. was ugly.

I don't these days really have more than one branch (same user.js userChrome.css etc.).

It looks to me that you don't store your profile directory in .mozilla, but rather in .config.

Curious to know if there's any reasoning for that. In the past my "~/.config/firefox/user.js" directory was a git repo.

It might make sense for me to adopt the location you have, and make that a submodule in .chezmoitemplates. I guess it would get updated then when I update my submodules.

bradenhilton commented 2 years ago

@slurikadre They can be set to any path you would like as they are absolute paths. You can make them relative by setting IsRelative to 1 under each profile and changing the profile paths appropriately. I went with .config simply because it works on all of the platforms I use. I also don't use it as a submodule so I'm not sure how it would work.

slurikadre commented 2 years ago

@slurikadre They can be set to any path you would like as they are absolute paths. You can make them relative by setting IsRelative to 1 under each profile and changing the profile paths appropriately.

Oh awesome.

I went with .config simply because it works on all of the platforms I use.

Yeah I am fond of the XDG specification. I kind of wish the whole .mozilla directory used XDG spec.

I also don't use it as a submodule so I'm not sure how it would work.

I use https://github.com/arkenfox/user.js/ so I was thinking of making that a submodule in .chezmoitemplates. I have a few exceptions in user-overrides.js.

There is an updater.sh script there which updates and merges the two files.

So I was thinking instead of my ugly script I'd just have 

#!/bin/sh

workTree="$XDG_DATA_HOME/chezmoi/.chezmoitemplates/firefox/user.js"
gitDir="$workTree"/.git

git --git-dir="$gitDir" --work-tree="$workTree" checkout "fx-desktop"
"$workTree"/updater.sh -b -d -s

I'd make the script run_before so that it runs, updates, the templates are made and moved into the right place.

slurikadre commented 2 years ago

So I've thought of an even easier way.

Have a script: dot_local/bin/scripts/chezmoi-scripts/run_before_init_firefox.sh

#!/bin/sh

srcDir="$HOME"/src
chezmoiTmpl="$XDG_DATA_HOME"/chezmoi/.chezmoitemplates/firefox/profiles

# Update latest git pull
#git --git-dir="$srcDir/arkenfox/user.js/.git" \
#    --work-tree="$srcDir/arkenfox/user.js" pull

# Run updater, merge user-overrides.js
"$srcDir"/arkenfox/user.js/updater.sh -b -d -s \
    -o "$chezmoiTmpl"/user-overrides.js \
    -p "$chezmoiTmpl"

This script will update the user.js to the latest version, and it uses the user-overrides.js i have specified. The updater.sh script is maintained by the Arkenfox project, so I don't have to maintain that. It merges the user-overrides.js in with the recommended hardening and privacy recommendations of the project.

Then I can basically just do what you've done with your chezmoi-firefox-setup and let chezmoi manage my actual profiles.

I make sure to add to .gitignore.

Technically the git --git-dir="$srcDir/arkenfox/user.js/.git" --work-tree="$srcDir/arkenfox/user.js" pull isn't required as the updater.sh script can update each time, if you don't specify the -d option, but I prefer not make a network connection all the time.

Therefore I comment the git line out and run another script that updates all my git repositories that I keep an eye on in my srcDir at the same time as I like to audit the changes.

bradenhilton commented 2 years ago

@slurikadre Can't you just use .chezmoiexternal to download the updater.sh then run it with a run_after?

slurikadre commented 2 years ago

@slurikadre Can't you just use .chezmoiexternal to download the updater.sh then run it with a run_after?

Oh I hadn't automated downloading updater.sh as I already have that in ~/src/arkenfox/user.js

What is the benefit of using .chezmoiexternal? I'm curious.

bradenhilton commented 2 years ago

@slurikadre From https://github.com/arkenfox/user.js/wiki/3.3-Updater-Scripts#small_orange_diamond-maclinux:

Download the updater.sh script and save it in your profile folder. You can run it without any command line arguments and it will backup your current user.js, download the latest arkenfox user.js and if it finds a user-overrides.js in the same folder it will append that to the user.js.

Based on the above, all you need to do is run the updater.sh without any arguments from within the profile directory with a user-overrides.js file optionally present.

What is the benefit of using .chezmoiexternal? I'm curious.

.chezmoiexternal basically achieves the same thing as the git part of your script, but is built in to chezmoi and is quite configurable. It's your choice though if you prefer your script.

It looks like the updater.sh script can also update itself, so you might not need to clone the entire repo and can instead just download that in order to get up and running.

slurikadre commented 2 years ago

Based on the above, all you need to do is run the updater.sh without any arguments from within the profile directory with a user-overrides.js file optionally present.

Good point.

.chezmoiexternal basically achieves the same thing as the git part of your script, but is built in to chezmoi and is quite configurable. It's your choice though if you prefer your script.

Oh, no, this looks like a much better solution. Thanks!

Honestly chezmoi continues to surprise me with magic the more I use it. My old shitty yadm sh scripts I had were so crap in comparison :grin:

It looks like the updater.sh script can also update itself, so you might not need to clone the entire repo and can instead just download that in order to get up and running.

Indeed, and I nearly always have curl available.

slurikadre commented 2 years ago

@slurikadre Can't you just use .chezmoiexternal to download the updater.sh then run it with a run_after?

So I think I've got it kind of working, except it's not doing the arguments..

[".local/share/chezmoi/run_after_updater.sh"]
    type = "file"
    executable = true
    url = "https://raw.githubusercontent.com/arkenfox/user.js/master/updater.sh"
    refreshPeriod = "168h"
    filter.args = ["-s", "-p", ".chezmoitemplates/firefox/profiles"]

The script itself runs, but in the wrong location:

Firefox profile:  /tmp

Maybe this is one @twpayne might know.

twpayne commented 2 years ago

Externals are for downloading files and directories. The filter is for modifying the downloaded data before writing it to its final location. The filter will only run if filter.command is set.

Externals do not run scripts.

slurikadre commented 2 years ago

Right, so i might have misunderstood how that works then. I can use externals to download it, but I think I'll still need a script to run it. as "run_after is going to run it in /tmp and not let me specify arguments.

twpayne commented 2 years ago

as "run_after is going to run it in /tmp and not let me specify arguments.

This is not correct. Please read the section on scripts in the reference manual.

slurikadre commented 2 years ago

I don't mean the

as "run_after is going to run it in /tmp and not let me specify arguments.

This is not correct. Please read the section on scripts in the reference manual.

The problem I had when trying to setting the external to .local/share/chezmoi/.chezmoitemplates/firefox/profiles/run_after_updater.sh was that the template in there would say user.js was missing as it hadn't yet been downloaded by the updater.sh script.

When I moved the location to .local/share/chezmoi/run_after_updater.sh it was saying that the running location was /tmp, which isn't what I want. By default it puts the user.js in the same location as updater.sh unless you specify a location with -p and tell it where the user-overrides.js is with -o.

slurikadre commented 2 years ago

Okay, so i figured out the problem. If you write a file to .chezmoitemplates it won't be run. If you write it somewhere else, it seems that chezmoi copies the script to /tmp to run it. That then makes the updater.sh script download it's user.js to /tmp.

So I think the only way to do this might be to have a script that runs the updater, and not have chezmoi run the updater script. That way arguments can be given or it won't run it in /tmp.

twpayne commented 2 years ago

chezmoi does not run scripts in /tmp. It does write script files to /tmp as scripts can be templates so they need to be generated from the template. The script's working directory is its equivalent location in your home directory.

If your script thinks it's being run in /tmp, then it is almost certainly a bug in your script. Replace the scripts contents with:

#!/bin/sh
pwd

to see this.

slurikadre commented 2 years ago

I don't believe it's a bug in the script, as that most certainly is very widely used by many people.

I think the problem might be with the $(pwd) that it uses https://github.com/arkenfox/user.js/blob/master/updater.sh#L248 I wonder if this is a bug? It seems to work correctly when you run ./updater.sh in the command line, that's likely all its been tested with hmm.

twpayne commented 2 years ago

I think the bug in arkenfox/user.js is this line:

https://github.com/arkenfox/user.js/blob/master/updater.sh#L158

Here, the user.js assumes that the profile directory is the same directory as where the script is, which is incorrect.

slurikadre commented 2 years ago

Hmm, so your right.

In any case though it seems it is not possible to run a script in .chezmoitemplates which means I'd have to have something to copy the user.js from somewhere else to that location.

If I am going to do that I might as well run the updater script and pass a parameter -p to defining where I want the user.js to end up. (Which does circumvent this bug).

Unless there is a better way.

twpayne commented 2 years ago

I haven't followed all the details of how Firefox stores it configuration and how user.js helps, but consider:

  1. Use an chezmoi external to download user.js to ~/bin.
  2. Use a runafter script to call ~/bin/user.js with the right arguments.
slurikadre commented 2 years ago

how Firefox stores it configuration and how user.js

These are actually merged into prefs.js when Firefox is opened, that's all, and I am not expecting chezmoi to have anything to do with that side of it.

What chezmoi actually does is plops the user.js file into the template user.

That means the user.js file just needs to be plopped into .chezmoitemplates/firefox/profiles and as it seems scripts can't run in .chezmoitemplates I think the most straightforward way is what you said above.

  1. Use an chezmoi external to download user.js to ~/bin.

My .chezmoiexternal.toml has:

[".local/share/chezmoi/dot_local/bin/scripts/firefox/updater.sh"]
    type = "file"
    executable = true
    url = "https://raw.githubusercontent.com/arkenfox/user.js/master/updater.sh"
    refreshPeriod = "168h"
  1. Use a runafter script to call ~/bin/user.js with the right arguments.
#!/bin/sh

srcDir="$HOME"/src
chezmoiTmpl="$XDG_DATA_HOME"/chezmoi/.chezmoitemplates/firefox/profiles

# Run updater, merge user-overrides.js
"$srcDir"/arkenfox/user.js/updater.sh -b -d -s \
    -o "$chezmoiTmpl"/user-overrides.js \
    -p "$chezmoiTmpl"

What I have noticed with this approach is if no user.js exists, it won't run and you'll get this error:

chezmoi: template: firefox/profiles/user:9:12: executing "firefox/profiles/user" at <{{template "firefox/profiles/user.js"}}>: template "firefox/profiles/user.js" not defined

I guess i could touch a $chezmoiTmpl/user.js if it doesn't exist. ie like

if ! [ -f "$chezmoiTmpl"/user.js ]; then
    touch "$chezmoiTmpl"/user.js
fi

But the issue with that is with run_after, the user.js is still going to be missing. and emit that error.

twpayne commented 2 years ago

Generally speaking, putting user.js anywhere in .chezmoitemplates is likely to break. Specifically, chezmoi treats every file in .chezmoitemplates as a template, and user.js is not a template. If the string {{ ever appears in user.js then it will not parse as a template.

To include files literally (without interpreting them as templates) you can use the include template function.

slurikadre commented 2 years ago

Okay so I decided to do away with the updater.sh script. I don't actually need it.

.chezmoiexternal.toml:

[".local/share/chezmoi/user.js"]
    type = "file"
    url = "https://raw.githubusercontent.com/arkenfox/user.js/master/user.js"
    refreshPeriod = "168h"

.chezmoitemplates/firefox/profiles/user:

{{ include "user.js" -}}

{{ include "user-overrides.js" -}}

.chezmoitemplates/firefox/profiles/userChrome

{{ include "userChrome.css" -}}

Then in:

dot_config/firefox/profiles/*/chrome/userChrome.css.tmpl

{{- template "firefox/profiles/userChrome" . -}}

dot_config/firefox/profiles/*/user.js.tmpl

{{- template "firefox/profiles/user" . -}}

The only problem I have is .chezmoiexternal is executed after the template. Meaning if it's not created I get this error:

chezmoi: template: firefox/profiles/user:1:3: executing "firefox/profiles/user" at <include "user.js">: error calling include: open /home/slurikadre/.local/share/chezmoi/user.js: no such file or directory

I do like the approach of doing away with updater.js altogether though. Less moving parts, less things to break.

It means I don't have to deal with bugs https://github.com/twpayne/chezmoi/issues/1226#issuecomment-955242894. I also don't have that userjs_backups (created when updater.sh was run), which it seems you can't disable that.

If only .chezmoiexternal had like a:

getFirst = true

option so that the user.js file existed before the template goes to look for it..

Then all I would have to do is add .user.js to .gitignore.

Every so often Chezmoi would get me a new user.js, or I could delete it to force downloading of a new one if a new release of Firefox is released. This is where my usecase differs from @bradenhilton.

The arkenfox/user.js project is a community run project which looks for the best security, privacy anti-fingerprinting options available for Firefox and enables them. What I like about it is they seem to be pretty serious on QA, meaning that the changes they make are carefully vetted before being committed. They use searchfox (formerly dxr.mozilla.org) to verify what the options do, before changing them.

This means things are never broken (in my experience) and it works nicely.

bradenhilton commented 2 years ago

@slurikadre I will say I recommended using .chezmoiexternal before I had the understanding of arkenfox/user.js that I have now. I assumed it was simply a static user.js file that you wanted to use as is.

Let's examine what we know (feel free to correct me if I'm wrong):

Based on this, we can surmise:

This sounds like a great use case for a run_before_ script. Something like the following (pseudocode):

<chezmoi source dir>/run_before_init_arkenfox.sh or whatever you would like.

if arkenfox updater.sh exists:
    exit

download arkenfox updater.sh
run arkenfox updater.sh with arguments/overrides to create a user.js file

You may need to place the updater.sh script in your Firefox profile folder, and have it generate and output the user.js file to your .chezmoitemplates/firefox/profiles, which can then be used in templates etc.

You'll be checking if updater.sh exists every time you run chezmoi apply, if you don't want to do that you can use a run_once_before_ script, but that won't redownload updater.sh if it gets deleted.

You can also create an empty user.js file in your .chezmoitemplates/firefox/profiles in the event that updater.sh fails to download or run. You'd probably have to commit/push the empty file to your source control and then ignore any future changes to it.

See how far you can get with something like that.

@twpayne Slightly off-topic - have you ever considered something like a .chezmoiscripts directory? That is, a directory containing scripts which are intended to be run only by chezmoi. The . would make chezmoi not create the directory in the target directory, but being a special chezmoi directory, the scripts would still be run.

I'm currently doing this with a chezmoi_scripts directory in my source directory but IIRC, an (empty) chezmoi_scripts directory is created in the target directory. Also, having potentially tens or hundreds of run_* scripts in the source directory root can be a bit ugly IMO, so it would be nice to hide those somewhere.

I can open a separate issue if needed.

slurikadre commented 2 years ago

@slurikadre I will say I recommended using .chezmoiexternal before I had the understanding of arkenfox/user.js that I have now. I assumed it was simply a static user.js file that you wanted to use as is.

Let's examine what we know (feel free to correct me if I'm wrong):

  • The updater.sh script is capable of:
    • Downloading the arkenfox user.js file, and merging it with user defined overrides.
    • Updating itself.

Correct, but ideally it would be nice if we could work around even running that script.

  • You are getting errors related to the user.js file not existing when chezmoi is generating and applying the target state.

Correct

Based on this, we can surmise:

  • All you need is the updater.sh script, as well as an optional user-overrides.js file.
  • You need a user.js file to exist before chezmoi generates and applies the target state.

But do I really even need the updater.sh script? It seems to me that chezmoiexternal can download the user.js from github and a template can merge the two together (user.js and user-overrides.js).

The only problem seems to be that you can't make a chezmoiexternal download occur before processing templates. I'm thinking this should actually be a feature. Ie. if a special boolean is specified on the chezmoiexternal, it is downloaded first before anything else happens.

This sounds like a great use case for a run_before_ script.

That does seem like one way of doing it. Which is why I was a bit confused when you both suggested run_after.

Something like the following (pseudocode):

Indeed I did start with this:

#!/bin/sh

"$XDG_DATA_HOME"/chezmoi/updater.sh -b -d -s \
    -o "$XDG_DATA_HOME"/chezmoi/user-overrides.js \
    -p "$XDG_DATA_HOME"/chezmoi

You may need to place the updater.sh script in your Firefox profile folder, and have it generate and output the user.js file to your .chezmoitemplates/firefox/profiles, which can then be used in templates etc.

Taking into acount @twpayne 's warning https://github.com/twpayne/chezmoi/issues/1226#issuecomment-955400519 I decided to just store it in "$XDG_DATA_HOME"/chezmoi

You'll be checking if updater.sh exists every time you run chezmoi apply, if you don't want to do that you can use a run_once_before_ script, but that won't redownload updater.sh if it gets deleted.

Right, ideally I'd do-away with the updater.sh script as it seems chezmoiexternals are more than capable of downloading a single file.

You can also create an empty user.js file in your .chezmoitemplates/firefox/profiles in the event that updater.sh fails to download or run. You'd probably have to commit/push the empty file to your source control and then ignore any future changes to it.

That's what I'm presently doing.

@twpayne Slightly off-topic - have you ever considered something like a .chezmoiscripts directory? That is, a directory containing scripts which are intended to be run only by chezmoi. The . would make chezmoi not create the directory in the target directory, but being a special chezmoi directory, the scripts would still be run.

I'm currently doing this with a chezmoi_scripts directory in my source directory but IIRC, an (empty) chezmoi_scripts directory is created in the target directory. Also, having potentially tens or hundreds of run_* scripts in the source directory root can be a bit ugly IMO, so it would be nice to hide those somewhere.

I really like the idea of that :+1: I'd been storing them in a chezmoi_scripts dir too.

bradenhilton commented 2 years ago

But do I really even need the updater.sh script? It seems to me that chezmoiexternal can download the user.js from github and a template can merge the two together (user.js and user-overrides.js).

I think that depends on how exactly the updater.sh script merges the overrides. If they are simply appended to the user.js file, you could do this manually. Anything more complicated than that and you'd probably need a script which means you'd essentially be rewriting updater.sh.

That does seem like one way of doing it. Which is why I was a bit confused when you both suggested run_after.

That was just because you wouldn't really have any other choice currently if you used .chezmoiexternal. I asked about the order in which things are executed earlier which you can see in https://github.com/twpayne/chezmoi/discussions/1568.

Probably not recommended, but you can also try using a run_after_ script which uses chezmoi execute-template to create files directly in the target directory, however this has its own suite of problems.

You may need to place the updater.sh script in your Firefox profile folder, and have it generate and output the user.js file to your .chezmoitemplates/firefox/profiles, which can then be used in templates etc.

Taking into acount @twpayne 's warning #1226 (comment) I decided to just store it in "$XDG_DATA_HOME"/chezmoi

I don't believe this warning applies here. I was specifically talking about placing the final user.js file which is outputted from the updater.sh script in the appropriate template directory. Unless the computed file has text/template syntax this should work just fine.

slurikadre commented 2 years ago

But do I really even need the updater.sh script? It seems to me that chezmoiexternal can download the user.js from github and a template can merge the two together (user.js and user-overrides.js).

I think that depends on how exactly the updater.sh script merges the overrides. If they are simply appended to the user.js file, you could do this manually. Anything more complicated than that and you'd probably need a script which means you'd essentially be rewriting updater.sh.

Yes, it only merges the files together.

twpayne commented 2 years ago

I think a lot of issues in this thread can be solved by not using chezmoiexternals to download user.js and using curl in a script instead, as @bradenhilton suggests.

For a bit of background, chezmoi tries to be "declarative where possible, imperative where necessary". Externals are an example of the former: they declare that a file or directory in your home directory should be equal the contents of a downloaded file or archive. In this declarative world, there is not concept of time. Things simply are or are not. chezmoi's escape hatch from the declarative world is scripts. The "download user.js and then use it" is imperative by nature: first one thing has to happen, then the other, and so should be solved with scripts.

@twpayne Slightly off-topic - have you ever considered something like a .chezmoiscripts directory?

I hadn't considered this, but it does sound like a reasonable idea. Please do open an issue for this.

twpayne commented 2 years ago

Hopefully most of this has now been resolved. Please re-open if needed.

bradenhilton commented 2 years ago

@twpayne I personally still think this should be covered in the docs at some point, but I'm not sure how it should be structured or where it should be.

I was thinking it could be broken up into generic solutions (copy/move to source directory, then copy/move back etc.) and application specific solutions such as my Firefox setup, but perhaps this is too complicated.