Is yadm the right tool for me? (advanced)

[ghost]

Okay, so I have wanted to make my dotfiles public for ages. The main reason I haven't is because I'm scared to accidentally leave anything in there that I shouldn't.

Over the last few months I've been extracting any secret data. Secret data I don't really want in version control at all, regardless of whether it's encrypted this includes sec.conf, X.509 certificates/keys, ~/.gnupg/private-keys-v1.d, ~/.ssh/id_rsa etc. I store these offline but at two different sites. Even encrypted I don't to share the ciphertext with anyone. Most people seem to just stop here.

However I want to go one step further. Have a public version of my dotfiles and a private version. The reason for this is because the private version might contain things such as domains that are linked to me, my IPv6 globally linked address, usernames that I posess etc. While these things won't grant authentication they may still be used to map me.

Therefore I am going to have a private version as well. This would have the unsantized copy. I don't particularly want to push this to github, gitlab etc.

I am thinking that symlinking to a dotfiles directly may still be the way to go. I think this bare repo method might not really work for me.

The other thing I am hoping is that friends can make their own private versions from my public version. If they had a suggestion they thought I might like, they could check out a copy of the public version, branch that, and send a pull request back.

I've included a picture to demonstrate what I am trying to do.

My question is should I bother with any "dotfile" tools. I noticed these tools in the Arch wiki?

I had a look at yadm and it looks most promising. Particularly because of the bootstrap feature. That being said, it might be better if I use something like ansible or Salt for bootstrapping and deploying to my stuff.

I included a picture here of how this should look.

Below is the graphviz code, in case you think you have an improvement.

digraph graphname {
    node [shape=rectangle, style="filled"];

    master [fillcolor="#ff9999", label="master - general files across all hosts (public)"];
    desktop [fillcolor="#ff9999", label="Desktop (public) \n sanitized configs \n personally identifiable \n information removed"];
    laptop [fillcolor="#99ff99", label="Laptop \n (private)"]
    workstation [fillcolor="#99ff99", label="Workstation \n (private)"]
    friendsworkstation [fillcolor="#ffffbb", label="Friend's Workstation \n (private)"]
    prFromFriend [fillcolor="#ff9999", label="Friend opens PR (public)"]

    { rank=same laptop workstation friendsworkstation prFromFriend }

    host2 [fillcolor="#ff9999", label="Router (public) \n documentation\: \n 192.0.2.0\/24, 2001:db8\:\:\/32"];
    host2private[fillcolor="#99ff99", label="Router (private) \n real IPs: 192.168.1.0\/24, \n real assigned 2001 ipv6 GLA range"]
    host3 [fillcolor="#99ff99", label="Router (private) \n real IPs, domains etc"]
    host3 [fillcolor="#ff9999", label="Server (public) \n example.com etc "];
    host3private [fillcolor="#99ff99", label="Server (private) \n realdomain.tld"];
    vm [fillcolor="#ff9999", label="Virtual Machine/Container (public)"];
    vm1 [fillcolor="#99ff99", label="Virtual Machine 1 \n (private)"];
    vm2 [fillcolor="#99ff99", label="Virtual Machine 2 \n (private)"];

    master -> desktop
    desktop -> laptop;
    desktop -> workstation;
    desktop -> friendsworkstation
    prFromFriend -> desktop;
    master -> host2 -> host2private;
    master -> host3 -> host3private;
    master -> vm;
    vm -> vm1;
    vm -> vm2;
}

[TheLocehiliosan]

The "bare repo method" you mention above is precisely how yadm works under the hood.

I think what you are asking for is a way to manage files in place, but have the content published to the Git repo be sanitized in a custom way. I think that actually might be possible, if you don't mind doing some of the heavy lifting yourself.

I think it's possible to set a "filter" attribute in a .gitattributes file that refers to a custom "filter driver" of your own design. Checkout the documentation here for filter. https://git-scm.com/docs/gitattributes

You could write a driver that is informed by a configuration file that you keep private. Whenever content is added to or updated in the repo, if it has the filter driver enabled, your driver has the opportunity to sanitize the data that gets put in the repo. Likewise, when data is checked out of the repo and placed in your worktree, the driver has the opportunity to change the sanitized data back into the real data (informed by your configuration).

That's the most seamless way I could image to manage a sanitized set of dotfiles. Perhaps that's an idea you could run with. This is also an idea that I think would be completely compatible with yadm.

I would say Ansible/Salt might be a high cost of entry for any friends that want to try to use the data. If you go that route, I suggest making the end resulting public repo, something simple that doesn't involve much more than a repo and bootstrap script.

[ghost]

I think what you are asking for is a way to manage files in place, but have the content published to the Git repo be sanitized in a custom way. I think that actually might be possible, if you don't mind doing some of the heavy lifting yourself.

I think it's possible to set a "filter" attribute in a .gitattributes file that refers to a custom "filter driver" of your own design. Checkout the documentation here for filter. https://git-scm.com/docs/gitattributes

This is EXACTLY what I am looking for. This would then warn me if I try to commit something with sensitive data. Of course it will also be prudent to manually check any new configuration file that is entered to make sure that anything that is sensitive actually exists in the filter.

So for example I could set a filter that filters for things like "username", "password", "2001::", 192.168.1.0, "192.168.2" "192.168.3", "my name", "pgp key id".

You could write a driver that is informed by a configuration file that you keep private. Whenever content is added to or updated in the repo, if it has the filter driver enabled, your driver has the opportunity to sanitize the data that gets put in the repo. Likewise, when data is checked out of the repo and placed in your worktree, the driver has the opportunity to change the sanitized data back into the real data (informed by your configuration).

So you're suggesting a simple shell script that calls sed on a bunch of the things in the .gitattributes? This would "automate" the process, and in that case I'd only have to give it a look over before committing to be confident.

If you have an examples of what a driver might look like (or you use one yourself), feel free to share! 😃

That's the most seamless way I could image to manage a sanitized set of dotfiles. Perhaps that's an idea you could run with. This is also an idea that I think would be completely compatible with yadm.

Taking the advice of /u/ClutchHunter he said:

I wonder if you might be overcomplicating a bit? I've split stuff up into OS-specific dirs

The thing I was thinking might be good about yadm is that I wouldn't need to have separate branches for desktop, server, router, virtual machine etc would I? I could use your alternate files feature.

As far as git would be concerned it would look like:

digraph graphname {
    node [shape=rectangle, style="filled"];

    dotfiles [fillcolor="#ff9999", label="master (public) \n Desktop, laptop, workstation, server, vm"];
    friendsworkstation [fillcolor="#ffffbb", label="Friend's Workstation (private)"]
    prFromFriend [fillcolor="#ff9999", label="Friend opens PR (public)"]
    { rank=same dotfilesPrivate friendsworkstation prFromFriend }

    dotfilesPrivate [fillcolor="#99ff99", label="master (private) \n Desktop, laptop, workstation, server, vm"]

    dotfiles -> dotfilesPrivate
    dotfiles -> friendsworkstation
    prFromFriend -> dotfiles
}

• Non-sensitive files committed straight to red branch
• Pulls from red branch to green branch
• Files never go from "green branch" to "red branch".

I would say Ansible/Salt might be a high cost of entry for any friends that want to try to use the data. If you go that route, I suggest making the end resulting public repo, something simple that doesn't involve much more than a repo and bootstrap script.

Thinking about it now, I don't think I'd benefit from it. yadm has a bootstrap feature that would do what I want. You show initialization of submodules for vim, I could get it to run pacman?

My understanding was in regard to Ansible/Saltstack those are really more for remote orchestration. Ie if I have 1000 servers and I need to partition and format them all to be exactly the same, and then push configs to them.

This isn't really what I am doing. I'm doing the reverse pulling from some git repo on a system that already has git, and yadm.

Additionally I like how yadm doesn't depend on anything funky like 'ruby' or 'go' or something that very well may not be on a system. This may be best as you said entry for friends as it's fairly likely they will have sh.

I could also meet my other request, and keep sensitive files encrypted with my pgp key, using your encryption option. I could push these to my "green branch" directly, as these are not stored anywhere on the internet, gitlab/github etc. I could use .gitattributes to make sure none of these files ever end up on the "red branch".

[ghost]

Files never go from "green branch" to "red branch".

I had another thought about this, and I kind of noobed up. I had been using the word "branch" when I actually mean "fork". Late night, and I'm a noob.

I had another thing about this as well. What am I trying to achieve? well all I am really worried about is the "private fork" committer information being pushed to the "public fork."

If I configured yadm to push directly to my private fork (green), and then was able to push certain commits to the public fork (red), filter them and then remove the actual comitter name and put the public name on there.

Also: I'm going to be tracking files in $HOME and in /etc. The neatest way I can think of doing this is to copy the files from /etc (that I have modified) to a local directory, ie ~/etc.

The permissions of the "real" files in /etc cannot be allowed to change (that would introduce all sorts of security problems), and I don't want to run yadm as superuser anyway. Additionally symlinks can't work to files that require the correct permissions. The files in /etc rarely change anyway.

Sample files:

.
├── etc
│   └── iptables
│       └── rules6-save    <-- Exists on ##Router##Linux##gateway
└── home
    ├── .ssh
    │   ├── config         <-- Exists on ##Workstation##Linux
    │   └── id_rsa         <-- Exists on ##Workstation##Linux
    └── .weechat
        ├── irc.conf       <-- Exists on ##Workstation##Linux
        ├── sec.conf       <-- Exists on ##Workstation##Linux
        └── weechat.conf   <-- Exists on ##Workstation##Linux

Initialized repo in $HOME:

user@host:~$ yadm init
Initialized empty shared Git repository in /home/user/.yadm/repo.git

Get file from router:

user@host:~/$ scp -r root@router:/etc/iptables/rules6-save ~/etc

Change class to Router:

user@host:~/$ yadm config local.class Router

Add file to repository:

user@host:~/etc$ yadm add ~/etc/iptables/rules6-save

Change class back to Workstation to commit some more files

user@host:~/etc$ yadm config local.class Workstation

Commit ssh config:

user@host:~$ yadm add ~/.ssh/config

Now what I'm not sure if is this is the correct way to use alternates. The documentation didn't say I have to call the file ~/etc/iptables/rules6-save##Router##Linux##gateway or am I supposed to switch contexts like that?

I don't want to really be actualy doing the comitting on my router, because it doesn't even have git. (Note in this case the router is just a raspberry pi with debian on it).

Add encrypted file:

echo ".ssh/*_key" >> ~/.yadm/encrypt

Allows me to keep my files encrypted. I don't like to upload these to the internet anywhere! I guess I could say yes, if I was pushing to the private fork.

user@host:~$ yadm encrypt ~/.ssh/id_rsa_thingfox_key
Encrypting the following files:
.ssh/id_rsa_thingfox_key

Wrote new file: /home/user/.yadm/files.gpg
It appears that /home/user/.yadm/files.gpg is not tracked by yadm's repository.
Would you like to add it now? (y/n)
n

user@host:~/.weechat$ echo ".weechat/sec.conf" >> ~/.yadm/encrypt
user@host:~/.weechat$ yadm encrypt ~/.weechat/sec.conf 
Encrypting the following files:
.ssh/id_rsa_thingfox_key
.weechat/sec.conf

These files aren't super secret, though they do have private information I want to filter out.

user@host:~/.weechat$ yadm add ~/.weechat/irc.conf 
user@host:~/.weechat$ yadm add ~/.weechat/weechat.conf

I did observe when comitting I could not do:

user@host: yadm commit user.email "me@private.example.com" user.name "Private Citizen"
error: pathspec 'user.email' did not match any file(s) known to git.
error: pathspec 'me@private.example.com' did not match any file(s) known to git.
error: pathspec 'user.name' did not match any file(s) known to git.
error: pathspec 'Private Citizen' did not match any file(s) known to git.

So I had to set:

user@host:~$ git config --global user.email me@private.example.com
user@host:~$ git config --global user.name "Private Citizen"

Now out of these files I need to create a filter:

user@host:~/yadm-test-area$ grep -r . |grep SECRET

If you notice here, there's not strictly any passwords or usernames, but it would be nice to filter SECRET_IRC_NETWORK and just delete the whole line.

home/.weechat/irc.conf:SECRET_IRC_NETWORK.addresses = "irc.secret.example.com"
home/.weechat/irc.conf:SECRET_IRC_NETWORK.ssl = on
home/.weechat/irc.conf:SECRET_IRC_NETWORK.ssl_cert = "~/.weechat/ssl/SECRET_IRC/SECRET_IRC-SECRET_NAME.pem"
home/.weechat/irc.conf:SECRET_IRC_NETWORK.ssl_priorities = "NORMAL:-VERS-SSL3.0"
home/.weechat/irc.conf:SECRET_IRC_NETWORK.ssl_dhkey_size
home/.weechat/irc.conf:SECRET_IRC_NETWORK.ssl_fingerprint = "SECRET_FINGERPRINT"
home/.weechat/irc.conf:SECRET_IRC_NETWORK.ssl_verify = on
home/.weechat/irc.conf:SECRET_IRC_NETWORK.sasl_username = "SECRET_USERNAME"
home/.weechat/irc.conf:SECRET_IRC_NETWORK.nicks = "SECRET_NAME"
home/.weechat/irc.conf:SECRET_IRC_NETWORK.username = "SECRET_USERNAME"
home/.weechat/irc.conf:SECRET_IRC_NETWORK.realname = "SECRET_NAME"

With these I'd like to just change SECRET_HOST to HOST, SECRET_NAME to USERNAME

home/.ssh/config:Host SECRET_HOST
home/.ssh/config:    Hostname SECRET_HOST
home/.ssh/config:    User SECRET_NAME
home/.ssh/config:    IdentityFile ~/.ssh/id_ed25519_SECRET_NAME

With this one I'd want to change the 2001:MY:SECRET:ASSIGNED:RANGE::/64 to 2001:db8:AAA:AAA:AAA::/64 (reserved address for documentation)

etc/iptables/rules6-save:-A INPUT -s `2001:MY:SECRET:ASSIGNED:RANGE::/64 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT

I think what you are asking for is a way to manage files in place, but have the content published to the Git repo be sanitized in a custom way. I think that actually might be possible, if you don't mind doing some of the heavy lifting yourself.

I think it's possible to set a "filter" attribute in a .gitattributes file that refers to a custom "filter driver" of your own design. Checkout the documentation here for filter. https://git-scm.com/docs/gitattributes

You could write a driver that is informed by a configuration file that you keep private. Whenever content is added to or updated in the repo, if it has the filter driver enabled, your driver has the opportunity to sanitize the data that gets put in the repo. Likewise, when data is checked out of the repo and placed in your worktree, the driver has the opportunity to change the sanitized data back into the real data (informed by your configuration).

Now my understanding is the filter could be two-way ie:

• When I go to push from my private fork to my public one 2001:MY:SECRET:ASSIGNED:RANGE::/64 would become 2001:db8:AAA:AAA:AAA::/64
• When I go to push from my private fork 2001:db8:AAA:AAA:AAA::/64 could become 2001:MY:SECRET:ASSIGNED:RANGE::/64

I very much appreciate the help, and will be writing this all up on my github page when I get it working.

I'll set up two public forks, to simulate, ie public_dotfiles and private_dotfiles (but in reality both will be public).

Is where I put the test files mentioned in this post: https://github.com/thingfox/files_for_testing_yadm

[ghost]

So I think I solved the alternative files thing. I renamed them all with git mv. I obviously misread the documentation when I thought that the naming convention was a part of yadm.

user@host:~$ yadm init
Initialized empty shared Git repository in /home/user/.yadm/repo.git/

user@host:~$ yadm add ... each file...

user@host:~/.weechat$ yadm commit -a
[master (root-commit) f4f8514] Initial import dotfiles
 7 files changed, 641 insertions(+)
 create mode 100644 .gnupg/gpg.conf##Workstation
 create mode 100644 .ssh/config##Workstation
 create mode 100644 .ssh/id_rsa_thingfox_key##Workstation
 create mode 100644 .weechat/irc.conf##Workstation
 create mode 100644 .weechat/sec.conf##Workstation
 create mode 100644 .weechat/weechat.conf##Workstation
 create mode 100644 etc/iptables/rules6-save##Router

The repository is at https://github.com/thingfox-private/dotfiles

Now to figure out how to:

• filter files a filter driver I'm a bit stuck on this one
• push them from thingfox-private/dotfiles to thingfox/dotfiles

The thing I am concerned about is during that push process how do i prevent the commit showing up: ie

commit f4f851428894a9011816561604fd413de7ab5602
Author: Thing Fox Private <thingfox-private@thingfox-private.example.com>
Date:   Tue Feb 12 02:59:48 2019 +0000

    Initial import dotfiles

Essentially I don't want anyone to be able to figure out publicly from the commit logs on https://github.com/thingfox/dotfiles.git where the commits came from.

[jonasc]

Some of the things you seem to be struggling with are less yadm related things but more git stuff as generally yadm hands most of the things straight through to git.

So to prevent the name and email address from showing up differently on different hosts you have to set them to be the same by yadm config user.name/email (or yadm gitconfig user.name/email?). You can also prevent timezone information to pop up in the git commit date.

[jonasc]

For .gitattributes consider the following minimal example:

# .git/config
[filter "clandestine"]
  clean = sed --file=clean.sed
  smudge = sed --file=smudge.sed
  required

This creates a filter named clandestine which calls sed with the script clean.sed for removing secret data and smudge.sed to restore secret data. required means that the files will not be added if the filter fails (i.e., returns a non-zero exit code) for some reason.

The respective scripts are very basic and the idea is just that you don't want to expose your secret ip address 1.2.3.4 to the public but want to replace it with MY.SECRET.IP.ADDRESS.

# clean.sed
s/\b1\.2\.3\.4\b/MY.SECRET.IP.ADDRESS/g

# smudge.sed
s/\bMY\.SECRET\.IP\.ADDRESS\b/1.2.3.4/g

Now set which files should be automatically cleaned:

# .gitattributes
*.conf filter=clandestine

Add two files with the same content:

# test.conf / test.ini
Host 1.2.3.4
Host 8.8.8.8

After adding the two files to the git git add test.conf test.ini you can verify that the filter actually worked:

$ git diff --cached
diff --git a/test.conf b/test.conf
new file mode 100644
index 0000000..7709ff4
--- /dev/null
+++ b/test.conf
@@ -0,0 +1,2 @@
+Host MY.SECRET.IP.ADDRESS
+Host 8.8.8.8
diff --git a/test.ini b/test.ini
new file mode 100644
index 0000000..3104ddd
--- /dev/null
+++ b/test.ini
@@ -0,0 +1,2 @@
+Host 1.2.3.4
+Host 8.8.8.8

As you can see in test.conf the filter automatically removed 1.2.3.4 with MY.SECRET.IP.ADDRESS.

[ghost]

@jonasc awesome example thanks! https://github.com/TheLocehiliosan/yadm/issues/143#issuecomment-462477711

Some of the things you seem to be struggling with are less yadm related things but more git stuff as generally yadm hands most of the things straight through to git.

Yeah you're right. I have in the past only really had experience with forking (other people's public repositories), branching, of my own and submitting PR requests with minor changes, so I am learning a lot about git with a real example. I am terrible at learning from just "RTFM" 😄. Thanks for explaining it!

I have been thinking more now I don't actually think the "private repository" and "public repository" would be either a fork or a branch, but rather two distinct repositories. The process being for data going between the two:

• pull public > filter > private (commit as private identity)

• Use cp to copy files from one repository to the other.

• pull private > filter > public (commit as public identity)

In this case my private files would be in ~/.yadm/repo.git/ these would be files anywhere within my home directory.

I would also have a second "public" repository ~/dotfiles

So to prevent the name and email address from showing up differently on different hosts you have to set them to be the same by yadm config user.name/email (or yadm gitconfig user.name/email?).

Yes I must have missed that in the FAQ.

You can also prevent timezone information to pop up in the git commit date.

So I'm not so worried about this for my private repository (note this account is a simulation, and I'm well aware it is public.) When I come to doing this for real the private repository will be on a git server on my LAN. Commits to it would only be possible from the LAN. I will definitely use this when it comes to committing to thingfox/dotfiles.

ie:

user@host:~$ git clone https://github.com/thingfox/dotfiles
user@host:~/dotfiles$ git config user.name "Thing Fox"
user@host:~/dotfiles$ git config user.email "thingfox@thingfox.example.com"
user@host:~/dotfiles$ git config alias.utccommit '!git commit --date="$(date --utc +%Y-%m-%dT%H:%M:%S%z)"'

I was able to reproduce everything you did there. In practice I'm probably going to use perl instead of sed as I soon realized that sed is a bit annoying.

An example of that being: https://superuser.com/questions/440013/how-to-replace-part-of-a-text-file-between-markers-with-another-text-file

I have perl already. In fact git depends on it. One of my friends will very much appreciate that as he's been a perl programmer for years, but would probably have to think about sed. 😄

[abathur]

@thingfox If you haven't, (sorry if this got addressed here; I haven't read the whole thread) consider whether the secret/private data "need" to be under version control at all. It might be fine to exclude them from yadm and use a secret manager or encrypted backup tool to manage/save/restore them.

I'm not doing exactly what you are, but I've been slowly refining my repository and bootstrap towards being able to quickly stand up a mostly-configured system (NixOS desktop, macOS laptop) flexibly and without leaving a bunch of risky footprints. I'm a long way from done, but there's a lot of potential here... A few thoughts:

• My bootstrap restores my main project/work directory from a snapshot in an encrypted backup repository. The backup repo locations (whether LAN or remote), credentials, API keys, and even which repo to use, are all stored in a secret manager, which my bootstrap prompts me to unlock at the appropriate time.
• A separate phase of the bootstrap also restores backups of the running state for my browser and terminal emulator, so that I can port my windows/tabs/scrollback/history/etc.
• I currently have some keys/licenses/etc. encrypted in yadm and have similar paranoia about posting them publicly; if I wanted to scrub the repo I'd definitely be turning these into attachments in the secret manager or including them in backups.
• If I wanted to scrub usernames/IPs/hostnames/domains/etc. from some config files but definitely wanted the configs themselves to stay under version control, I'd look at turning them into templates, populate the templates with values from the secret manager, and only track the templates.

[ghost]

@thingfox If you haven't, (sorry if this got addressed here; I haven't read the whole thread) consider whether the secret/private data "need" to be under version control at all. It might be fine to exclude them from yadm and use a secret manager or encrypted backup tool to manage/save/restore them.

Yep, I intend to do that.

I'm not doing exactly what you are,

I was engineering a perl script that will filter out sensitive data and dump it in a separate file, this can then be uploaded to the repository encrypted.

The script looks for certain "start" and "end" comments, for example this example from ~/.ssh/config:

#####ssh config block @file_name ~/.ssh/config######

Host NSA
    Hostname 203.0.113.1
    Port 22
    User nsa
    IdentityFile ~/.ssh/id_ed25519_nsa

Host CIA
    Hostname 203.0.113.2
    Port 22
    User cia
    IdentityFile ~/.ssh/id_ed25519_cia

#####ssh config block @file_name ~/.ssh/config######

Host virtuamachine
    Hostname 127.0.0.1
    Port 22
    User virtualmachine
IdentityFile ~/.ssh/id_ed25519_vm

1. The filter perl script will look for the sensitive comment block using the regular expression
2. The filter script will cut the portion inside the comments and put it in a separate file (smudge.txt which will be encrypted). When ~/.ssh/config is committed it will look like this:

#####ssh config block @file_name ~/.ssh/config######

#####ssh config block @file_name ~/.ssh/config######

Host virtuamachine
    Hostname 127.0.0.1
    Port 22
    User virtualmachine
IdentityFile ~/.ssh/id_ed25519_vm

3. When the smudge script is run it will look in smudge.txt for the contents, it will then:
   1. Cut the portion between the block comments and put it into the file being checked out
   2. Delete the block comments in the smudge.txt file. (Otherwise there would be duplicates when checking in again).

#####ssh config block @file_name ~/.ssh/config######

Host NSA
    Hostname 203.0.113.1
    Port 22
    User nsa
    IdentityFile ~/.ssh/id_ed25519_nsa

Host CIA
    Hostname 203.0.113.2
    Port 22
    User cia
    IdentityFile ~/.ssh/id_ed25519_cia

#####ssh config block @file_name ~/.ssh/config######

#####some other config block @file_name ~/path/where/other/file/is######
 .....
#####some other config block @file_name ~/path/where/other/file/is######

If I wanted to scrub usernames/IPs/hostnames/domains/etc. from some config files but definitely wanted the configs themselves to stay under version control, I'd look at turning them into templates, populate the templates with values from the secret manager, and only track the templates.

I wonder if Jinja templating might have been a better way to do this? This makes a lot more sense where I want to censor portions of a line such as an IP in an iptables config.

Not only that it wouldn't be relying on a censor block. The values would already not be there to begin with, and therefore couldn't "accidentally" get committed. Ie for example say I added another host but made a human error of not putting it in the #####ssh config block @file_name ~/.ssh/config###### section and didn't notice it when running git commit and git push.

[ghost]

Perhaps it would be possible for @TheLocehiliosan to update the jinja section to suggest that templates could be used in this way.

When I looked at that man file originally, I thought that jinja templating could only be used with the logic determining which version of the ALTERNATES file feature that is used. Not for importing secrets from a file that might be encrypted (ie using the ENCRYPTION option into a .j2 template.

If the envtpl command is available, Jinja templates will also be pro-cessed to create or overwrite real files. yadm will treat files ending in ##yadm.j2

Perhaps someone could help me with the above ssh example.

[abathur]

Good question, re: Jinja. I'm not certain and haven't tried it (to be honest, I'd forgotten about yadm's Jinja support--though I was myself already imagining manually using Jinja for what I described.) The README for envtpl (https://github.com/andreasjansson/envtpl) suggests any environment variables will be available in the template.

Not quite certain from your description there what flow you're imagining--I was just thinking of extracting the values into variables from a secret/password manager with a command-line client, and then building them into the template (you could further use the values yadm adds to the template to decide which variables get used, of course...)

[ghost]

Not quite certain from your description there what flow you're imagining--I was just thinking of extracting the values into variables from a secret/password manager with a command-line client, and then building them into the template (you could further use the values yadm adds to the template to decide which variables get used, of course...)

The idea was I'd check templates in, with the variables templated and then when checking out they would be replaced with certain values depending on which version was called ie ##CLASS.OS.HOSTNAME.USER.

For example if I was on my Linux workstation and I checked out my ssh config I would be getting ~/.ssh/config/config##Workstation##Linux##yadm.j2 which would drop ~/.ssh/config/config. Then this would additionally process the secrets. A possible place for that might be ~/secrets/.ssh/config##Workstation##Linux which I could use the ENCRYPTION option or store in a dm-crypt container.

I don't know whether this is possible with Yadm. I guess what I have to figure out is if I would use the smudge hook to specify the location of the secrets file, ie ~/secrets/.ssh/config##Workstation##Linux and does smudging happen before or/after the jinja processing occurs? My guess is the git hooks happen before the jinja templates are executed.

I guess it should also be possible to use a bootstrap command to get envtpl to put the secrets in there. The jinja templates section there explains it better. I guess I'd need to figure if it's possible to set a variable which is the path of the secrets file.

[ghost]

@abathur

I was just thinking of extracting the values into variables from a secret/password manager with a command-line client, and then building them into the template (you could further use the values yadm adds to the template to decide which variables get used, of course...)

So I've given this a bit of a look today using the test files files_for_testing_yadm.

In particular rules6-save##Router.tpl.

Using the command:

MY_RANGE=2001:MY:SECRET:ASSIGNED:RANGE \
envtpl --keep-template ~/.config/etc/iptables/rules6-save##Router.tpl

Which I could use in a bootstrap script I could put all the values I want back in. This is great for a single value in a file. Now envtpl apparently supports inputing a json structure and this could be useful in the config##Workstation.tpl example.

Consider this template, config##Workstation.tpl

Host {{ CENSORED_VALUE }}
    Hostname {{ CENSORED_VALUE }}
    Port 22
    User {{ CENSORED_VALUE }}
    IdentityFile ~/.ssh/id_ed25519_{{ CENSORED_VALUE }}

Host {{ CENSORED_VALUE }}
    Hostname {{ CENSORED_VALUE }}
    Port 22
    User {{ CENSORED_VALUE }}
    IdentityFile ~/.ssh/id_ed25519_{{ CENSORED_VALUE }}

Host BORING_HOST
    Hostname BORING_HOST
    Port 22
    User BORING_NAME
    IdentityFile ~/.ssh/id_ed25519_BORING_NAME

and the json structure to go with it:

[
    {"v":"SECRET_HOST_1"},
    {"v":"SECRET_NAME_1"},
    {"v":"SECRET_USER_1"},
    {"v":"SECRET_KEY_1"},
    {"v":"SECRET_HOST_2"},
    {"v":"SECRET_NAME_2"},
    {"v":"SECRET_USER_2"},
    {"v":"SECRET_KEY_2"}
]

Now with the template code to loop through the json structure: config##Workstation.tpl.j2

{% for x in CENSORED_VALUE | from_json
       %}{{ x.v }}{% 
   endfor
%}

I run it and I get:

$ CENSORED_VALUE=$(< ssh_values.json) envtpl <<< $(<config##Workstation.tpl.j2)
SECRET_HOST_1SECRET_NAME_1SECRET_USER_1SECRET_KEY_1SECRET_HOST_2SECRET_NAME_2SECRET_USER_2SECRET_KEY_2

This looks good, however when I try to feed it into the template file:

CENSORED_VALUE=$(< ssh_values.json) envtpl <<< $(<config##Workstation.tpl.j2) -o \
config##Workstation --keep-template config##Workstation.tpl

I get:

Host [
    {"v":"SECRET_HOST_1"},
    {"v":"SECRET_NAME_1"},
    {"v":"SECRET_USER_1"},
    {"v":"SECRET_KEY_1"},
    {"v":"SECRET_HOST_2"},
    {"v":"SECRET_NAME_2"},
    {"v":"SECRET_USER_2"},
    {"v":"SECRET_KEY_2"}
]
    Hostname [
    {"v":"SECRET_HOST_1"},
    {"v":"SECRET_NAME_1"},
    {"v":"SECRET_USER_1"},
    {"v":"SECRET_KEY_1"},
    {"v":"SECRET_HOST_2"},
    {"v":"SECRET_NAME_2"},
    {"v":"SECRET_USER_2"},
    {"v":"SECRET_KEY_2"}
]
    Port 22
    User [
    {"v":"SECRET_HOST_1"},
    {"v":"SECRET_NAME_1"},
    {"v":"SECRET_USER_1"},
    {"v":"SECRET_KEY_1"},
    {"v":"SECRET_HOST_2"},
    {"v":"SECRET_NAME_2"},
    {"v":"SECRET_USER_2"},
    {"v":"SECRET_KEY_2"}
]
    IdentityFile ~/.ssh/id_ed25519_[
    {"v":"SECRET_HOST_1"},
    {"v":"SECRET_NAME_1"},
    {"v":"SECRET_USER_1"},
    {"v":"SECRET_KEY_1"},
    {"v":"SECRET_HOST_2"},
    {"v":"SECRET_NAME_2"},
    {"v":"SECRET_USER_2"},
    {"v":"SECRET_KEY_2"}
]

Host [
    {"v":"SECRET_HOST_1"},
    {"v":"SECRET_NAME_1"},
    {"v":"SECRET_USER_1"},
    {"v":"SECRET_KEY_1"},
    {"v":"SECRET_HOST_2"},
    {"v":"SECRET_NAME_2"},
    {"v":"SECRET_USER_2"},
    {"v":"SECRET_KEY_2"}
]
    Hostname [
    {"v":"SECRET_HOST_1"},
    {"v":"SECRET_NAME_1"},
    {"v":"SECRET_USER_1"},
    {"v":"SECRET_KEY_1"},
    {"v":"SECRET_HOST_2"},
    {"v":"SECRET_NAME_2"},
    {"v":"SECRET_USER_2"},
    {"v":"SECRET_KEY_2"}
]
    Port 22
    User [
    {"v":"SECRET_HOST_1"},
    {"v":"SECRET_NAME_1"},
    {"v":"SECRET_USER_1"},
    {"v":"SECRET_KEY_1"},
    {"v":"SECRET_HOST_2"},
    {"v":"SECRET_NAME_2"},
    {"v":"SECRET_USER_2"},
    {"v":"SECRET_KEY_2"}
]
    IdentityFile ~/.ssh/id_ed25519_[
    {"v":"SECRET_HOST_1"},
    {"v":"SECRET_NAME_1"},
    {"v":"SECRET_USER_1"},
    {"v":"SECRET_KEY_1"},
    {"v":"SECRET_HOST_2"},
    {"v":"SECRET_NAME_2"},
    {"v":"SECRET_USER_2"},
    {"v":"SECRET_KEY_2"}
]

Host BORING_HOST
    Hostname BORING_HOST
    Port 22
    User BORING_NAME
    IdentityFile ~/.ssh/id_ed25519_BORING_NAME

Do you know how I might make envtpl put the value of v into the {{ CENSORED_VALUE }} instead of the whole json structure?

Also if you have any improvements there to the way I am doing this I'd much appreciate the advice.

[ghost]

So, I realized I was doing this the wrong way.

I decided to go with this json data:

[
    {"Host":"NSA", "Hostname":"203.0.113.1", "User": "nsa", "IdentifyFile":"~/.ssh/id_ed25519_nsa"},
    {"Host":"CIA", "Hostname":"203.0.113.2", "User": "cia", "IdentifyFile":"~/.ssh/id_25519_cia"}
]

Then loop over the data object and put it out into the file:

{% for v in CENSORED_VALUE | from_json %}
Host {{ v.Host }}
    Hostname {{ v.Hostname }}
    Port 22
    User {{v.User }}
    IdentityFile {{ v.IdentifyFile }}
{% endfor %}
Host BORING_HOST
    Hostname BORING_HOST
    Port 22
    User BORING_NAME
    IdentityFile ~/.ssh/id_ed25519_BORING_NAME

Then with this input command:

$ CENSORED_VALUE=$(<ssh_values.json) envtpl config##Workstation.tpl --keep-template -o config##Workstation

and walah:

Host NSA
    Hostname 203.0.113.1
    Port 22
    User nsa
    IdentityFile ~/.ssh/id_ed25519_nsa

Host CIA
    Hostname 203.0.113.2
    Port 22
    User cia
    IdentityFile ~/.ssh/id_25519_cia

Host BORING_HOST
    Hostname BORING_HOST
    Port 22
    User BORING_NAME
    IdentityFile ~/.ssh/id_ed25519_BORING_NAME

[ghost]

@TheLocehiliosan

Okay so now I'm testing this specifically in the context of yadm. With my example how should I go about applying the values from the json file?

I thought about using the bootstrap but that doesn't seem right as that happens afterwards and then I can't make use of alternate files ie because of -o.

I am having trouble with what I should name my ssh config template (currently config.tpl##Workstation). The problem is when I checkout I get config.tpl not config.

Should it be named config##yadm.j2? I looked at the way that Jinja2 templates are used specifically to yadm, ie ##yadm.j2 but if I do that how do I correctly take in my values from hosts.json?

[ghost]

So it occurred to me I could use ##yadm.j2 to control which parts of the bootstrap script are run.

For example my bootstrap script

#!/bin/bash

{% if YADM_CLASS == 'Workstation' -%}
    CENSORED_VALUE=$(cat ~/template_data/ssh/hosts.json) envtpl --keep-template ~/.ssh/config##Workstation.tpl -o ~/.ssh/config
    IRC_SERVERS=$(cat ~/template_data/weechat/irc_servers.txt) envtpl --keep-template ~/.weechat/irc.conf##Workstation.tpl -o ~/.weechat/irc.conf
    SEC=$(cat ~/template_data/weechat/sec.json) PASSPHRASE='tiddles' envtpl --keep-template ~/.weechat/sec.conf##Workstation.tpl -o ~/.weechat/sec.conf
{% elif YADM_CLASS == 'Router' -%}
    MY_RANGE='2001:db8:AAA:AAA:AAA' envtpl --keep-template ~/.config/etc/iptables/rules6-save##Router.tpl -o ~/.config/etc/iptables/rules6-save
{% elif YADM_CLASS == 'VirtualMachine' -%}
   echo "NOTE: Some configs for virtual machines"
{% else -%}
   echo "ERROR: Unknown class selected"
{% endif -%}

Then I simply delete the old bootstrap, check out the right class and give execute privileges.

rm -f ~/.yadm/bootstrap; yadm config -f local.class Workstation; chmod +x ~/.yadm/bootstrap
yadm bootstrap; yadm perms

[ghost]

I think I'm done now. @TheLocehiliosan do you want to add my repo to https://yadm.io/docs/examples this isn't my full dotfiles those are on a private repo, but I am putting the ones across here that have general logic that would be useful to others.

I noticed a lot of the other examples don't really use templates, or use envtpl to template the content of their config files like I have. Some of them are also "very specific". I have documented how it works in README.md.

Some notable things I have done:

1. Build SSH hosts from json data file: hosts.json

2. Build my bashrc PATH from a json array: path.json

3. Allow yadm to bootstrap vim plugins or not, depending on if an external variable is set. Sometimes I just want to re-bootstrap my files without downloading external things.

4. Build server strings up for weechat using input json data servers.json

5. It might also be worth mentioning the | from_json filter uses these internal python decoding rules https://docs.python.org/3/library/json.html#encoders-and-decoders

@abathur

I was myself already imagining manually using Jinja for what I described.

I was just thinking of extracting the values into variables from a secret/password manager with a command-line client, and then building them into the template (you could further use the values yadm adds to the template to decide which variables get used, of course...)

😀 done any improvements?

[TheLocehiliosan]

@thingfox - Go ahead and create a PR adding your example to the /_docs/070_examples.md file. Base the PR on the dev-pages branch. I haven't really had time to delve into your scenario, because I've been writing a bunch of documentation right now, but I hope to parse through it soon, and I'm glad you found a workable solution.

[ghost]

Done https://github.com/TheLocehiliosan/yadm/pull/148

[ghost]

6. I was able to bootstrap a list of packages stored in a json file and then choose to bootstrap pacman + yay or apt-get depending on what distribution I was on.

[markstos]

My solution for public vs. private dotfiles was to make my dotfiles completely private. This way I don't accidentally leak something I shouldn't, nor do I spend time maintaining the distinction.

I love open source and I love that people share their dotfiles as examples. If I see a situation where my dotfiles feel like they would be a particularly useful example, I can publish a copy of those separately, like a Gist, or give them directly to someone. In practice, I haven't found that my dotfiles actually contain much that someone hasn't already published somewhere, so I don't think the world is missing much from this arrangement and my life is simpler.