Dotbot 2.0

[anishathalye]

This issue is meant for tracking work on Dotbot v2.0, the first major release since v1.0 in March 2014. All features that are backwards compatible are being included in Dotbot v1.0 as they are added. We'll only switch to the name "v2.0" if/when we decide to add features that are not backwards compatible.

Completed features

• Directive plugins (see #65)
• Default options (see #34 and #77)

Accepted features / changes

• Improved error messages (see #73)
• Backup (see #90)

Features under consideration

• CLI flag to only run certain sections (see #44 and #49, 2nd item)
• Platform specific link paths (see #57)
• Be able to write to files/folders that require special privileges (see #49, 3rd item)
• Uninstall command to delete symlinks specified in install config (see #49, 4th item)
• Automatic backup / replacement of dotfiles (see #12 and #36)
• Support interactive execution (see #74)
• Make Dotbot usable as a Python library (see #75)

Rejected features

(none)

---

Design decisions

(nothing here yet)

To be discussed

• What file format do we want to use? There are a couple options I can think of: JSON/YAML file (declarative), other data format (e.g. HOCON), or Python DSL. Maybe there are other completely different ways to do it too.
• Because people use Dotbot as a submodule, we need to be careful when figuring out how to ship the upgrade. Should it be on the master branch? Should Dotbot 2.0 forever live in a 2.0 branch with master being reserved for bugfixes for 1.0?

---

Release information

The release date is currently unknown. Work on Dotbot v2.0 has not started yet.

(This top post will be edited as stuff happens. This issue will be used for discussing the design of v2.0.)

[anishathalye]

There may be some config file API changes required to have a clean way to support the XDG Base Directory Specification. See #23.

[anishathalye]

It would be great to provide better error messages when there are problems with the config file. Currently, if there are YAML syntax errors, the error messages are decent. If the config file is valid YAML but doesn't make sense for Dotbot, the errors are pretty bad. If there is going to be a new config file format for Dotbot 2.0, this should be kept in mind so that we can be sure to have high-quality error messages. See #8 and #27 for related discussion.

[anishathalye]

In the current config file format, there isn't a clear place to put default options. This seems like it is a nice feature to have, and this should be kept in mind if designing a new config file format. See #34.

[joshblum]

+1 for the error messages

[anishathalye]

Having automatic replacement and backup of old dotfiles would be nice. This was discussed in #12 and #36. Perhaps the cleanest way to support this would be through default options as described in #12.

[anishathalye]

Some more new features were suggested in #49.

[scarfacedeb]

@anishathalye what do you think about adding meta/comment properties to the executors in 2.0? (As described in https://github.com/anishathalye/dotbot/pull/1)

[casey]

I updated casey/dotbot-user-study with a summary of the results. It may be of interest when figuring out what people are doing with dotbot and designing 2.0.

[anishathalye]

@scarfacedeb I'm not sure... If we can find a clean enough API, then I might not be opposed to it.

[anishathalye]

@casey Wow, that's really interesting. It'll certainly come in handy while working on the design of Dotbot 2.0.

[anishathalye]

Okay, I rewrote the first post to have a (continuously updated) summary of what's going on with the design and implementation for Dotbot 2.0.

This issue should be used for discussing the things mentioned in the first post and v2.0 in general.

[anishathalye]

Just thinking out loud about what the Dotbot 2.0 config file format might look like...

We can consider the following Dotbot 1.0 config (based on my current dotfiles config):

- clean: ['~', '~/.config']

- link:
    ~/.agignore:
      path: agignore
      relink: true
    ~/.dotfiles: ''
    ~/.gitconfig:
      path: gitconfig
      relink: true
    ~/.gnupg/gpg.conf:
      create: true
      relink: true
      path: gpg.conf

- shell:
  -
    command: git update-submodules # after linking ~/.gitconfig
    stdout: true

Here's a format that looks YAMLish but is cleaner for use with Dotbot:

defaults:
  link:
    relink: true

clean: ['~', '~/.config']

link:
  ~/.agignore: agignore
  ~/.dotfiles: ''
    relink: false
  ~/.gitconfig: gitconfig
  ~/.gnupg/gpg.conf: gpg.conf
    create: true

shell:
  - git update-submodules # after linking ~/.gitconfig
    stderr: true

So it's almost like the YAML, but there are some differences. At the toplevel, it's a list by default, and so there's no need to have a list with dictionaries of single elements inside. In addition, dictionaries map keys to values, but can have additional named properties attached. Same thing for arrays.

The changes aren't huge, but I think it significantly improves readability.

Unfortunately, this no longer maps cleanly to JSON/YAML. And it would require a custom parser. I wonder if there are any markup languages that support structures like this...

Another option is to use a Python DSL. Maybe it could look something like this:

from dotbot.dsl import *

clean([
  '~',
  '~/.config'
])

link.default(relink=True)
link({
  '~/.agignore': 'agignore',
  '~/.dotfiles': _('', relink=False),
  '~/.gitconfig': 'gitconfig',
  '~/.gnupg/gpg.conf': _('gpg.conf', create=True)
})

shell([
  _('git update-submodules', stderr=True)
])

We could also support more than one format, allowing either a declarative YAML (for simpler configs) or a Python DSL config (for more complicated setups).

[anishathalye]

Or something very different -- an S-expression based syntax:

(defaults
  (link (relink true)))

(clean "~" "~/.config")

(link
  ("~/.agignore" "agignore")
  ("~/.dotfiles" ""
    (relink false))
  ("~/.gitconfig" "gitconfig")
  ("~/.gnupg/gpg.conf" "gpg.conf"
    (create true)))

(shell
  ("git update-submodules"
    (stderr true)))

[kaluzki]

As i started to busy myself with dotfiles i landed pretty quickly in your repository. What i liked is especially the aspect "DotBot is designed to be lightweight and self-contained, with no external dependencies and no installation required.". But i also find that including of DotBot itself as submodule goes again this principle. Because the configuration file is actually the only thing i'm interested in.

For this reason i wrote a little script "https://git.io/dotfiles.sh" which installs ones dotfiles repository from github e.g. "curl https://git.io/dotfiles.sh -L | sh -s anishathalye". DotBot is currently supported as installation method for the following configuration files:

.dotbot.yaml dotbot.yaml .dotbot.json dotbot.json .install.conf.yaml install.conf.yaml .install.conf.json install.conf.json

For my personal dotfiles i use YAML format which covers my wishes entirely.

[anishathalye]

Huh, that's interesting to hear. Any idea how many people are using that tool?

In general, the curl pipe sh pattern is dangerous. It requires that you not only trust GitHub (with HTTPS, that's reasonable), but also that you trust the script, which could change at any time.

With including Dotbot as a submodule, you only need to trust GitHub. You don't even need to trust the Dotbot repo -- if you audit my code and add that as a submodule, if that code in the Dotbot repo later turned malicious somehow, the submodule hash wouldn't match.

So there is that tradeoff...

[rubenvereecken]

I only started using dotbot as of today after a desperate search for the ideal tool. Happy so far.

There's some good to be said for using a well-known format (YAML) over an SDL, especially if the latter doesn't bring that many advantages. The Lisp-like syntax looks horrible.

The Python DSL looks interesting. That'd just be a regular Python file right? Interesting for some more intelligent logic.

[MaikuMori]

The current submodule pattern is good. It's self-contained. You clone/pull the repo and you're ready to install or update.

Logic in config files would be nice but I think there are only few cases where you would want it. If you want full blown logic you could just write everything in bash or any scripting language yourself and not use dotbot.

Creating new config language in my opinion is not a good idea. First of all it's overkill for something like this. Secondly it would just be another thing for the end-user to learn. The language should be something popular and not too verbose. Supporting multiple configuration languages is nice as long as it doesn't take too much time away from the main focus of the project.

As for me I think the install update process could be better. The errors could be better as well. It would be nice if dotbot provided manual actions with smart options. Like ask user what to do if file is already there. Users who want to fully automate everything should have a way to disable interactive execution.

Maybe dotbot could detect new dotfiles and help you automate moving them to your dotfiles repository and setting up links.

Basically all sorts of smart/interactive things would be nice additions and would bring more to the table than long discussions about favorite configuration format that people have.

[bchretien]

If dotbot is easily usable as a Python module (not just through the CLI), people could design more complex setups directly in Python. That means you get the full power of Python and the ease of use of dotbot, without adding support for more complex configuration files.

[MaikuMori]

@bchretien that's very good point I was also thinking about. If it was combination of CLI tool and library then you have the best of both. I also like the recent addition of plugins.

[anishathalye]

@rubenvereecken I agree with your thoughts on YAML vs a DSL vs the Lisp-like syntax. And yeah, the Python DSL will be just a regular Python file.

---

@MaikuMori Hmm, I guess I agree with your opinion about logic in config files... it might get out of hand, and then there's no more simplicity. And I also agree on not creating another config file language. If v2.0 supports YAML, then we might as well support JSON too; not sure about other formats.

What do you mean by the "install update process"?

Yeah, I agree that the error messages should be better. See #73.

Interactive execution is a cool idea. Adding to "features under consideration" above. See #74.

There's already a script @Vaelatern wrote for automating dotfiles setup: https://github.com/Vaelatern/init-dotfiles (it's linked in the readme for Dotbot). I don't think this functionality needs to be built into Dotbot itself, especially since it's probably a one-time thing.

---

@bchretien Interesting idea! Adding to "features under consideration" above. See #75.

[aykamko]

Regarding implementing default options (#34):

One solution that stays mostly true to vanilla YAML and JSON would be to parse defaults from a specially-formatted comment section at the top of a config file, similar to vim modelines:

#- dotbot-defaults:
#-  link:
#-    force: true
#-  shell:
#-    stdout: true

- link:
    ~/.vimrc: vimrc
- shell:
  - echo hi

This is a lighter solution than implementing a full custom YAML parser where everything is a list by default at the top level. We'd need only to read the file, pull the first few specially commented lines (if they exist), and then parse as regular YAML/JSON after striping the special chars.

The only downside to this approach is that we wouldn't be following strict JSON format if we allowed the same approach in JSON configs, since the JSON standard doesn't allow comments. But we can easily sanitize the comments out before passing the body of the config to a standard parser.

And this wouldn't be an issue with the proposed Python DSL, since we could just use a Python directive in that case. :stuck_out_tongue:

@anishathalye Thoughts? I could work on this and whip up a pull request.

[casey]

This could be implemented using multi-document YAML streams using the '---' delimiters, to keep it pure YAML:

link-defaults:
  force: true
shell-defaults:
  stdout: true
---
- link:
    ~/.vimrc: vimrc
- shell:
  - echo hi

[aykamko]

@casey Whoa, didn't even know that was a thing. This seems like the best approach, even if we break from standard JSON a bit.

[casey]

@aykamko Yeah, YAML is pretty slick ^_^

JSON config files could be easily supported by splitting the document on --- lines, and then passing the chunks to the JSON parser.

I think this might be a really promising approach, since it's super simple, keeps YAML config files pure-YAML, only extends JSON config files a little bit, and should be backwards compatible with all previous config files.

Since defaults are a dictionary and directives are a list, the config parser can very simply validate a config file; it may be a single document containing a list of directives to run, or a stream of two documents where the first is a dictionary, which gets used as defaults, and where the second is a list of directives, to be run.

The algorithm would look like:

def parse_config(yaml_string):
  documents = parse_yaml(yaml_string)
  count = len(documents)
  if count == 0:
    sys.exit('config was empty')
  elif count == 1:
    directives = documents[0]
    run_directives(directives)
  elif count == 2:
    defaults = documents[0]
    directives = documents[1]
    apply_defaults(documents[0])
    run_directives(documents[1])
  elif count > 2:
    sys.exit('config had too many documents')
    # we could also allow more than two documents, so that
    # people could run multiple sets of directives with different
    # defaults, but it's probably best to forbid that until someone
    # asks for it

[anishathalye]

Hmm, that looks pretty good. Come to think of it, we could also do it like this as a single YAML document (and this way, it translates to JSON nicely too).

- defaults:
    - link:
        force: true
    - shell:
        stdout: true
- link:
    ~/.vimrc: vimrc
- shell:
  - echo hi

[Darksider3]

Pls use a default configuration format... YAML is good one, or just a simple INI or JSON. But please, please, a default one.

Isssue 73: What means "Improved error messages", exactly? I'm not THAT experienced with python, but i think that i could contribute it; Backup the files too...

[anishathalye]

Currently, if there are errors while stuff is running, the error messages aren't super useful. #73 is meant to fix that.

[Darksider3]

Hey. Sorry for the long time I didn't answer again. What I meant is: What is, in your opinion, usefull? I.e.: "Link X can't be set up because of wrong rights in Directory Y, can't write"? An example would be nice, as I can start with writing :-)

[bbenne10]

One feature that dotbot 1.0 lacks is ootb support for encrypting secrets. For instance, I'd love to be able to checkin my zshenv file, but it contains a couple of environment variables that define my identity on pushover. A plugin to use gpg to deal with these secets might be nice.

Then again - stuff like blackbox exists. So maybe dotbot doesn't need to reinvent the wheel?

[waynr]

Has anyone considered creating a pypi package to ship the dotbot CLI?

[casey]

What use case were you thinking of? Since most people use dotbot to install config files in repositories, and it's easy to get dotbot via submodules (or the mercurial equivelant ) I'm not sure what usecase dotbot via pypi would enable.

On Saturday, July 2, 2016, wayne notifications@github.com wrote:

Has anyone considered creating a pypi package to ship the dotbot CLI?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/anishathalye/dotbot/issues/35#issuecomment-230089047, or mute the thread https://github.com/notifications/unsubscribe/AAAHmfpW26AFczJS5e3XWhoArk980IDdks5qRhSygaJpZM4ENu__ .

[waynr]

I'll go ahead and admit upfront I am bias against the use of submodules, mostly because of the layer of complexity they add to managing a git repo; I really just don't think they are a good alternative to any packaging system's dependency resolution system.

Aside from that, there is at least one workflow that may be somewhat difficult with the existing install and management approach, and that is managing dot files on multiple machines in a push-oriented fashion. With my current dotfiles repo I manage to do this with a complex combination of git CLI and ssh in scripts stored directly in the repo and I imagine something similar would be necessary for dotbot, especially given the tendency I have noticed in discussions here for folks not to consider things like git CLI, ssh, and pyyaml as dependencies and to prefer such purity when managing their dotfiles--my current difficult-to-manage bash-based approach was originally driven by a similar ethos.

Then there's the #75, which is requesting that dotbot be made available as a python library. If this happens the best way to distribute said library (in my opinion) would be as a pypi package, in which case why not also ship a CLI tool?

[anishathalye]

The nice thing with submodules is that the whole thing can be self contained. Installing dotfiles becomes a git clone ... && cd dotfiles && ./install, even on a fresh system without requiring installing any other software.

Though I have nothing against making Dotbot available as a PyPI package, as long as it doesn't affect the current preferred method of use.

[adarnon]

It would also be nice to add support for defining packages to install (through apt/yum/etc.) Right now I'm maintaining a separate script that installs vim/tmux/etc. and it makes sense to define these in dotbot

[anishathalye]

@adararnon there are a couple plugins that might help with that.

[nshtg]

Tbh in my opinion setting up your server/pc (installing vim etc) should not be handled in your dotfiles anyways. Use something like ansible (or a simple bash script) for that.

[Anachron]

Hey guys I would love to have encrypted files that automatically encrypt/decrypt using my GPG key. Should be fairly easy to add. (decrypt all files having .enc.gpg as trailing string into the same location and put those newly created files into .gitignore)

[brennie]

@Anachron You can accomplish that yourself using .gitattributes and filters:

https://gist.github.com/g-k/9087422 https://git-scm.com/book/en/v2/Customizing-Git-Git-Attributes

[ashkitten]

What do you think about adding file permissions? I'm thinking of using dotbot to manage my etcfiles, but not sure if that's feasible at the moment.

[anishathalye]

If there's demand for permissions, I'd be open to adding it (gotta balance simplicity with having more features)

[flyingchipmunk]

@adararnon I just whipped up a yum plugin and listed it in the wiki here

[Anachron]

@flyingchipmunk thanks but I was not looking for a yum plugin? :) I was looking for a gpg one.

[flyingchipmunk]

sorry tagged you by accident :)

[angeliski]

Hey @anishathalye Any provision about that version? I would like help, your dotbot is great!

[anishathalye]

@angeliski I don't think there's any rush on the backwards-incompatible features. Everything else is being integrated into Dotbot 1.0 anyways.

[angeliski]

@anishathalye Amazing! I read about #57 , but now I see the #11 and can resolve my problem. Anyway, thanks for the great job. I will see I can help a any opened issue

[anishathalye]

@angeliski cool! always happy to integrate new functionality