search

arcticicestudio / snowsaw

A lightweight, plugin-driven and dynamic dotfiles bootstrapper.
MIT License
97 stars 7 forks source link

The way to Go: Project Future Rewrite #33

Open arcticicestudio opened 5 years ago

arcticicestudio commented 5 years ago

The title should hint and summarize what this document is all about: The future of the snowsaw project formed by a complete rewrite in the awesome Go language.

Even though the project is still in a very early development state with only two release versions, this rewrite is a large step forward a way more stable project foundation and better designed code base.

:construction: This is a living document which means it is work in progress, not completed yet and will be extended!

All implementation details and requirements are documented and tracked in the corresponding issues:

To test the current development state or keep track of the completed tickets check out the epic/gh-33-the-way-to-go branch. See the linked ticket above and the development workflow section below for more details.

Please report every bug to help making the project more stable. Every feedback is always welcome! :muscle:

A Small Excerpt From The Project History

The origin of the project is a port of the great Dotbot. I've searched for a tool to manage my .dotfiles and found long-time and stable projects like GNU Stow, Ansible or homesick as well as many more through great resources like GitHub's official .dotfiles website and awesome lists like awesome-dotfiles, but unfortunately none of them could fulfil all my requirements:

  1. KISS, DRY and the UNIX philosophy — Many developers can not resist the temptation to create monolith tools that are overloaded with solutions for multiple use cases and different targets. In my opinion development should always focus on the UNIX philosophy (“Do One Thing and Do It Well“) as well as the KISS (“keep it simple stupid“) and DRY (“Don't repeat yourself“) principles to create tools that are easy to develop, maintain, use, scale and provide a high reliability. In the next list points these principles will also match other parts of my requirements.
  2. Don't wrap Git commands — Many existing .dotfile manager try to provide more features to edit, update and persist the tracked files than necessary. They add CLI interfaces with commands like add, update or commit that are nothing else than wrapper around the Git add/commit core commands. Such features only add unnecessary complexity to the tool, reducing the transparency of what is really happening “under the hood“ and destroying the purpose of the UNIX philosophy (“Do One Thing and Do It Well“) as well as the KISS (“keep it simple stupid“) and DRY (“Don't repeat yourself“) principles. The only reason for such features might be that users don't need to know some simple Git basics (or Git at all), but if you're creating and tracking .dofiles the chance that you're not familiar with Git is close to zero. If you're modifying your .dotfiles in any way, Git provides you with all necessary tools and even if you're new to Git there are fantastic resources like Atlassian's Git guides and documentations that'll teach you the basics within several hours.
  3. No automatic Git actions — There are also many tools that automatically react to changes in existing files or new ones like adding, committing and pushing them to the connected repository. Like describes in the Nr.1 above, such features just blur the power of Git and reducing the transparency of what really gets executed: multiple sub-processes calling the actual Git commands.
  4. Easy integration and high portability — .dotfiles are the toolbox of every developer and often one of the main buttress for productivity, at least that's what I've experienced many times for me as well as for others. Everyone can work and develop thousand times faster and more fluidly when the apps and tools you use are configured to work without problems and fit your needs, your shell and terminal are set up with all the goodies like aliases and your favorite CLI apps are right at your hand. .dotfiles are one instrument to achieve this no matter if you're freshly setting up your new machine, working remotely via SSH or on someone else: Clone/Copy the .dotfile repository from your server or GitHub, download your .dotfile manager and let it bootstrap your files should be the only steps to get you up and running. That's a small example why portability of a .dotfile manager is important: it shouldn't require external dependencies, runtimes, interpreters, libraries or anything else. Just download the (binary) executable and you're done for 1/3 of these small steps. To learn more about the fact that snowsaw is written in Python, that in contrast to the listing indeed requires a interpreter/runtime (Python 3), please read the section below about the previous design decisions.
  5. Modular .dotfile structuring — Most tools require all dotfiles to be placed into one directory of the root of the Git repository which does not allow the user to structure the files and folders to match the personal preferences. The files must match the exact names and directory structures like their target symlinks or copy paths that doesn't allow customization or the sorting of files into some kind of category folders to aggregate them based on the target application, system or whatever ways users like to arrange them.
  6. Dynamics and configurability can go hand-in-hand with simplicity — The UNIX philosophy as well as the KISS principle both match the argument that the resulting app should “do one thing and do it well“, but this doesn't limit it's functionality to predefined tasks. The app should always provide sane defaults, but should also allows users to configure them if they doesn't match their personal preferences. This targets various aspects like the definition of conditions to only process files when they match, e.g. only for specific hostname(s) or when running on specific OS type(s). This requirement is related to the previous point (Nr.5), where users can use such configurations to achieve custom file and directory repository structures as well as making it easy to process only specific files based on conditions.

Based on these requirements I tested a lot of the existing tools and the ones that matched the most were Ansible and Dotbot, but unfortunately both also couldn't fulfil the requirements of being portable. Ansible can convince with large ecosystem, a granular configurability and the usage and extensibility with modules, but also comes with a lot of overhead for small projects like a .dotfile repository. It is mainly targeted for the commercial administration of large, distributed systems and the setup is way too over engineered for such a use case. Dotbot also provides flexible configuration features and can also convince through it's modularized design by using dedicated plugins for tasks like linking, copying or execution of commands through a shell process, but there were also features missing that were a must-have for me. The day after the evaluation was the birth of snowsaw.

Previous Design Decisions

Even though Dotbot is written in Python, that pulls in the dependency to the Python 3 interpreter and runtime, I've decided to base snowsaw on it. The decision was quite easy because to the time I've evaluated existing tools there was no stable and reliable project that was written in a portable language like Go, C/C++, Rust or anything else that (statically) compiles into a single (binary) artifact and also fulfils most of my requirements listed above. In comparison to other similar projects like Dotbot that are written in Python, there is only one external Python library dependency next to the Python 2/3 runtime itself. It is not essential and only adds support to optionally write configuration files in YAML instead of only using JSON, but this is not a must-have requirement for snowsaw.

The facts described above lead to the decision to port Dotbot and implement the missing features. Even though I'm a long-time Linux user and have some experience in Python (wrote some scripts where a shell script might be too complicated), I don't like script languages at all and always prefer type-safe compile-time languages like Go, Rust or Java. The only exception is JavaScript when used for websites or Electron/Web apps with React which is in my opinion the best way to build a UI since web technologies like CSS were invented for it. Next to this, Python also comes with the Python 2 to Python 3 ecosystem split-up and a likely broken package management, global vs. local package installations with pip (that also has a slightly complicated installation process itself) that can cause problems with native OS package managers (apt, yum, pacman etc.) because pip bypasses their tracking logic.

However, since Dotbot provides most of my desired features I decided to stay with Python.

snowsaw Goes Its Way

Like described in the project history above, the only reason to use Python was because of its Dotbot origins. During the development of more (requested) features and the fixing of bugs I often faced some problems everybody faces when writing in a language in which one is not so experienced. I always see such problems as opportunities to learn more about something new and gain experience, but after a while I unfortunately lost the interest iun Python for many reasons also described in the previous sections above.

In the meantime I expanded my knowledge in Go and until today I get more and more into love with this awesome language with each line of code. Some days ago I decided to take a few days off from porting all of Nord's port project to the shiny new website and wandering through my currently over 620 (!!!) notifications about open issues and PRs which are scattered in all my projects and other contributed repositories. After landing at snowsaw and trying to wrap my head around some of the pending tasks and how to solve them (with Python skills that are already dusted again :smile:), I had the lightning thought (and wish) that it would be awesome if snowsaw would be written in my favorite language: Go. And that's the reason I'm currently writing this wall of text :smile:

What To Expect

Before rewriting and reviving the project from its kind of „hyper sleep“ I want to make the process clear to all snowsaw users. Even though this started as a project for my personal use, it got some more attention and quite and larger user base. This means simply implementing everything and pushing it to the develop and master branches with a new version will break many users expectations and maybe their .dotfile setup too.

In order to carry out the project rewrite I want to clarify some general aspects and details:

Bye Bye Loose Plugin Architecture

One of the larger features of snowsaw was the plugin architecture that allows extend snowsaw's functionality by dropping a Python script into the plugins directory in order to let snowsaw handle other tasks defined in any snowblock configuration file. By default snowsaw came with the three core plugins clean, link and shell to provide basic and most of the time completely sufficient tasks to handle almost everything needed to manage .dotfiles. As far as I can tell (information only based on public repositories on GitHub!) most users of snowsaw never used custom plugins since the bundled ones served all necessary functions. This is a more or less relevant information since this means the omission of this feature for the new Go implementation will have almost no impact on the usability. Adding a new plugin to handle other tasks was possible by satisfying the snowsaw.Plugin interface that requires the plugin to implement the can_handle() and handle methods. This more or less unstable pattern is the reason why this section's headline uses the „loose“ plugin architecture wording since Python is not designed for type safety as well as concepts like strict interface implementations. snowsaw was instructed to assume that the plugin author has read the documentations regarding the required behavior and return values of these functions.

Luckily Go is a type safe language and it's language design makes heavy use of interfaces that require correct implementations, but due to it's nature of being a compilation language it is not that easy to introduce a plugin system. I've spend a lot of time to think about a way to keep the previous plugin-driven architecture up for the rewrite and evaluated the following possible solutions:

Go Standard Library plugin Package

Go comes with the plugin package by default that allows to load and resolve symbols of other Go artifacts, a so called „Go plugin“. It allows to load the files from anywhere on the same filesystem and make use of any exported type or function. It was first introduced in Go 1.8 and at the time sounded like the perfect solution to build modular and dynamic applications with endless expandability. Anyway, one downside was the restriction to be only compatible with Linux. Later on, Go 1.11 added support for macOS and support for Windows is on it's way. A Go plugin can be easily compiled by simply using go build with the specific -buildmode=plugin flag in order to compile the target packages to a .so file. There are also more supported build modes, e.g. to create a shared library that can be imported into any other language like C or Python (buildmode=shared or buildmode=c-shared) or also to create position independent executables (PIE) through the -buildmode=pie flag. Anyway, I don't want to go into details here, but if you want to take a deep dive into this topic please take a look at the official plugin package documentations, go help buildmode and go help build as well as many other references and tutorials out there.

As beautiful as that sounds, there are also several difficulties when using Go plugins making it too hard to maintain and develop for such a small project like snowsaw. This is not the marching solution to let users add in their own code, they need to adhere to many rules, configurations/setups and conventions when building a custom Go plugin due to the following points:

Hopefully all these bullet points will be obsolete later on when the plugin packages gets improved with future Go versions, but in the meantime this is not the desired solution.

There are other plugin systems designs out there, e.g. by using Go's net/rpc package that allows the main application to communicate with plugins through remote procedure calls. A more advanced solution is the awesome go-plugin project by Hashicorp that brings all these functionalities out-of-the-box with a easy-to-use API, many additional feature and also full support to use the awesome gRPC project instead of the more basic (and limited) net/rpc package. They're using their own package in famous and busniess-critical projects like Terraform and Vault and I've also used it in some other private/public/dayjob projects. It's performance can not be compared to native Go plugins, but even in production with really heavy throughput there is no noticeable problem or bottleneck. Anyway, even though a gRPC based solution for plugins for snowsaw would work really well, it is too over engineered and only brings in unnecessary complexity for such a small project that aims to lightweight and tries to follow the KISS princicle and Unix philosophy.

These are some facts which must be considered when snowsaw would use Go plugins and these are also all reasons why snowsaw won't adapt to this concept. For more details, please read the official Go plugin package documentations, join the official Gophers Slack workspace and take a look at posts like this in the official /r/golang subreddit.

Long story short: The initial Go implementation of snowsaw won't use a plugin architecture anymore, but will come with necessary functionalities out-of-the-box to handle almost every use case for dotfile management. There will be a kind of „task“ API with interfaces that'll be implemented by snowsaw's core features and it will be exposed as exported types, allowing users to implement custom task handlers to extend snowsaw's capabilities. Later on a detailed documentation will be added plus resources to simplify the process of compiling the project together with custom task handlers, e.g. a Dockerfile that can be used to automatically place custom code in the correct package folder, build the project and copy the resulting artifact from the container to the host while leaving the host system in a clean state without the requirement to even clone and set up snowsaw's repository.

Next Steps

This document will serve as the epic issue and keeps track of all the sub-tickets that are listed at the top below the introduction paragraph. Before starting the actual implementation I will create the design concept tickets that'll be used to build the repository, documentation and code base from scratch. Note that this might take some time since it is not a high priority task and will be done step-by-step when there is some time left from the more urgent tasks like the Nord port project data transitions.

Development Workflow

Since this issue represents the main epic there will be a branch all results of the sub-tickets and stories will be merged into. As soon as everything is finally completed this branch will be merged into the main develop branch and later on into master to create a new version tag and deploy it. This way the rewrite can live together in parallel with the current code base without leaving it in an unusable state.

Build With & For The Community

Even though snowsaw was mainly developed for my personal use cases it is a open source project that means everyone can contribute to push the project forward and help to form its future.

If you like to test the new rewrite or keep track of the actual development state you can check out the epic/gh-33-the-way-to-go branch and follow the design concept documents and linked implementation ticket listed above.

Please report every bug to help making the project more stable. Every feedback is always welcome! :muscle:

wren commented 3 years ago

Hi! This project looks really interesting. Did it ever get off the ground?

arcticicestudio commented 3 years ago

@wren I'm still dogfooding the existing implementation (epic/gh-33-the-way-to-go branch) to see if it fits and works fine with an existing setup. So far I am satisfied and I think it's going in the right direction. The current blocker is just that I'm short on time to work on my all of my OSS and private projects. The Nord project continues to occupy a large part of my time, but some of the upcoming changes will (hopefully) reduce some maintenance overhead so that I can spend more time on other projects again. Next up will be some clean ups of the current implementation and improvements I planned while building other Go projects, e.g. setting up a build system with my wand project, migrating the repository to the GitHub flow and the master branch to main and adapting my new repository organization for Go projects. Actual features and improvements like the YAML support and a way to use “wildcard bootstrapping“ are then on the list.

In summary, that means that the plans documented in this issue are still up-to-date and, as soon as time permits, more improvements will follow. I'll keep the entry post up-to-date and post comments here so subscribers get notified 😄

I'm also always interested in feedback from others so feel free to clone the epic/gh-33-the-way-to-go branch and run go build -o snowsaw . from within the repository root to build the binary. There is currently no managed installation process so the binary must be manually placed somewhere in your PATH, but support for package managers like AUR, Homebrew and Scoop will also follow later on.