search

stevegrunwell / asimov

Automatically exclude development dependencies from Apple Time Machine backups
MIT License
1.62k stars 93 forks source link

The Asimov Roadmap #51

Open stevegrunwell opened 4 years ago

stevegrunwell commented 4 years ago

Asimov hasn't been getting the love it deserves, and I'm happy to welcome @sudar as a co-maintainer (see #48) to help things along.

I have a lot of ideas for where I'd like to see Asimov go, but want to open them up to the community for thoughts + discussion before implementing.

Near-term

Get Asimov submitted to Homebrew (#2)

This has been a goal since the beginning of the project, but Homebrew's policies dictate that the repository needs to meet a certain threshold of stars (50) on GitHub in order to be considered. Fortunately, this was exceeded some time ago, so now it's a matter of getting the Homebrew recipe in-place. I had started (and then re-started) this effort, but #34 looks the most promising. I'm totally the blocker in reviewing and approving :/

Introduce automated testing (#31)

The logic as it stands right now is pretty straight-forward (match a config/dependencies file with its corresponding installation directory and, if found, exclude from Time Machine) but having something that could run in a CI environment would be extremely useful to make sure we're not breaking things as we go.

Eventually

Introduce a configuration standard (#27)

A number of issues have hinted at the need for a user-level Asimov configuration file, enabling individual users to configure not only the patterns that should be matched but also other directories (#28), whitelist specific directories (#14), and more.

Issue #27 proposed something like an .asimovrc file, which could be used to store Asimov configurations.

Expand beyond development dependencies

Asimov was designed for developers, but there have been some great suggestions around other things that can be excluded from Time Machine (for example, Spotify caches, suggested in #39).

If Asimov becomes something of the de facto tool for keeping Time Machine backups limited in scope, there are a number of other file types that we might want to include:

Possibly rewrite in something besides Bash?

Asimov is currently written in Bash, since it's essentially performing filesystem traversal, basic pattern matching, and then interfacing with the tmutil program; if we want to get into more sophisticated functionality (for example, asimov list to see everything excluded from Time Machine backups, it's probably best to consider rewriting in a different language.

My default direction would be as a Symfony Console application, but that's mostly because PHP is my primary language (and PHP comes pre-installed on MacOS). An abstraction around tmutil will make the tool easier to work with, and Symfony components are designed to be well-tested. Distributing it as a PHAR means we still only have one file being installed.

The drawback here is that MacOS typically ships with older versions of PHP, which means the package would either need to be written using older PHP conventions, PHP would need to be marked as a dependency within Homebrew, and/or support would need to be restricted to versions of MacOS that ship more recent (read: PHP 7.2+) releases.

Make Asimov the go-to tool for interfacing with tmutil

The default tmutil tool shipped with MacOS isn't super-well documented, and is much more a developer tool than something intended for typical users. While I don't believe a GUI is necessary (Time Machine already provides that), providing a user-friendly Time Machine interface would be helpful.

Potential commands could include:

add - add files/directories to the exclusion list config - read/write entries from the user's .asimovrc file ls/list - Show all files/directories that have been excluded from Time Machine prune - Locate any directories/files that have been added to the exclusion list but no longer exist (which would address #38) rm/remove - Remove files/directories from the exclusion list scan - Scan the filesystem for any new files/directories to be excluded. This is the default behavior of Asimov today.

muuvmuuv commented 4 years ago

A rewrite would be great and I think PHP is good as long as we use PHP 7.4> (with typing support) but IMO a more low level language would make much more sense for this project. I would consider something like GO, Rust or Nim. All of them can be compiled to a single executable, shipping with brew is easier AND they come with great typing support and beautiful compiler errors. I don't know how familiar you or anyone else involved in this project is with those languages, but I had no difficulties learning Nim or Go (the basics) and I'm coming from JavaScript, PHP and Python.

muuvmuuv commented 4 years ago

Something else that come to my mind with prune. I would also – besides some location where the exclude-list/config is stored – add a place where Asimov caches the files/dirs it has added to TM. So when prune is running it would remove files/dirs from TM that are no longer in the Asimov config (based on the cache) and files/dirs that no longer exists. TM would be somehow in sync with the config.

In addition to that I would do a prune together with scan to prevent issues and make it more user-friendly. Somehow it is kind of the same, right?

I would also add my opinion to the config file again; After reading the above I think it would make sense to put the Asimov config in ~/.config/asimov (unix standard) and place everything there:

sudar commented 4 years ago

@stevegrunwell

Thanks for including me as the co-maintainer 🎉

Also I agree with the list of priority that you have mentioned. Let me know if you want me to get started with any of them, especially adding the homebrew support.

tomtastic commented 4 years ago

Excellent roadmap, but is PHP a wise choice of language given the old version on MacOS - isn't this what Apple do to built-in software (fail to keep updated) shortly before they deprecate it entirely?

Would Python or perhaps better Go be options that @stevegrunwell and @sudar would be comfortable with?

stevegrunwell commented 4 years ago

I would agree re: PHP maybe not being the best choice. PHP's definitely something @sudar and I are both comfortable with (we first met working as engineers at a PHP-focused company), but doing any sort of rewrite in a language that we can safely assume will continue to be pre-installed on MacOS will be ideal.

Of course, if we're targeting a Homebrew installation, it's easy enough to say "if we don't have a current version of PHP, install it as a dependency", so not shipping a current version by default on MacOS may not be a dealbreaker.

I'm also kind of interested in playing with Go, personally, but Python might be a good middle-ground. It should also be pretty straight-forward to pick up, if xkcd has anything to say about it:

xdcd comic #353, "Python"

@sudar, do you have a preference? I feel like PHP will be the fastest/easiest to implement and really focus on building out these features without worrying about the language barrier, but I'm open to learning a new language if it makes sense to do so. We could also rewrite in PHP now, get everything hammered out, then replatform a second time down the road if there are big performance wins to be had?

codexp commented 4 years ago

I vote for PHP +1

And I think it doesn't have to be cutting edge PHP. Even 5.4+ would be sufficient to do most of the things that Asimov can do today.

Much more important would be to know the distribution of any language that might come as an option. And whether it is backwards compatible.

PHP is backwards compatible, so you just have to decide how far back you'd want to support older systems.

sudar commented 4 years ago

@stevegrunwell

Being someone who have been caught in the let's rewrite everything before we release loop so many times in my life, especially with pet projects I feel we should do things in the following order.

stevegrunwell commented 4 years ago

I'm on board with this; let's get the current version into Homebrew before looking at rewrites.

solipsist commented 4 years ago

I love this little utility. It keeps TM backups running fast and smooth.

As for feature priorities, I believe that addressing hourly backups is a good idea as discussed here: #54

A common issue with tools like these is new changes in MacOS. I think the feature set is fine and all I wish is that it's maintained and kept up-to-date. I'd be happy with what there is if I can keep using Asimov for years to come with smooth upgrades as we switch 10.16, 10.17 and so on.

Regarding language and performance, Ruby or Perl would probably be good choices.

Many thanks for sharing your work!

partyka1 commented 4 years ago

Excluding paths based on assigned tags would be neat - one could exclude specific files/directories directly from Finder using context menu and get graphical overview what's been excluded

t0r0X commented 3 years ago

Update: Python, Ruby, and Perl packaged with macOS have been deprecated, see section "Scripting Language Runtimes" in https://developer.apple.com/documentation/macos-release-notes/macos-catalina-10_15-release-notes Also PHP: https://alanstorm.com/programming-languages-leaving-in-mac-os-11-big-sur/

Not many options left: Zsh and Bash. Maybe Zsh is the way to go, or rely on dependencies from e.g. Homebrew.

stevegrunwell commented 3 years ago

Well, that's fun. Since Asimov is very much a developer tool I'm not overly opposed to defining dependencies for the Homebrew installations, but would definitely want to keep them loose (for example, if it's PHP, support several versions back; you don't have to install PHP 8 just to run Asimov, but as long as you have something in the 7.x branch you're okay).

Keeping it as a shell script is still an option, of course, but unless there's a ZshUnit that I'm unaware of the test suite will still need to be written in another language.

t0r0X commented 3 years ago

😁 Like e.g. these?

S.c.n.r. I'm fine and happy with any working decision, and am really glad I found Asimov, because it solved my issue and your work made me research deeper into these parts of macOS (I'm still a curios developer, after all these years).

muuvmuuv commented 3 years ago

What about using Swift because its macOS only ^^. I bet integrations to TimeMachine etc. would also be better.

tg44 commented 3 years ago

I started a repo; https://github.com/tg44/heptapod far from feature-ready, but I can do most of the things from code as asimov could right now, possibly faster with multithreading, and already handles configs with search and exclude paths. If sb interested :)

(I'm totally noob at golang, this code was written in about 16h, and I had another 8ish ours in the lang, so any tips are also apprechiated!)

chmac commented 2 years ago

Would it be crazy to suggest keeping asimov as simple as possible?

I have written all kinds of open source software myself, and the maintenance is always hard work. Sometimes I have motivation and time, sometimes I don't. The simpler the tool, the less maintenance, at least that's my thinking. Although maybe @tg44's heptapod is the simpler alternative?

dboune commented 2 years ago

Just found Asimov. I think it's fine as a shell script. Go or rust or python are all fine too, but not necessary.

All I really want for is a bit of config.. To add a directory on a second local volume where I keep my dev stuff. To add my own additional definitions, etc. Otherwise, this seems to work nicely.

cpboyd commented 1 year ago

Just to echo @dboune, is there any support for secondary volumes?

cpboyd commented 1 year ago

For anyone finding this, you can clone the repo and modify ASIMOV_ROOT from ~ to any path. Ideally, add support for ASIMOV_ROOT as an array/list like ASIMOV_SKIP_PATHS

I'm not sure if there's a good way to allow user customization here for homebrew installs, but it'd probably be nice for both of those parameters.