How to ensure the daemon starts, stays running, and watches the right Git repos

tkellogg commented 2 years ago

dura can't do anything for you if it's not running. How can we ensure that it's started & always running? I'd like dura to be nearly invisible after installation. It should "just work".

dura serve is setup so that the old process will finish it's loop iteration and then terminate. The new process overwrites ~/.config/dura/config.json with it's PID, and the old process terminates at the beginning of it's next iteration. (dura kill overwrites the pid with null, which causes the existing daemon to terminate itself).

Idea: start from `~/.bashrc` or `~/.zshrc`

When a new terminal session opens, it runs some file like ~/.bashrc. Here, we can run dura serve & to start & demonize it.

To make this work, add this to your ~/.bashrc or ~/.zshrc file:

dura serve &

Positives:

It ensures that the demon starts
Somewhat easy to setup

Negatives

It only works if the user uses the terminal. Some people only use Git plugins via their editor.
Doesn't restart failed daemons

Idea: bash/zsh prompt function

Set the user's prompt function to run dura serve &. This means it runs after every command

Positives

Better assurance that the daemon keeps running at the right time
We can also have the function do dura watch . on every directory we cd to, to ensure all Git repos are watched.

Negatives

Harder to install (could be mitigated)
Harder to support more shells
Only works if the user uses the terminal. Some people only use Git plugins via their editor.

Idea: Git hook

Run dura serve & and dura watch . on a global Git hook (e.g. after pull & commit)

Positives

You generally run git pull before doing any work, regardless if you use the Terminal or an IDE plugin
Would target precisely the repos that you actually use.

Negatives

Hook scripts are exclusive. If you wanted to run other hook scripts, you'd have to shut off the dura hooks.
Doesn't include repos that you've already cloned & pulled. So there's a gap between when you install dura and when it actually does it's job correctly.

bj0 commented 2 years ago

why don't you just install a systemd service?

ChampokDas commented 2 years ago

I haven't taken a look at any of the internals of the project at all.

I think making dura a service to be handled by a manager would be best to achieve "always running, invisible" state with minimal input from the user, can depend on it being alive or if it's unavailable, just restart it. Then you have a standard interface for plugins in a editor/IDE + terminal users.

Since I'm mainly a terminal user, I'd be completely satisfied with just having server/client model similar to how things are done with tmux, initial startup of the dura server and then further calls will be clients to it.

tkellogg commented 2 years ago

Systemd is a good idea. I neglected to mention it because systemd isn't cross platform; I'm looking to support this project with low effort. But I'm not opposed to the idea either.

nalzok commented 2 years ago

Does this work on my OS?

Mac: yes

Linux: probably

Windows: possibly

If macOS have first-class support, then you should definitely look into launchd(8). You can use RunAtLoad to ensure the daemon starts, KeepAlive to ensure it stays running, and WatchPaths to watch the right Git repos. See launchd.plist(5) for more details.

tekacs commented 2 years ago

For the serving aspect, if targeting Mac, Homebrew can take care of producing and managing a plist (and auto start) for launchd:

https://docs.brew.sh/Formula-Cookbook#launchd-plist-files

or

https://rubydoc.brew.sh/Formula#service-class_method

Homebrew makes it almost trivial (just one line), if that’s a viable method for the project.

rjc commented 2 years ago

Hello,

It would be nice to see dura being a true daemon like say, ssh-agent, etc. and not simply runs in the background via Unix shell's &. Also, once started with &, the shell needs to remain open - otherwise, if dura responds to SIGHUP the process will get killed.

As to which repositories to monitor - this can be either as simple as find . -type d -name .git -prune or a static list, like myrepos uses.

When it comes to a service, I'd simply leave it up to an individual. Like I've mentioned earlier, something like ssh-agent - or any *-agent for that matter - does its job just fine when started from a user's login shell - no service necessary.

goodevilgenius commented 2 years ago

I think this really should depend on the system. Most operating systems have some daemon management service.

It wouldn't be too difficult to include a systemd service file and a launchd plist file, and then accept contributions for other systems.

mihai-stancu commented 2 years ago

@rjc the SIGHUP can be mitigated by using nohup dura serve &

mihai-stancu commented 2 years ago

Autoscan proposal:

Declare a configuration flag intended to activate autoscan. Finding all directories named .git. Finding the .git/config file & parsing it. If the config contains a [dura] section with an enabled = true flag, then start watching it.

flypenguin commented 2 years ago

i understood that one dura process watches all of the configured git folders, correct?

service - systemd on linux is a good idea and most know how to use it. i am using a mac for years now and actually never had anything to do with launchd daemons ... . i guess the "perfect" mac way would be an addition to the system settings, like for example macfuse, gpgsuite or witch do. (yeah, GUI, shame on me, i know ;). since this is probably a LOT of effort, the launchd thing might just be the way to go. (actually, if that system settings GUI would be a thing, i'd pay a couple of bucks in the app store for it - open source MAY earn money).

adding - i really like dura add $DIR. perfect would be if $DIR is a git repo, add this. if $DIR contains git repos (maybe just depth 1), add those.

low key :)

rjc commented 2 years ago

@mihai-stancu

@rjc the SIGHUP can be mitigated by using nohup dura serve &.

Sure, my point was to making it a proper daemon and not resort to Unix shell tricks to keep it running in the background :^)

As to finding all Git repositories below a certain directory, I don't necessarilly think this is a good idea having had a think about it. What if the repositories in question aren't under $HOME? What if one's using a large group NFS share? Again, a list similar to .mrconfig should suffice.

mihai-stancu commented 2 years ago

@rjc agreed, reinventing the concept of a system service by using various neckbeardery defeats the purpose of low maintenance.

Sure, the "autoscan" proposal can hit snags with large directories and slow filesystems, but it can be complementary to the list of directories in order to allow near zero configuration.

rjc commented 2 years ago

Just to disambiguate as there are two concepts being mixed up - daemon and a system service - they're not the same thing. dura should most definitely be/have the former - I think it should not be a system service as that would imply it starts monitoring Git repositories for everyone omn the system. Yes, I'm aware that (at least) systemd and launchd have a concept of user services but, apart from Linux and macOS, most systems do not. I've been banging about ssh-agent as this is the way I see a user daemon - it starts as soon as the user logs on (why would you need it to start earlier?), then either it just runs, or one can communicate with it via a socket, etc.

pkoch commented 2 years ago

Let's maybe add a note to the README, or the top post of this issue, for a command that scans people's homedir?

Something like find ~ -type d -name .git -prune | xargs -I= bash -c "cd =/..; dura watch"

rjc commented 2 years ago

For those who had missed the link I posted earlier, please have a look at the related software section on myrepos page - if not direct solutions, I'm sure you'll be able to find some interesting ideas there.

Also, I see a lot of bash in the text/examples - any chance we could agree on sh, as in POSIX shell?

tkellogg commented 2 years ago

@pkoch I created #25 for issues around the find | xargs approach to discovery. I like the approach, but it needs some work beyond the README.

tkellogg commented 2 years ago

@mihai-stancu I'd like dura to default to watching as much as possible. It seems like there's a lot more potential frustration if you find out after-the-fact that dura wasn't watching a repo. There's enough holes in this proposal that I'll settle for @pkoch's plan, a la #25

@rjc Correct, this is certainly a per-user daemon, not a system service. As currently implemented, it uses ~/.config/dura/config.json as a locking mechanism to ensure a single process is running (see #2). We could change that to be a viable system-level service, but I initially avoided it to cut down on corner cases around file permissions. Let's keep options open for now — focus on better installation instructions in the README and/or installers.

jhuntwork commented 2 years ago

I agree with the comments about avoiding shell tricks to launch a service in the background and if you really want a daemon, to add in proper support.

However, on most systems if you want the service launched automatically and managed it's actually a lot less fragile to supervise a child process running in the foreground. Correctly keeping track of the running state of another, non-child, process is problematic. For best results, this should really be in the realm of supervisors, i.e., projects like supervisord or s6, or even systemd.

I would suggest that you continue to design dura primarily to launch in the foreground and log to stdout/stderr and possibly provide sample configurations to run under various supervision suites according to platform. This would make it the most flexible, even allowing nice functioning in a container, if there is a good use case for that, and leaves the issue of managing a stable long-running process to tools that already excel at that. It even lets you hook in standard log collection utilities that those supervisors may support.

On a side note, I also agree with @rjc's request to use sh over bash in your code/examples. sh is a standard guaranteed to be present, bash is not. And while bash does exist in so many places that it feels universal, it really isn't. There are many systems I manage where it is absent.

tkellogg commented 2 years ago

@jhuntwork I created #30 to print to stdout (unfortunately I worked away from that pattern last night).

What would be a good resolution to this discussion? Wiki pages describing how to setup systemd, etc? Installation scripts?

nevinpuri commented 2 years ago

Hi, I propose an idea of developing various IDE extensions which automatically run dura in the background. This would mean that dura would be watching files as soon as the user opens their IDE, and the user could have a better UX while using dura (imagine clicking and seeing the changes, vs going through the dura branches via the command line).

I can create a really basic vscode extension to try and better represent what I mean.

Let me know what you guys think. I'd imagine going this route would mean that the dura project would have to be turned into a GitHub organization, in order to better keep track of all the repositories.

This idea would also make dura easier to run on various systems, as you would not have to worry about system specific things, such as systemd. Instead, you could just let the IDE extension launch the executable.

jhuntwork commented 2 years ago

@jhuntwork I created #30 to print to stdout (unfortunately I worked away from that pattern last night).

What would be a good resolution to this discussion? Wiki pages describing how to setup systemd, etc? Installation scripts?

I think that depends on what you intend to provide and support. For example, with Linux do you expect you will create official packages for Ubuntu? If so then you'll probably want to include with the package actual service definitions for systemd. And then the question becomes, how far down that path do you want to go? Do you want to support other Linux distros, perhaps ones that don't use systemd? What about for Mac, or Windows?

For me, if the behavior of a tool is clear through its help messages (and other documents) then that is good enough to implement a service on my system, and probably for others as well. A simple example in the documentation about how to run it on a platform-independent supervisor would probably be a good enough start.

It seems like @Nevin1901 already has a good idea about managing this through a vscode extension - and while that doesn't cover every user of git, it's probably a great setup for a lot of people. Kind of proves the point too about having the core dura code be simple and run in the foreground and let something else manage it how it will.

tkellogg commented 2 years ago

@Nevin1901 Yeah, let's see a prototype. Make your own repository and if it seems good I'll make a dura organization and add it to it.

fritzr commented 2 years ago

+1 for @rjc 's human-read+writable config file suggestion over a json config. This would make it super easy for users to maintain an explicit list of git repos to watch, and if an "autoscan" feature is implemented it would make it easy for users to control exactly what is scanned or whether to scan at all. Like @rjc says if $HOME is NFS-mounted, or especially if cross-fs links are followed, an automatic blanket find with no depth limit in the user's home directory every five seconds could be catastrophic. It may make sense as a default on install for some users but it would be silly not to have an easy way for users to disable/control it.

Regarding reliable startup, I don't think it's unreasonable to leave that as an exercise for the user with suitable suggestions for popular OSes, sort of like how you have it setup already. Like tmux, it wouldn't hurt if every command that isn't start implicitly runs start first. Also like tmux, and also as @rjc suggested, I vote for properly daemonizing the server process rather than relying on shell jobs.

abrooks commented 2 years ago

On Linux, inotify (a kernel feature with libraries in most languages, here's one for rust) will avoid polling and will efficiently generate events even for large directory trees.

rjc commented 2 years ago

@abrooks This has already been mentioned, with notify being better (cross-platform) library.

JakeStanger commented 2 years ago

Autoscan proposal:

Declare a configuration flag intended to activate autoscan. Finding all directories named .git. Finding the .git/config file & parsing it. If the config contains a [dura] section with an enabled = true flag, then start watching it.

Autoscan + systemd service feels like the best way to me personally, although not necessarily as above. Perhaps a ~/.config/dura/config.toml or whatever format, where you declare directories to recursively search:

directories = ["/home/jake/Programming", "/home/jake/aur"] # lots of git repos live in these places
max_depth = 3 # avoid scanning every directory on the system like node_modules or whatever

Perhaps directories could be opt-out either by having an ignored_directories key in the config, or with a .duraignore file?

nevinpuri commented 2 years ago

I have created a very simple vscode extension which prompts the user to both autostart dura, and also asks the user if they want to watch whatever workspace they're in. You can see the repository here. This is just a proof of concept (want to add many features in the future), so I'm curious as to what you guys think of it.

The extension is not yet on the vscode marketplace, but I can see what to do with it if everyone likes the idea.

As of the license, I can also change that. I wasn't sure which license to use, so I just used Apache 2.0 (the same one as in this repository).

Let me know what you guys think of the extension, and if we should keep it.

tkellogg commented 2 years ago

@JakeStanger what do you think about include and exclude fields within directories?

@Nevin1901 thats great! I'd love to get feedback from other VS Code users

JakeStanger commented 2 years ago

what do you think about include and exclude fields within directories?

Yeah that could work, ideally I think the short-hand string version above should default to "include all", and then perhaps in situations where you need more control you could define paths (or globs?) to include/exclude.

Something like:

[[directories]]
path = "/home/jake/Programming"
exclude = ["Work"] # relative to path

[[directories]]
path = "/home/jake/aur"
include = ["my-aur-package", "modified-aur-package"]

Not sure if TOML is the best language of choice (not a huge fan myself) but it feels the most rust-y

tkellogg commented 2 years ago

@JakeStanger That works for me. Are you going to make an attempt at it?

JakeStanger commented 2 years ago

I'll try and have a crack tomorrow, will report back with progress

richard-hajek commented 2 years ago

My current setup is invisible and works as following:

dura.service started on user login ( can be auto installed )

[Unit]
Description=

[Service]
ExecStart=/usr/bin/dura serve
Restart=always
RestartSec=3

[Install]
WantedBy=default.target

And cd hook in my .bashrc, Can not be auto installed :(

function cd () {

    if [[ -d "$1" ]] && [[ -d "$1/.git" ]]; then
        (
            builtin cd "$1"
            dura watch > /dev/null &
        )
    fi

    builtin cd "$@" && ls --color=auto --group-directories-first
}

Captures 99% use cases

richard-hajek commented 2 years ago

I just got the best idea - Create an hourly systemd timer, and use it to search for new repos, using the locate command

benriemer commented 2 years ago

Hi, cool project. Do you know the watchman project from facebook?

You can create a unix domain socket for communication with the main service and every client who is monitorin a new filepath.

Maybe it helps https://facebook.github.io/watchman/docs/socket-interface.html

tkellogg commented 2 years ago

Thanks! I have heard of watchman, but never looked into it. On my mac, I setup launchd and that works very well, although it's not cross platform.

tubbo commented 2 years ago

If I were setting this up and I wanted to run dura watch . every time I cd-ed into a Git repo, I'd probably use the chpwd() hook:

# zsh
chpwd() {
  if [[ -d $PWD/.git ]]; then dura watch; fi
}

# bash
cd() {
  builtin cd "$@" || return
  if [[ -d $PWD/.git ]]; then dura watch; fi
}

So perhaps a subcommand like dura extend-shell could be run in the .zshrc or something, which would output the path to the script that would do one of the above...

# .zshrc

source "$(dura extend-shell [zsh|bash])"

There's probably a better interface for this, and you might even be able to do it as part of Homebrew's shell completion step when it installs autocompletion for whatever shell you're on.

tkellogg commented 2 years ago

I love this suggestion! Having a subcommand for it is a good idea too

kohane27 commented 1 year ago

To piggyback on other suggestions, I have the following zsh function:

v() {
  if [[ -d "$1/.git" ]]; then
    (
      dura watch >/dev/null &
    )
  fi
  nvim
}

This function checks if the given directory is a git repo and if so, it runs dura watch in the background before opening Neovim. I only want to dura watch a repo that I will edit in nvim.

tkellogg / dura