Alter plugin discovery

[tt]

Now that my first pull request got pulled, I figured I'd try another experimental and debatable change.

I tried making hk behave more like Git based on how it is extended. Instead of defining HKPATH (and as a consequence altering the PATH variable temporarily), plugins can now live anywhere in the default search directories and are added if they match "hk-*".

This will provide more options for distributing plugins; making something that depend on a lot of brew formulae? Ship it as a formula! Testing a simple script? Put it in ~/bin. While we may want to standardize delivery of our stock plugins, I think this flexibility is preferable for all other use cases.

[kr]

I've been mulling over this idea for a while, but it'll take some strong evidence that it's really better than an HKPATH.

Plugins aren't shell commands, so they don't belong in the shell's search path. They are, however, hk commands. It would never be right for a user to run hk-whatever directly from the shell, so why even make it an option? (As well as cluttering up tab completion.)

As for plugin distribution, it seems all those options are available with either PATH or HKPATH. A homebrew formula can install a file into /usr/local/lib just as well as /usr/local/bin. A user can edit a file in place in any directory, not just ~/bin, and they are free to put ~/lib/hk (or, hell, ~/bin) in the hk search path if they want.

Having said that, we should at least consider changing the default HKPATH to something more inclusive and correct, like, say: /usr/lib/hk:/usr/local/lib/hk:$HOME/.hk/plugin

Also I agree it's gross to temporarily setenv PATH just for the purpose of calling LookPath. It'd be better to copy and edit the function.

[tt]

It's technically true that you can extend your HKPATH to include PATH, but I wouldn't want having to think about if hk sudo would invoke /usr/bin/sudo or a plugin.

(This is really two changes: Using the default search directories and requiring a prefix. If we continue to use special search directories, we don't need the prefix, whereas we definitely will if we use the default search directories.)

My primary argument for this is to not ignore operating system natives. As a user, I also don't want to maintain search directories on a per-tool basis. I know how PATH works. It may be that you shouldn't execute plugins directly, but plugins are plain executables that just get a special set of environment variable, so there's nothing wrong in doing it. After all, it's how hk will execute them too.

Technically, I feel that the need to either rewrite LookPath or reset PATH is an indicator that something is wrong. We're trying to reinvent something we can get for free.

[kr]

Just because the shell's search path mechanism (LookPath) is similar to what we want doesn't mean we should be lazy and abuse it for our own purposes. It's an hk plugin, so we shouldn't call it a shell command. Say what you mean.

Executing an hk plugin without those special environment vars is a recipe for confusion. Some of them might work, and most are likely to break. Even if it did work reliably all the time, having two nearly identical ways to do exactly the same thing (run hk sudo or hk-sudo) is about the easiest-to-fix design mistake: just remove one of those two ways.

http://en.wikipedia.org/wiki/Hick's_law

If git is the precedent you want to cite, on my machine git keeps about 160 executable files in /usr/libexec, none of which are in my shell's PATH.

As a user, you should never have to think about HKPATH. Even as a plugin developer, you don't have to mess with HKPATH unless you really want to develop your plugins in a weird place for some reason.

[tt]

I really do think of plugins as shell commands. The hk executable is just a convenient wrapper. That's also why I will prefer using it over invoking the plugins directly; I like convenience.

I completely agree that it's better to have a single way of doing something than two, but for me the downsides of having HKPATH is worse. Not only as someone who is interested in the codebase, but especially as a user who'd love to install plugins via brew, add my own simple extensions and not having to maintain yet another environment variable. (I hate GOPATH too; it completely messes up the way I usually organize my projects!)

Git "plugins" live in your PATH, but the provided Git porcelain doesn't.

[kr]

Hk plugins aren't shell commands. Using hk to run them isn't just a convenience. We wouldn't be having this discussion if hk plugins were lua scripts run with an embedded interpreter (which is a distinct possibility we've discussed on and off), and there would still be an HKPATH to find them. The fact that plugins happen to be implemented as unix executable files instead of lua scripts doesn't really change the nature of what they are. It's an implementation detail.

As I mentioned, as a user, you can still install plugins via brew, and you don't have to maintain another environment variable.

Even if the downsides of HKPATH are worse for you personally, the vast majority of users will never even have to think about it, and removing hk-sudo and friends from their search path makes life easier. It's worth it to make life slightly more difficult for developers if it makes users' lives easier. (And in this case I don't really think it even makes developers lives worse, but if you feel that way I respect that.)

[tt]

With the right environment, all plugins can be run without the host - even Lua scripts. (hk-sudo is a special case because you'd actually be able to invoke that directly without altering the environment and it's also a very special case in that it's irrelevant to non-Herokai.)

I definitely agree on your point that it can make sense to reimplement LookPath instead of forcing that to dictate the design, but I really think the reuse make for a simpler design too (also for users). A compromise could be to use the PATH first and then have a known plugin location where hk itself would install plugins into.

This is also a power user thing, but what if I distribute my plugin via brew? Would the formula then resolve HKPATH and try to put the link the same place? The reuse of PATH means it can just do business as usual.

What was your reasons for "mulling over this idea for a while"?

[kr]

With the right environment, all plugins can be run without the host - even Lua scripts.

There is a lot of complexity hidden in that seemingly-innocuous phrase, "with the right environment". Just because you can do something doesn't mean it's a good idea.

Also, this is missing the point of bringing up lua, so let me use a different example. Plugins don't necessarily even have to be files on disk. They could be entries in a json document or records in a sql database. Now plugins do happen to be executable files, but this is not part of the user interface, it is part of the internal hk-to-plugin interface, which is an implementation detail for the user.

Please consider this from the user's point of view. Making plugins into commands provides zero benefit. It just adds a second way to do exactly the same thing.

but what if I distribute my plugin via brew? Would the formula then resolve HKPATH

Homebrew doesn't consult PATH for commands, so why should it consult HKPATH for hk plugins? It would simply put them in /usr/local/bin and /usr/local/lib/hk/plugin, respectively.

That said, please don't distribute plugins with homebrew. If you do that, they won't auto update, and they won't be available on non-darwin platforms. This defeats the purpose of making a plugin in the first place.

What was your reasons for "mulling over this idea for a while"?

I'm not sure what you mean. It's just another idea, one of probably dozens I've considered for hk. Most of my ideas are pretty bad and I don't bother to share them with others.

[tt]

Please consider this from the user's point of view. Making plugins into commands provides zero benefit. It just adds a second way to do exactly the same thing.

Making plugins into commands is also not a loss; any decent auto-completion will complete hk before hk-plugin. I still think the pros outweighs the cons, but I'm also the one feeling the HKPATH pain. I get we don't agree on this, but I really think inventing similar infrastructure for something we can get for free is a somewhat violation of the "Heroku loves Unix."-principle.

That said, please don't distribute plugins with homebrew. If you do that, they won't auto update, and they won't be available on non-darwin platforms. This defeats the purpose of making a plugin in the first place.

I get we want plugins to mostly auto-update and be available to all platforms, but it may not make sense to everything. Perhaps you're writing something that integrates with a tool for a specific platform (i.e. a hk-debug may use HTTP filtering capabilities of that platform to provide diagnostics). I think the flexibility of executables is the strength of this model. We can provide a package that runs everywhere, but you're free to use some obscure scripting language and take a lot of system dependencies.

A good example are highly specialized plugins as hk-sudo. It's written in Bash. This doesn't run on Windows, but that's not really a problem for us. Perhaps it would be easier to just distribute a separate hk-sudo.bat for Windows users than to deal with the complexity of distributing a truly cross-platform plugin. (The mandatory file extensions in Windows is likely a separate issue with the current plugins.)

Homebrew is just one example. Because Ruby plugins obviously requires Ruby, distributing it via RubyGems can make sense. It's not that we want to promote this, but I also don't think we should make it more complicated for the users because there are legitimate use cases. Yes, Homebrew and RubyGems can install into a special path, but then the normal user does have to know about HKPATH.

I'm not sure what you mean. It's just another idea, one of probably dozens I've considered for hk. Most of my ideas are pretty bad and I don't bother to share them with others.

It was just to figure out if you explored this for other reasons than those already enumerated.

[kr]

Making plugins into commands is also not a loss.

This is false. Having two ways to do the same thing is confusing. It's a clear design mistake, and in this case easy to avoid.

inventing similar infrastructure

If this were plan 9, we'd actually be able to reuse the system infrastructure (bind mounting a bunch of directories onto a single path), but we'd still want to use a path other than /bin.

Unix doesn't give us that option, we have to either reuse the "infrastructure" (a 15-line function) along with the interface in every detail, including polluting the user's command space with things that aren't commands, or we have to copy that 15-line function. It's clear which option is less awful.

Yes, Homebrew and RubyGems can install into a special path, but then the normal user does have to know about HKPATH.

This is false. Normal users never need to think about HKPATH. Plugin distribution systems never need to look at HKPATH. They install into /usr/local/lib/hk/plugin, which is in the default value of HKPATH, by far the most common case.

It is exactly analogous to PATH in this respect.

[tt]

This is false. Having two ways to do the same thing is confusing. It's a clear design mistake, and in this case easy to avoid.

You'll always have multiple options. You're just making it less accessible and I'm not sure it's that much of a difference considering it's not very likely you'll ever know that the plugins are available in the search path.

Unix doesn't give us that option, we have to either reuse the "infrastructure" (a 15-line function) along with the interface in every detail, including polluting the user's command space with things that aren't commands, or we have to copy that 15-line function. It's clear which option is less awful.

I wish I knew what the Git design considerations where. I'm still feeling they must have had the reason we're missing.

This is false. Normal users never need to think about HKPATH. Plugin distribution systems never need to look at HKPATH. They install into /usr/local/lib/hk/plugin, which is in the default value of HKPATH, by far the most common case.

It is exactly analogous to PATH in this respect.

It is not.

Normally, you'd define your PATH to contain $(brew --prefix)/bin and $GEM_PATH/bin where in this case we're requiring that Homebrew and RubyGems if not understand the HKPATH value then the concept. This imposes the requirement that they're able to link into a non-standard location (which is something at least Homebrew will refuse; RubyGems probably not so much even though I defined GEM_HOME(!)).

This also still assumes that users don't change HKPATH entirely. (On a similar note, it's perhaps relevant that the Windows file system has no concept of /usr/local, so the current solution will require us to have the default depend on the platform.)

[kr]

You'll always have multiple options.

The question is whether we should present multiple options, not whether the user could theoretically have multiple options if they went through crazy contortions.

Normally, you'd define your PATH to contain $(brew --prefix)/bin and $GEM_PATH/bin where in this case we're requiring that Homebrew and RubyGems if not understand the HKPATH value then the concept.

Normally, you don't touch your PATH, and it includes /usr/local/bin by default, and homebrew uses /usr/local as its prefix by default, installing into $prefix/bin.

Normally, you don't touch your HKPATH, and it includes /usr/local/lib/hk/plugin by default (including when unset), and homebrew still uses /usr/local as its prefix by default, installing into $prefix/lib/hk/plugin.

If you, as a user, want to change your HKPATH entirely, that's up to you. Same goes if you replace your PATH entirely.

[kr]

I think we've been through this discussion pretty thoroughly, and there's still no compelling reason to change the current behavior. I'm happy to keep discussing this, either here or in some other forum, but let's close the issue to avoid cluttering the list of open issues.

[tt]

I think we've been through this discussion pretty thoroughly, and there's still no compelling reason to change the current behavior.

I disagree. There is only one reason to stay with the current design:

• The cluttering of the search path (and therefore auto-completion).

Besides the fact the cluttering the search path is what Git does for extensibility* without it being a practical issue, contrast this with:

• The current implementation is broken. (That can be changed, but it adds complexity.)
• HKPATH is, overridden or not, still an extra environment variable. Simplicity in design and in implementation is better; there's less to document and it works with the existing tools; want to know which plugin will execute for hk foo? Use which hk-foo! This is one of the benefits of not using Lua scripts; let's capitalize on that.
• HKPATH is a leaky abstraction; if it doesn't include the default path, things might fail. Why would it be configurable at all then? (That can be changed too, but then both hk and dependency managers need to parse it. The best option is probably to have a default that's never part of the environment, but works as a fallback internally. This means most users will never have anything in any of the HKPATH directories meaning it's only a power user feature and therefore the cluttering of the search path is not really an issue.)
• HKPATH is one thing on Unix systems and something else on Windows.
• Commands will have a prefix (which they arguably could in any case), so you won't have a sudo executable besides the one that ships with the system. This may seems irrelevant, but the consequence of this is also that you can't use export HKPATH=$PATH because everything would look like a plugin.

*) I tried looking for design discussions on the Git mailing list, but only found a thread where someone suggested a a different plugin architecture.

[kr]

Using PATH instead of HKPATH isn't a simpler mechanism, it's just different.

Surface area is definitely good to avoid.

HKPATH adds surface area (new env var) for a small number of users. Using PATH adds surface area (command clutter) for a large number of users.

if it doesn't include the default path, things might fail

Yes, this is also true of PATH. Why do you consider this a problem for hk but not for unix? If you decide to set your HKPATH to a weird value, that's up to you. If you want it to continue working with things like homebrew, don't do that. This is how PATH works too. If you set PATH=/my/random/place, guess what, all your homebrew commands will be unavailable.

[kr]

Git is not, in general, a paragon of good design or of good unix. It has many excellent ideas, and we should steal from it liberally, but any elements we use need to stand on their own merits.

[kr]

you can't use export HKPATH=$PATH

This is somewhat baffling. Why do you want to do that? I'm not in the habit of setting one variable to the value of some other, unrelated variable. I don't TERM=$SHELL or USER=$HOME, so why would I HKPATH=$PATH?

[kr]

The best option is probably to have a default that's never part of the environment, but works as a fallback internally.

This needs to be addressed specifically. We absolutely mustn't do this. It would defeat the purpose of HKPATH, which is to give the user control over where hk looks for plugins.

[kr]

This has been a useful conversation! It's great to talk through these issues and be able to say explicitly what our design principles are and how they apply to practical work. It also made me notice a bug in our current plugin lookup, see #81.

:icecream: