Connect `executables` with `targets`

[straight-shoota]

shard.yml allows defining executables that are supposed to be installed in the main project's bin/ folder, and it allows defining build targets. But they are currently not related. If executables need to be built, the only way to do that is run a build command in postinstall. Ideally, that's just shards build.

name: example
version: 0.0.0

targets:
  foo:
    main: src/foo.cr

scripts:
  postinstall: shards build

executables:
- foo

I think it would be a good opportunity to automate that by hooking up executables with the available targets information.

Operation would be quite simple: When shards cannot find a file for a declared executable, but there is a target definition of the same name, it runs shards build $executable_name. Then the executable file should exist.

This works without disrupting existing workflows where the executable is already available in the source or built by a postinstall hook because existing files would be preferred.

An immediate result is simplification of the shard.yml, with the additional benefit of turning imperative instructions into declarations (which helps portability).

So the above shard.yml could be shortened to this:

name: example
version: 0.0.0

targets:
  foo:
    main: src/foo.cr

executables:
- foo

No need to define shards build (or the equivalent using make or whatever) in postinstall. This probably won't work for all shards, because some have more complex builds. But I figure it should be good for the vast majority of typical development tools.

As a further enhancement, instead of building directly upon installation, the executable could actually be a shim which invokes the build command on demand. This avoids waiting for executables to build when running shards install and only builds them if they are actually used. That causes some delay when used the first time, but that should be acceptable: If you want to use it, you have to wait for the build anyway at some point. But if you don't use it, there's no need to build it!

This proposal is based on some ideas previously mentioned in https://forum.crystal-lang.org/t/shards-postinstall-considered-harmful/3910/14?u=straight-shoota

[Vici37]

I really like this idea of being able to list out executables that map to build targets, and let shards take care of the building rather than implementing a build script of a sort via make (or any other build tool of choice).

As a further enhancement, instead of building directly upon installation, the executable could actually be a shim which invokes the build command on demand. This avoids waiting for executables to build when running shards install and only builds them if they are actually used. That causes some delay when used the first time, but that should be acceptable: If you want to use it, you have to wait for the build anyway at some point. But if you don't use it, there's no need to build it!

This portion I dislike, but maybe I'm not seeing the benefit - why would I want to wait for a tool to build the first time I invoke it, instead of when I run shards install? When I run shards install, it's already a fire-and-forget command and I can revisit that terminal a little later (or at least be confident that there are no inputs required by me). When I run a tool, I expect to provide input / immediate feedback. Forcing me to wait while I'm expecting immediate feedback is a poor developer experience, even if it's only once in awhile.

I can't speak for others, but I only include executables / tools that I actively use in my shards.yml, so this may impact me more than what you yourself experience @straight-shoota.

[Sija]

This portion I dislike, but maybe I'm not seeing the benefit - why would I want to wait for a tool to build the first time I invoke it, instead of when I run shards install?

@Vici37 IIUC The reason is to reduce the shards install run time. Building Ameba for instance is slowing down the process considerably.

[straight-shoota]

@Vici37

I only include executables / tools that I actively use in my shards.yml

Use cases differ. I would argue that I might not need all these tools if I just want to checkout the code of your shard, or maybe to build it and run tests? Think about an imaginary tool for database migration: If your database doesn't change much, it might only be used once in a while. There's really no need to have it built every time you update the database driver. I have written about this more extensively in https://forum.crystal-lang.org/t/shards-postinstall-considered-harmful/3910. The gist:

Every time I run shards install for a shard using ameba, ameba builds itself. But that’s only useful if you want to contribute to the shard. Most of the time, I don’t need it. And I certainly didn’t ask for it. Why do I have to wait for ameba to build? Ameba shouldn’t build itself every time it’s installed. I picked ameba because it’s popular and has stressed my patience many times now. But many other shards are very similar.

And I'm not suggesting that lazy building should be the only option. If you want to have all executables build on shards install, that should certainly be possible. We can discuss which should be the standard behaviour and if there should maybe be a user option for selecting the behaviour.

[Vici37]

That's fair. Reading your reasons for wanting to pull and build a given shard here, it does look like there's existing shards flags to get the behavior you want, such as --skip-postinstall and --production.

I'm not opposed to creating a new sub-sub command to handle this phase (i.e. shards install tools or some such), to split out the lifecycle of a dependency's executable from the dependency source code. Overall, I'd want to apply the principle of least surprise as the developer experience, and I think an async flow (with or without shims) would lead to more surprises rather than less.