`postinstall` flexibility

[didactic-drunk]

Gitter discussion provided to save time and not misquote @oprypin .

@oprypin

i see. well, maybe my main dissatisfaction comes from how rigid postinstall is. any tiniest problem and everything just bails out and you're basically taking on a task of maintaining a fully universal build of a lib.

@didactic-drunk

Yes. If only postinstall allowed

postinstall:
  linux-musl: ./build/linux-musl.sh # needs variant for some reason
  linux: ./build/linux.sh
  windows-msvc: ./build/windows-msvc.cmd
  windows-mingw: ./build/windows-mingw.cmd
  posix-default: cd build/posix && make
  windows-default: cd build/windows && make
  default: make

And tried keys in order of most specific to less specific arch trying posix-default or windows-default 2nd to last before default.

@oprypin

@didactic-drunk: well it looks like you're already miles ahead of the rest in terms of thinking this through. I'd support this

Why my build scripts are so complicated

Attempting to improve UX (developer experience DX, is that thing?). There are 3 unix use cases. 1) Building a rpm/deb package (or personal executable) that relies on the distribution packaged libraries. 2) Building a run on any distro binary that can't rely on a system installed libsodium. 3) Same as 1 but the system libsodium is ancient and can't be used (This is very common). The build scripts check the system version (but allow overrides) and uses them by default if new enough otherwise download and build the version tested with sodium.cr. None of this is a concern for windows which never has a packaged libsodium

shards install takes care of everything unless you want to override a value provides the best DX even if the build script is more complex. I doubt I'd have half as many users if it required: manually check your distro version. If it's >= x install else download and build it yourself.

Assume you make a program using compression and encryption (not too uncommon). I'll use zstd.cr and sodium.cr for this example. More than half the time zstd and libsodium are too old and need to download a newer version. For windows it always requires downloading a prebuilt package. If I ask users to do this manually the friction in either building for their own use or forking to make a small improvement is too large.

Not just building. Do I ask them to check sha256 and minisign?

Can't you let the user handle it?

On windows. Click download. Find the download. Move it to the build directory. Unzip it. Rename from libsodium-1.0.18 to libsodium or have a complicated build script that attempts to use the latest version in the specified directory. If I go with a build script finding the version I may as well just download and unzip it for them and I'm back where I started.

[straight-shoota]

This is too much of a build system in shards. I don't think going in that direction is a good idea. Shards is a dependency resolver. Other tools are much better at building stuff.

And it's really not necessary that shards switches over platform-specific instructions. You can do that as easily in a shell script or with a build tool. Shards should just get a single command to run, and - if necessary - that command can be used to take different paths depending on the environment. Writing a portable shell script shouldn't be too hard for posix systems. The only trouble is windows (see #468) but we can make that work. It just means there's going to be two postinstall scripts, one for posix (sh) and one for windows (cmd). If we can't find a reliant way to make this work implicitly, we might need to do this:

postinstall:
  linux: ./build.sh
  windows-msvc: .\build.cmd

[devnote-dev]

Is there any specific reason for not implementing this? I don't find "too much of a build system" as a valid reason here. I would understand if this were an issue to redesign Shards as a build system which has its own set of issues, but the core of this issue is just platform-specific handling for postinstall. If such a feature was that problematic, why keep it in Shards altogether? The argument to "use a build tool" or other tool for handling platform-specific requirements also isn't valid in my opinion, given that such tooling varies largely across different platforms (for example, make on Linux vs Windows).

I believe this is also a major blocker for Windows adoption within the Crystal community. For a long time (and to this day) the community has been very much Linux first, with Windows support being a fairly recent addition to Crystal and even newer to the community. As a result, there has been a big initiative to get existing shards to support Windows, but many that make use of external libraries or require platform-specific configurations can't (easily) do this because of Shards' own lack of support. Some alternatives have been to execute a Crystal file in postinstall that handles platform-specific logic/configuration (which has been discouraged by members of the community), another slightly worse alternative is trying to handle said logic/configuration in the postinstall hook, however, most either stick to platform-specific scripts or don't provide one at all, forcing the user to handle the rest of installation themselves.

At the very bare minimum, this functionality should be implemented to further help bridge support for Windows in the community, it wouldn't make Shards any less of a dependency resolver. That's just my 2 pence on the situation though.

[oprypin]

Yes, something needs to change, because shards likes to pretend that it always leaves you with a fully working installation (and bails out on error and doesn't really like you meddling in its business), but then at the same time we say it shouldn't be a build system. I think these two sentiments are incompatible.

[straight-shoota]

I believe postinstall is almost exclusively used for functionality that closely resembles a build system.

Building dependencies in a portable manner is hard. Not even talking about POSIX and Windows. It's already hard enough to consider different configurations of the same operating system or distribution. I don't think there is any way to ensure a predictable environment for building arbitrary dependencies.

The idea of postinstall providing a hook that automatically provides any non-crystal dependencies upon shard installation sounds nice. But it's a dream detached from reality.

I don't know any other reasonable use case for postinstall.

[oprypin]

Then we may have to update the expectation that's been there for years that one never needs to enter the lib folder and meddle with it. In fact the expectation was so strong that in the past there was an attempt to hide this directory. Which I strongly argued against. https://github.com/crystal-lang/crystal/pull/9280#issuecomment-627854577

But now @straight-shoota if I understand what you're saying, this sounds like maybe users would have to meddle with the directory then. Which is OK by me if that's the message we decide to send.

[straight-shoota]

Maybe. I don't think that's a necessary conclusion. It's a possible one, for sure.

All I'm saying is that I believe postinstall in its current form is not a solution to the problem it tries to solve.