Publish a Windows build (crystal.exe)

[oprypin]

Right now it is kinda-possible to get a pre-built Crystal (.exe) for Windows, by clicking into one of the GitHub Actions runs. Unfortunately, this requires being logged in to GitHub Web (yes, no anonymous access through web and no access through APIs at all).

Of course, the end game will be to have a Crystal build for Windows as part of the Releases, but right now there isn't yet such a release to speak of (0.34.0 didn't have essential fixes for Windows yet).

My main point here is that we should be ready for that when the release comes. It would simplify a lot of workflows (including Crystal's own CI), and I'm personally very excited about it.

The minimal thing to publish might even be the artifact archive that the workflow produces (at the commit corresponding to the release tag); for start it would suffice to manually download it from there and upload it under the release. (We could also tell GitHub Actions to do that automatically when a tag appears, but it requires the release to also already be there, which might not fit with the current release workflow.)

Of course, the contents of the artifact (just crystal.exe and static libs) aren't exactly in line with the other systems' releases, which have share/crystal/src (source code) and bin/crystal (shell wrapper, to which there is currently no Windows equivalent). And there have been no discussions at all about the directory structure that Crystal should be installed as. But I would strongly prefer having some download even if these details aren't fleshed out.

Publishing the nightlies themselves somewhere would also be nice. I would appreciate having an official download URL for any crystal.exe even before the release happens, but maybe that's asking too much right now. And I hear that the story with Mac nightlies also isn't so great as it requires hoops through CircleCI APIs, so maybe there's some common solution here.

[oprypin]

So it would be nice to publish some kind of binary artifact for the 0.35.0 release, even just for Crystal itself to stop relying on bootstrapping in its CI setup.

I could just keep relying on the Artifacts functionality (look up the CI build that corresponds to the commit that's tagged as 0.35.0), but I'm afraid that the storage for it is not persistent, it'll just disappear eventually. So actually this is not possible, a persistent location really is needed.

Adding @bcardiff for specific questions:

1. Would you be fine to just take the binaries directly from the current continuous build, and put them in the GitHub release?
2. If not, is there anything we need to change in Crystal to make it viable to include binaries for the release?

For reference, the current binary artifacts include just these 5 files:

• crystal.exe (built without --release flag, fully statically linked)
• Static libs in release mode, /MT:
  • gc.lib
  • pcre.lib
  • yaml.lib
  • z.lib

Might be nice to also include source code in the archive.

[RX14]

I think i'd prefer the code that produced any release tarball to live in here...

[bcardiff]

This repo is responsible of keeping track of how the releases are built. Since the sha is used in the main repo, there it is all the information on how to keep the lineage of the compiler.

I think we can publish some windows binary as long as we include in this repo some scripts that builds them.

The scripts in the different folder need to know the sha and crystal version to build plus using which previous released tar.gz should be used.

This means that for a couple of version this process implies cross compiling from linux. If at some point we want to bootstrap a windows lineage we can do it, but it will be a second phased.

I would probably delay that second phase until we discuss a windows installer.

But as a preview for windows, cross-compiling and building a .zip (?) package sound reasonable to me.

Since the windows process is run in GitHub Actions instead of CircleCI we will have some duplication when updating the distributions-scripts.

Let me know if the branch filter in circleci config need some clarification. I would expect something similar to happen on this new workflow.

[oprypin]

Uhh I didn't reply for a while, but the main reason for that is I actually don't understand what you mean, especially the last sentence :s

---

Some inline comments:

This repo is responsible of keeping track of how the releases are built.

(Technically, both the crystal repo itself and this one are used for builds.)

I think we can publish some windows binary as long as we include in this repo some scripts that builds them.

If one added a trivial (one-line) script here to just defer to the main repo, it would technically also be "both the crystal repo itself and this one" :yum: But that's just a side note, something I found funny.

This means that for a couple of version this process implies cross compiling from linux. If at some point we want to bootstrap a windows lineage we can do it, but it will be a second phased.

Importantly, there is no way to produce a Windows build having just Linux, it's either Linux + Windows, or just Windows -- both ways are possible (and are built continuously) now, the only hurdle for the latter is that there is no permanently-stored crystal.exe to refer to. That is what I really care about, the rest is side details.

I would probably delay that second phase until we discuss a windows installer.

I don't fully understand why you want to delay switching to Windows-only, as that's so much simpler. I guess you want to get the process fully "clean" without possibly-non-reproducible steps (if you consider GitHub Actions non-reproducible)? I suppose nothing stops you from doing that later when the process is fully established, just re-publish a few prior steps.

But as a preview for windows, cross-compiling and building a .zip (?) package sound reasonable to me.

OK, done! That is already being produced by GitHub Actions, just grab it from there. I could also make some changes to the workflow if you want, e.g. it could probably be made to publish the .zip into the release by itself.

Since the windows process is run in GitHub Actions instead of CircleCI we will have some duplication when updating the distributions-scripts.

OK, I have no input there, but if this phrase has some secondary meaning to it, I don't know what it is.

Let me know if the branch filter in circleci config need some clarification. I would expect something similar to happen on this new workflow.

Umm this is where I'm entirely confused. Just to clarify, (if that's what you mean) I have no intention of learning any existing CircleCI config or adding to it. Especially if you insist on a two-stage (two-OS) config.

Ideally we'd get to a situation with just one OS before having to finalize a solution, otherwise it'd need a lot of difficult work that would be thrown out right afterwards.

Or maybe you just mean I should look at the config and make something similar with GitHub Actions but in distribution-scripts repo? Not sure what's meant here, and not sure what's best. Anyhow, I haven't looked at the CircleCI config.

---

As a summary to clarify my intention,

Unfortunately, I don't intend to be the main author of a proper solution based on distribution-scripts and/or CircleCI. I also don't think we're ready for a proper solution yet. I'm just asking to publish some Crystal binaries, and ideally (but not necessarily) attached to the GitHub release of the main repo. That's why I have directly asked about the possibility of one particular temporary solution, without any intention of it becoming authoritative.

1. Would you be fine to just take the binaries directly from the current continuous build, and put them in the GitHub release?

[RX14]

I'd prefer to either publish something just for the CI and work out a temporary solution just for that (not attached to a release at all), or work out a full "extract and run on windows" .zip file, similar to the linux tarballs.

Publishing just the compiler executable somewhere where people who don't know the windows port details can find it isn't something I'd like to do.

[oprypin]

work out a full "extract and run on windows" .zip file

Generally this is a longer-term solution, which I'm not focusing on. You could kinda do it now, - just directly combine the artifact archive with the source code archive. But as there's no installer or such, this always requires setting some environment variables.

I'd prefer to either publish something just for the CI and work out a temporary solution just for that (not attached to a release at all)

That's completely fine by me. Ideally it'd be a very stable location, so perhaps https://github.com/oprypin/install-crystal could rely on it as well.

[straight-shoota]

You could kinda do it now, - just directly combine the artifact archive with the source code archive. But as there's no installer or such, this always requires setting some environment variables.

That's totally fine. We can use a wrapper script like on POSIX to set CRYSTAL_PATH relative to the script, just like the tarball for linux.

[RX14]

Yeah - lets focus on just a s3 bucket or something simple to upload windows crystal.exes to for the short-term. The immediate problem is having a stable storage location with a stable URL. That can be solved rather simply. Lets sort the rest later.

[bcardiff]

Even if we publish the current CI's output in the GitHub release artifact I would not use that binary as input for future releases. We will be missing the reference on how that binary was built in the first time.

We could publish a binary as a preview, but I would avoid using it in any automated/integration until there is a more stable a feature complete zip / installer.

What I imagine is that in this repo we can have a:

• windows/prepare.sh to perform something similar to the linux-job. Having as argument the previous linux tar.gz and the crystal sha1.
• windows/build.bat to perform something similar the windows-job. Having as argument the directory of the crystal.obj file and other arguments.

We could have a windows/deps.bat that would build the dependencies and decouple that. We can store those in S3 and use them as input in windows/build.bat. That is windows/deps.bat can be manually triggered as we are doing today in this repo to update the llvm for darwind pkg.

This prepare and build would be run in GitHub action on nightlies and tagged releases. Once they are published for a first time (at least) we could drop the prepare and change the build to use an input the url of the previos windows .zip package, hence using windows to build the next windows compiler.

I know this process is not as comfortable but is a way to keey a better reproducible build script to keep the lineage of the compilers.

Think for example that rebootstraping the windows compiler directly from the github actions is something that can't be done easily in a couple of years when the services is changed in a substancial way.

[oprypin]

Hmm I had covered that before.

I guess you want to get the process fully "clean" without possibly-non-reproducible steps (if you consider GitHub Actions non-reproducible)? I suppose nothing stops you from doing that later when the process is fully established, just re-publish a few prior steps.

---

Really the most annoying problem is the need for MSVC to be installed and the environment set up for it (which is also not a fully transparent process). So the straightforward approach would just assume that it's active. The non strightforward process would need to look for MSVC itself, and it could be in various locations... And with any kind of approach, this relies on proprietary software to be kind enough to still be installable :)

---

.bat

Yeahh that one is probably not expressive enough, need powershell. Which brings its own complications, like Windows not allowing you to run powershell scripts unless you reconfigure the system for it??

[bcardiff]

I didn't mean to force CMD. A powershell script is valid. Assuming that MSVC is installed is valid to me. As long as it is expressed in a README I'm fine.

It's is a grey area. To me it make sense to have something that almost usable in a context that is not GitHub actions. Not because GA is bad, but to ensure we can move and use the process eventually in other environments, like locally building previews, etc.

[oprypin]

OK.

We could have a windows/deps.bat that would build the dependencies and decouple that. We can store those in S3 and use them as input in windows/build.bat.

How would the running and the storing happen, btw? Because either someone would need to manually run Windows, or GitHub Actions would need to write to S3, or there'd be a manual copying step in between? How will we set this up in reality?

[bcardiff]

In Linux currently we build all the deps. In osx obminus we have two main targets: crystal or llvm. The llvm is run manually in this repo in CircleCI on a specific branch name. We can add GitHub actions to this repo to build llvm on windows and whatever other dependency we think is time consuming.

[RX14]

I feel we're still getting off in the weeds. We're trying to sort out a CI only bootstrapping problem. This build isn't going to users, so we shouldn't worry about it's lineage. We're just trying to get rid of the "linux" part of the windows CI.

When we release we can cross-compile and think more about these details.

[rishavs]

How about adding a scoop > Extras bundle for Crystal, while a lot of these details are being fleshed out? Scoop can hide all the gnarly internals and provides scripting and dependency management. Here is a sample for rust-up https://github.com/chawyehsu/dorado/blob/bf8643c053c831b3f7b394eab4edf172000bcbb9/rustup.json#L18

Being able to just do scoop install crystal on windows would be amazing.

I think, right now it is better to just package the relevant bits like cl.exe and link.exe with the crystal.exe, instead of building all dependencies from source as well. Later when all the details have been hashed out, we can do a much more involved process.

[rishavs]

Since the other thread got closed, hoping to get the conversation restarted here. Can we look at adding the crystal.zip with the windows compiled files to the release page? That's the smallest first step we need to take. we can look to an ideal solution later.

[oprypin]

This is still very relevant.

The latest Crystal tag is usable for Windows and has no advantage over Crystal nightly.

https://github.com/oprypin/install-crystal is also very content with prebuilt Crystal as it is being uploaded from the artifacts. The archive is usable with minimal setup.

However, that action is forced to offer only Crystal nightly for Windows because there is no stable location for the build of a particular tag; an artifact build will expire eventually.

A solution that would work well is to attach the archive from the artifact directly to the release. But I can see that maybe it'll confuse people.

A weird solution that I'd also accept is to get someone to bump the artifacts build on GitHub once every 3 months...

I can also very easily just copy these archives onto some website of my own, but that'll just look shady to most.

[straight-shoota]

But I can see that maybe it'll confuse people.

How so? Isn't that the same we're doing for linux and mac?

[oprypin]

Well those builds are directly usable, while the ones for Windows require setup, and, you know, the actual Windows support is not "production-ready" to say lightly.

[rishavs]

With 1.0 out, can we look back into this? Crystal on windows is usable today for many usecases bar networking based ones, and we should make it easy for people to use Crystal for these use cases (like desktop apps, gamedev etc).

We should have an approach for easy installation of Crystal on Windows today. Anything we can do to support package managers like Winget, Chocolatey, Scoop etc will go a long way in unblocking windows user in using Crystal.

[rishavs]

Really the most annoying problem is the need for MSVC to be installed and the environment set up for it (which is also not a fully transparent process).

Can we not look at the simplest path forward and list VC20XX as a dependency for now, and then look at removing this dependency at a later date?

A solution that would work well is to attach the archive from the artifact directly to the release.

This is the one I favor as well. Simply having access to the Crystal windows builds in the Release would allow us to setup crystal via the windows package managers.

[rishavs]

Returning here after some months to hopefully rekindle this discussion;

With 1.10 in the horizon, hoping this would get picked up. I know we are doing some dist script cleanups at https://github.com/crystal-lang/crystal/pull/10639

Just having the crystal.exe and the stdlib files as part of the release artifacts/zip would make it so much easier to use Crystal with any windows package manager. This is very low hanging fruit and IMO has great impact for people planning to use Crystal for non-web related use cases on Windows (gamedev, windows apps, GUIs etc)

[Blacksmoke16]

Is it still a goal to have an actual crystal.exe release asset? Or are we good to close this with the introduction of the installer via https://github.com/crystal-lang/crystal/pull/13578?

[oprypin]

Not only an installer, but also an archive that directly includes crystal.exe is being built both in CI and releases.

https://github.com/crystal-lang/crystal/actions/runs/10472900288 https://github.com/crystal-lang/crystal/releases/tag/1.13.2

So certainly, this seems to be fully resolved