The future of pre-compiled gems

[MSP-Greg]

Thank you for being here. I’d like your opinions on the below. There may also be information you’re unaware of. Of course, I’ve placed it at the end…

Intro

The term ‘fat-binary gem’ has often been used. I think the term ‘pre-compiled gem’ is probably a more descriptive term. I’ll use the abbreviation PCG for them.

Ruby builds 2.3 and earlier are done with a proprietary build system known as RubyInstaller (RI).

Ruby builds from 2.4 onward are known as RubyInstaller2 (RI2) builds, and use a non-proprietary build system (MSYS2/MinGW). Again, thanks to Lars Kanis for his work on it and the runtime.

Should PCG’s exist?

In recent discussions, the issue of RI2 (ruby 2.4 +) adoption has been discussed, and the lack support for PCG’s has been mentioned as a strong deterrent, especially for using Ruby 2.5. The idea of stopping support for PCG’s has also been mentioned.

I believe that support for PCG’s should continue for the following reasons:

11. New Windows users are often unfamiliar with building. Additionally, there is the case where very knowledgeable MacOS/*nix users may be trying to test on Windows, and require an extension gem for testing. They also may be unfamiliar with build tools.
12. Gems requiring OpenSSL. To build a PCG covering Ruby 2.0 thru 2.5 (or trunk) requires 4 different OpenSSL packages in two different build systems. Same for testing. Don’t think that’s common knowledge…
13. Compilation requiring additional code/packages. RI2 has mechanisms for helping with this, but new users may still run into problems.

Ease of PCG construction

If PCG’s should be supported, what can be done to improve their support for current Ruby versions?

There are tools available to help build PCG’s. Regardless, gem owners/maintainers are lagging behind with releases supporting new Ruby versions.

After Ruby trunk testing on Windows was stable, I started encouraging gems to test using Appveyor, and also use trunk. Most of the gems I worked with were extension gems.

Recently, I considered the issue of PCG’s lagging behind.

My first thought was that they should be built on Appveyor, as all the required software is installed on their standard image. No one has to deal with cross-compiling.

Later, after experimenting with it, I realized that, given that testing involves compiling, why not combine PCG construction along with testing. Run it real-time, so every commit/PR constructs the PCG’s and tests them.

So, this repo contains the shared PowerShell files for that integration. To use it in on Appveyor, it requires changing the appveyor.yml file, and three additional files.

The first is a PowerShell script, which defines constants related to the gem, and also functions that are called by the shared framework. The second file is a ruby file that builds the gem. Third file is a rakefile specific to testing on Windows, as many rakefiles involve several tasks and often require gems that are not needed for testing.

Note that because the system tests by installing the gem, test files must be included in it.

Notes on the framework

21. I needed code to do some pre and post processing, but the main task was to loop thru several Ruby versions and perform the same task. Also, when used on Appveyor, some PostScript objects were being called (handling test results, saving artifacts). Hence, I chose PowerScript.
22. It runs locally with the addition of one file with the local paths. All the gems I’ve done PR’s for I’ve run locally first. There is also a script in an Appveyor repo (and my fork) that will install all the Ruby related files needed for local building. It installs DevKit files, but not MSYS2/MinGW (RI2 build system) files.
23. It makes use of the naming conventions for Ruby versions on Appveyor. Because of this, when Appveyor adds new Ruby versions, they will automatically be added to the PCG’s.
24. I’m open to helping with changing the PowerShell code to Ruby, although I think PowerShell is a better choice. I’m also open to moving the code elsewhere, like OneClick.
25. Until it’s adopted, I haven’t done a README, wiki, etc

Notes on Appveyor

31. Unlike Travis, for a given platform (32 or 64 bit) and Ruby ABI version, there is only one Ruby install available. Trunk must be downloaded.
32. RI2 builds use MSYS2/MinGW. Currently it is using gcc 7.3.0, and for versions Ruby 2.5 and later, OpenSSL 1.1.0. This may be helpful for testing, as the versions on standard Travis are much older. libyaml and zlib versions are also newer.

At the time of writing, I have done PR's using this system for puma, sqlite3-ruby, and bcrypt-ruby. I should be updating a PR on EventMachine this weekend

Thanks for taking the time to read this, Greg

[sodabrew]

Subscribe! (I know, I can do this with the little button the side, but I want to very visibly subscribe to the thread :) )

I maintain eventmachine and mysql2 and have spent sooo much time over the years sorting out Windows builds.

Aside: It's a very thankless task. Microsoft would be well served with their forays into Open Source, not least of which is now buying GitHub, to also evangelize to Windows users on best practices for engaging with open source.

[OSXLich-Doug]

My opinions (1.5 years after asked?!):

1) PCG required: technically no, but I subscribe to the user design principle that suggests you do everything you can to provide the user with a usable experience. Requiring a user (on any platform) to build their own native gems before some software can be used doesn't meet that design principle.

1a) With annual Ruby updates and ad-hoc (not scheduled by a public release plan) updates to native gems compounds the problem and users should not need to be exposed to this cycle or be required to build gems simply to continue using their software.

2) Therefore, should PCG exist: Absolutely yes, and a reliable, repeatable, sustainable and very public method of creating PCG and validating them should be available to all integrators and developers who may need to rely on native gem functionality.

2a) With the continuing challenges in safe-computing and cyber-risk, integrators and developers will need to be able to rapidly update their applications -- to include the Ruby version as support ends -- easily. The discussion about not updating to 2.4 or 2.5 because of build challenges at a user level will only be increased if developers don't worry about updating their applications because of the challenges with moving to an updated Ruby version.

3) I've been considering the best way to build PCG for a small audience I support, and given that the last generally available gemset with prebuilt binaries ended with Ruby 2.5 support, I've been eyeing cross-compilation as my likely response for 2.6 and 2.7. Fortunately, I tripped across a PCG that MSP Greg developed (sqlite3 4.0 in fact) and in my research on how it was produced, I landed here.

4) I intend to initiate builds for the GTK2 native gem, just as SQLITE3 was managed - which means I intend to try to follow this process. If successful, I'll swing back by and cross ref. Hopefully there will be others. . .

Thanks, MSP Greg!