Making ruby-packer 0.4 actually work on Windows (it never did)

[ronaldtse]

Problem

v0.4 windows release and v0.4 linux/mac one are different things, and windows version has lot of not working components, because its built without openssl, digit/sha1 and some other things, so windows version of ruby has limited functionality.

Adding necessary functionality for build

I managed to build it with openssl and also fixed digit/sha1, md5 etc, which is required for bundler to work properly for example and download any https link, but need some more tests and this fix could be not the end of the story and require more fixing because there are no much docs and I just have to go through ruby code and windows C Makefiles and hack things to see how it would work out. I also made few ruby extensions work for windows, which didn't work previously:

+                       ext\etc\etc.lib \
+                       ext\fiddle\fiddle.lib \
+                       ext\fiddle\libffi-3.2.1\.libs\libffi_convenience.lib \
+                       ext\mathn\complex\complex.lib \
+                       ext\mathn\rational\rational.lib \
+                       ext\ripper\ripper.lib \
+                       ext\socket\socket.lib \
+                       ext\win32\resolv\resolv.lib \

but it seems to be not enough yet

there are few not working things left: win32ole, dbm, gdbm, readline but not sure will it help to have finally working ruby environment or not

native ruby extentions never worked before on Windows, same about statically built ruby, so its kinda experimental R&D and seems like there are not much analogues

Building with cl and clang-cl vs mingw-gcc

this ruby-packer project is mostly built with microsoft toolchain - cl , link.exe etc, all Makefiles created for nmake, which is MS tool also

but if we talk about native dependencies, like nokogiri etc, they mostly focused on mingw-gcc windows toolchain, for example nokogiri doesn't support MS cl at all and seems we need nokogiri, right?

it seems hardly possible to build everything needed with cl or even clang-cl

It's like we need to build main part of ruby-packer with cl and dependencies with mingw-gcc

while ld has -R (--just-symbols) option there is no analogue for Windows and just building with /FORCE to ignore errors about undefined symbols leads to crash as we not able to call ruby functions after like rb_define_module etc probably microsoft linker does some dirty work with excluding calls to undefined symbols etc when common unix ld linker just keeps them inside even so they are undefined so these calls could be found later in ruby executable but not in windows case so I still can't make byebug working with cl even so dll is loaded from squashfs correctly with LoadLibrary() etc

don't know how to pass it

the way common ruby works on windows - we have ruby.dll and all extentions linked with this dll, but in case of ruby-packer we build exe statically

and this is kinda experimental, because no any official ruby doesn't support such static builds on windows we could try to build most of ruby with cl and try to build deps with mingw-gcc, hoping it would work

Compiling a statically-linked Ruby

also I created an issue on official ruby bug tracker here: https://bugs.ruby-lang.org/issues/16795 Feature #16795: build ruby.exe on Windows against ruby-static.lib omitting x64-vcruntime-ruby.dll (no '--enable-shared' option for Windows) - Ruby master - Ruby Issue Tracking System https://bugs.ruby-lang.org which means we can't build ruby with static lib on Windows, but its possible on linux/mac I solved it via hacks

Linking to a statically compiled Ruby

on linux it seems that byebug.so not linked with ruby shared library, or static one and I tried to link it with ruby statically

but the trick is ld linker which is used on linux/mac has -R filename in automatically generated byebug Makefile, and there is no such option on Windows, because native extensions on this platfrom always were linked with ruby.dll which is not an option in our case, as we link ruby statically for ruby-packer

so its like it passes undefined calls on linker stage to ruby functions like rb_define_module() and avoids 'unresolved external symbol' errors with -R flag and reading symbols like function names from static lib but not adding them to produced binaries this seems to be the key point also MS linker has no such an option by we could pass such errors with something like /FORCE flag

1. We should maintain a forked Ruby , which can be statically-linked.
2. Our ruby-packer should use this statically-linked Ruby and maintain patches

https://github.com/metanorma/packed-mn/pull/14#issuecomment-622989468

[ronaldtse]

Closed in favour of https://github.com/tamatebako/tebako/issues/1