Compile with Clang on Windows

ronaldtse commented 4 years ago

Make Windows compilable with Clang.

repu1sion commented 4 years ago

my commits are based on top of 0.4.0 release tag. this release is like 70 commits behind the original ruby-packer master branch. So I don't think this is good idea to merge them like this into your master (which is rather forked from ruby-packer master). The reason from my side to do so is because latest master doesn't work and we need something working to ramp up.

ronaldtse commented 4 years ago

From @repu1sion :

I was able to build metanorma with clang-cl.

When I run it from cli I see something like:

c:\archive\develop\ruby_packer\builds\metanorma_01>metanorma.exe
c:/__enclose_io_memfs__/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require': cannot load such file -- bundler/setup (LoadError)
        from c:/__enclose_io_memfs__/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
        from /__enclose_io_memfs__/local/bin/metanorma:5:in `<main>'

@w00lf do you know what problem this is? It seems that bundler is not finding some gems?

ronaldtse commented 4 years ago

@CAMOBAP795 would you have time to help on this? This is a Windows issue. Thanks!

w00lf commented 4 years ago

From @repu1sion :

I was able to build metanorma with clang-cl.

When I run it from cli I see something like:
c:\archive\develop\ruby_packer\builds\metanorma_01>metanorma.exe
c:/__enclose_io_memfs__/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require': cannot load such file -- bundler/setup (LoadError)
        from c:/__enclose_io_memfs__/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
        from /__enclose_io_memfs__/local/bin/metanorma:5:in `<main>'
@w00lf do you know what problem this is? It seems that bundler is not finding some gems?

Yes, it seems that the patch for dependencies lookup from ruby-packer compilator is not working at all here, although i still need to investigate it by myself from within, planning on doing it after i will compile new version.

ronaldtse commented 4 years ago

@w00lf Maybe this has to do with the difference between 0.4.0. and 1.0.0. It seems that most of the commits between 0.4.0 and 1.0.0 are refactoring ones, hopefully we can get everything working before end of year.

ronaldtse commented 4 years ago

@repu1sion There is a Chocolatey package for flex and bison: https://chocolatey.org/packages/winflexbison

In addition, the Ruby Development Kit can be installed using code like this:

https://github.com/metanorma/chocolatey-metanorma/blob/6fad7fb179ac66a13bc6bf8444f2f703d8d94c6f/metanorma.nuspec#L25-L27

https://github.com/metanorma/chocolatey-metanorma/blob/6fad7fb179ac66a13bc6bf8444f2f703d8d94c6f/tools/chocolateyinstall.ps1#L13-L25

repu1sion commented 4 years ago

@repu1sion There is a Chocolatey package for flex and bison: https://chocolatey.org/packages/winflexbison

Yeah, I know, I checked all powershell scripts you did etc. It's rather a note on how to build/install everything on local/not headless virtual machine because as we found with Mikhail its not obvious that you have to install Windows 10 SDK etc. And this way of installing packages also works (not for GHA of course).

repu1sion commented 4 years ago

also after lot of different experiments with GHA, it seems that if we switch to windows-latest (which is Windows 2019 server) we would have latest available MSVC with clang by default. The problem is this default clang is x86 one (32-bit version) and there is no x64 one. Sad but true. So in this case choco really helps, because it contains 'llvm' package with clang-9.0.0 x64 by default. Another thing is that its not able to update path somehow, so we also have to do it manually, but finally in my case clang-cl 9.0.0 x64 was found. Also default swapfile could be not enough, so probably it would need to be extended.

repu1sion commented 4 years ago

systeminfo on GHA shows next memory info:

Total Physical Memory:     7,168 MB
Available Physical Memory: 5,549 MB
Virtual Memory: Max Size:  8,959 MB
Virtual Memory: Available: 7,459 MB
Virtual Memory: In Use:    1,500 MB

9Gb could be not enough, need to investigate, could this swap be auto extended if its auto-managed and where is the limit.

repu1sion commented 4 years ago

Ok, finally got ruby-packer built on GHA with clang, windows platform. Few notes:

works with windows server 2016 only. doesn't work with 2019 because of this issue: https://github.com/pmq20/ruby-packer/issues/83
clang 9.0.0 (8.0.1 should be also ok)
ruby 2.4 (should work with 2.6 also)
ruby-gem 2.6.14 Also in case of cl there is out of heap issue (as expected) and in case of clang-cl it works fine. Max available virtual memory shows 9Gb (7 physical +2 virtual) but maybe swap was increased automatically in this case. And maybe in case of building packed-mn this will be not enough, so we would need to increase swap file, will see. This is based on ruby-packer 0.4.0 release, not master.

ronaldtse commented 4 years ago

That’s great @repu1sion ! Could you add the GHA files to this branch and remove the Travis configs?

I suspect the issue at https://github.com/pmq20/ruby-packer/issues/83 may be resolved by upgrading to ruby 2.6.

repu1sion commented 4 years ago

ok, here is the branch: https://github.com/metanorma/ruby-packer/tree/clang It builds well. And no, ruby 2.6 doesn't help with https://github.com/pmq20/ruby-packer/issues/83 anyhow. Also I tried all versions of ruby: 2.4, 2.5, 2.6, both installed from choco and via setup actions - it doesn't help. This bug is rather related to C compilation and some .NET issues, because it fails during linking and then running executable file miniruby.exe which is linked with cl/clang linker from C sources. And it links fine on older platforms like Windows-2016 and older and fails only on Windows 10 and Windows-2019 server. Also maybe we could fix it if we switch to newer version of ruby inside ruby-packer, like 2.5. So I'm going to merge few commits from master into our working branch and check the changes. Also maybe it would help with Mikhail's issue.

ronaldtse commented 4 years ago

Also maybe we could fix it if we switch to newer version of ruby inside ruby-packer, like 2.5.

Indeed, my point was to change the internal ruby to a new version. Let's make sure we also apply @w00lf 's patches on making the dependencies submodules in vendor/ too.

w00lf commented 4 years ago

As it turns out windows build for ruby packer has more issues then we expected before. In short, ruby executable that get compiled into in-memory fs needs base native extensions like opensll, zlib and others. For linux compilation ruby packer uses the static compilation into a directory and then links them by referencing in ldflgs env variable - https://github.com/metanorma/ruby-packer/blob/82caf52a341d6936f9c37c5a5bd586a7655348f6/lib/compiler.rb#L992, for windows though this gets completely skipped, for example, ncurses https://github.com/metanorma/ruby-packer/blob/82caf52a341d6936f9c37c5a5bd586a7655348f6/lib/compiler.rb#L592, readline: https://github.com/metanorma/ruby-packer/blob/82caf52a341d6936f9c37c5a5bd586a7655348f6/lib/compiler.rb#L612. That leads to a completely broken ruby installation inside in-memory fs. Latest comits in the original repo and in our fork all have these skips, so our only option now if we want to use ruby-packer for windows is to implement linking of openssl, zlib, ncurses and other vendor libraries for windows platform.

ronaldtse commented 4 years ago

our only option now if we want to use ruby-packer for windows is to implement linking of openssl, zlib, ncurses and other vendor libraries for windows platform.

Great find! In this case we will have to implement support of them on the Windows platform.

First of all though we should apply the necessary patches, and make the vendor directory submodules. We should maintain the enclose_io_* and autoupgrade_* files as patches to core Ruby instead.

w00lf commented 4 years ago

@ronaldtse @repu1sion so after some additional debugging i was able to find one of the reasons for load error behavier. Turns out that ruby packer performs some additional modifications to ruby installation itself and core library code. This is the code that broke gems require:

https://github.com/metanorma/ruby-packer/blob/040_clang/ruby/lib/rubygems/path_support.rb#L30

So for windows platform, all load paths will be excluded. I have fixed it in my patch and tried to recompile my little test application(https://github.com/w00lf/ruby-packer-test), but load behavior won't get fixed, now when i try to inspect Gem.path all looks correct but i cannot require any of core gems - application throws 'cannot load file exception'. In the end, i tried to hardcode deps load and try to inspect application from inside with byebug. That's when i have encountered error of "126: The specified module could not be found. - C:/enclose_io_memfs/local/vendor/bundle/ruby/2.4.0/gems/byebug-11.0.1/lib/byebug/byebug.so", i tried to require this module directly but that did not work. So it seems that the problem is not only in modified code but in the wrong windows compilation of ruby itself. Ruby packer repo has file with the description of all modifications performed to the original ruby installation - https://github.com/metanorma/ruby-packer/blob/040_clang/ruby/ruby.patch, we need someone with the knowledge of ruby source code that can inspect this file and identify what we are missing for windows setup, unfortunately, i don't have such deep knowledge and cannot help much here.

ronaldtse commented 4 years ago

@w00lf thanks for the discovery and attempted fix. Let me try to find someone who can get intimate with the hacked code to try fixing this.

ronaldtse commented 4 years ago

From what I see in ruby.patch, it involves four areas:

Inclusion of libsquashfs source
Inclusion of enclose_io* source
Inclusion of libautoupdate source
Hacking of Ruby core to run items 1 to 3.

Right now ruby.patch is >300K which is huge and impossible to go through easily.

Technically items 1-3 can be placed outside of the Ruby source directory, and 4 can be maintained as a separate patch.

w00lf commented 4 years ago

From what I see in ruby.patch, it involves four areas:

Inclusion of libsquashfs source

Inclusion of enclose_io* source

Inclusion of libautoupdate source

Hacking of Ruby core to run items 1 to 3.

Right now ruby.patch is >300K which is huge and impossible to go through easily.

Technically items 1-3 can be placed outside of the Ruby source directory, and 4 can be maintained as a separate patch.

Ruby packer performs modification of ruby's core require method and File class in order for them to work properly with in-memory filesystem. Its not clear why all these modifications do not work in windows case. Ruby.patch file is huge and was not documented at all, it was fully written by ruby packer creator - pmq20 and only he knows all nuances. At first, i thought that these require issues are linked to missing core dependencies(opensll, gdb, yaml) but after encountering this bug with *.so file require i think its actually related to was not finished ruby-core files modifications for windows platform

metanorma / ruby-packer

Compile with Clang on Windows #6