search

davispuh / ruby-libmspack

Ruby wrapper for libmspack. Library for compressing and decompressing Microsoft compression formats, CAB, CHM, HLP, LIT, KWAJ and SZDD.
The Unlicense
3 stars 4 forks source link

libmspack prebuilt binaries #4

Closed maxirmx closed 9 months ago

maxirmx commented 9 months ago

This issue has been spinned off from https://github.com/metanorma/ruby-libmspack/issues/2

Let's see what happens when I do fresh installation of libmspack gem My environment is Ubuntu-20.04; rbenv; Ruby 3.1.4

maxirmx@MSS-WS-N:~/Projects/excavate$ gem uninstall libmspack

Select gem to uninstall:
 1. libmspack-0.10.1.2
 2. libmspack-0.11.0
 3. All versions
> 3
Successfully uninstalled libmspack-0.10.1.2

You have requested to uninstall the gem:
        libmspack-0.11.0

excavate-0.3.5 depends on libmspack (~> 0.1)
excavate-0.3.4 depends on libmspack (~> 0.1)
If you remove this gem, these dependencies will not be met.
Continue with Uninstall? [yN]  y
Successfully uninstalled libmspack-0.11.0
maxirmx@MSS-WS-N:~/Projects/excavate$ gem install libmspack
Fetching libmspack-0.11.0.gem
Building native extensions. This could take a while...
Successfully installed libmspack-0.11.0
Parsing documentation for libmspack-0.11.0
Installing ri documentation for libmspack-0.11.0
Done installing documentation for libmspack after 1 seconds
1 gem installed

There is Building native extensions. This could take a while... message above, it means that some build task is executed.

Now, let's inspect what jhappens during build.

maxirmx@MSS-WS-N:~/Projects/excavate$ more /home/maxirmx/.rbenv/versions/3.1.4/lib/ruby/gems/3.1.0/extensions/x86_64-linux/3.1.0/libmspack-0.11.0/gem_make.out
current directory: /home/maxirmx/.rbenv/versions/3.1.4/lib/ruby/gems/3.1.0/gems/libmspack-0.11.0/ext
/home/maxirmx/.rbenv/versions/3.1.4/bin/ruby -I/home/maxirmx/.rbenv/versions/3.1.4/lib/ruby/3.1.0 -rrubygems /home/maxirmx/.rbenv/versions/3.1.4/lib/ruby/gems/3.1.0/gems/rake-13.1.0/exe/rake RUBYARCHDIR\=/home/maxirmx/.rbenv/versions/3.1.4/lib/ruby/gems
/3.1.0/extensions/x86_64-linux/3.1.0/libmspack-0.11.0 RUBYLIBDIR\=/home/maxirmx/.rbenv/versions/3.1.4/lib/ruby/gems/3.1.0/extensions/x86_64-linux/3.1.0/libmspack-0.11.0
gcc -fexceptions -O -fno-omit-frame-pointer -fno-strict-aliasing -fPIC -I./libmspack/mspack/ -D_FILE_OFFSET_BITS\=64 -DHAVE_MEMCMP\=1 -DHAVE_TOWLOWER\=1 -DHAVE_TOLOWER\=1 -DHAVE_FSEEKO\=1 -DHAVE_CTYPE_H\=1 -DHAVE_LIMITS_H\=1 -DHAVE_STDLIB_H\=1 -DHAVE_ST
RING_H\=1 -DHAVE_WCTYPE_H\=1 -o x86_64-linux/cabc.o -c ./libmspack/mspack//cabc.c
gcc -fexceptions -O -fno-omit-frame-pointer -fno-strict-aliasing -fPIC -I./libmspack/mspack/ -D_FILE_OFFSET_BITS\=64 -DHAVE_MEMCMP\=1 -DHAVE_TOWLOWER\=1 -DHAVE_TOLOWER\=1 -DHAVE_FSEEKO\=1 -DHAVE_CTYPE_H\=1 -DHAVE_LIMITS_H\=1 -DHAVE_STDLIB_H\=1 -DHAVE_ST
RING_H\=1 -DHAVE_WCTYPE_H\=1 -o x86_64-linux/cabd.o -c ./libmspack/mspack//cabd.c
gcc -fexceptions -O -fno-omit-frame-pointer -fno-strict-aliasing -fPIC -I./libmspack/mspack/ -D_FILE_OFFSET_BITS\=64 -DHAVE_MEMCMP\=1 -DHAVE_TOWLOWER\=1 -DHAVE_TOLOWER\=1 -DHAVE_FSEEKO\=1 -DHAVE_CTYPE_H\=1 -DHAVE_LIMITS_H\=1 -DHAVE_STDLIB_H\=1 -DHAVE_ST
RING_H\=1 -DHAVE_WCTYPE_H\=1 -o x86_64-linux/chmc.o -c ./libmspack/mspack//chmc.c
gcc -fexceptions -O -fno-omit-frame-pointer -fno-strict-aliasing -fPIC -I./libmspack/mspack/ -D_FILE_OFFSET_BITS\=64 -DHAVE_MEMCMP\=1 -DHAVE_TOWLOWER\=1 -DHAVE_TOLOWER\=1 -DHAVE_FSEEKO\=1 -DHAVE_CTYPE_H\=1 -DHAVE_LIMITS_H\=1 -DHAVE_STDLIB_H\=1 -DHAVE_ST
RING_H\=1 -DHAVE_WCTYPE_H\=1 -o x86_64-linux/chmd.o -c ./libmspack/mspack//chmd.c
gcc -fexceptions -O -fno-omit-frame-pointer -fno-strict-aliasing -fPIC -I./libmspack/mspack/ -D_FILE_OFFSET_BITS\=64 -DHAVE_MEMCMP\=1 -DHAVE_TOWLOWER\=1 -DHAVE_TOLOWER\=1 -DHAVE_FSEEKO\=1 -DHAVE_CTYPE_H\=1 -DHAVE_LIMITS_H\=1 -DHAVE_STDLIB_H\=1 -DHAVE_ST
RING_H\=1 -DHAVE_WCTYPE_H\=1 -o x86_64-linux/crc32.o -c ./libmspack/mspack//crc32.c
gcc -fexceptions -O -fno-omit-frame-pointer -fno-strict-aliasing -fPIC -I./libmspack/mspack/ -D_FILE_OFFSET_BITS\=64 -DHAVE_MEMCMP\=1 -DHAVE_TOWLOWER\=1 -DHAVE_TOLOWER\=1 -DHAVE_FSEEKO\=1 -DHAVE_CTYPE_H\=1 -DHAVE_LIMITS_H\=1 -DHAVE_STDLIB_H\=1 -DHAVE_ST
RING_H\=1 -DHAVE_WCTYPE_H\=1 -o x86_64-linux/hlpc.o -c ./libmspack/mspack//hlpc.c
gcc -fexceptions -O -fno-omit-frame-pointer -fno-strict-aliasing -fPIC -I./libmspack/mspack/ -D_FILE_OFFSET_BITS\=64 -DHAVE_MEMCMP\=1 -DHAVE_TOWLOWER\=1 -DHAVE_TOLOWER\=1 -DHAVE_FSEEKO\=1 -DHAVE_CTYPE_H\=1 -DHAVE_LIMITS_H\=1 -DHAVE_STDLIB_H\=1 -DHAVE_ST
RING_H\=1 -DHAVE_WCTYPE_H\=1 -o x86_64-linux/hlpd.o -c ./libmspack/mspack//hlpd.c
gcc -fexceptions -O -fno-omit-frame-pointer -fno-strict-aliasing -fPIC -I./libmspack/mspack/ -D_FILE_OFFSET_BITS\=64 -DHAVE_MEMCMP\=1 -DHAVE_TOWLOWER\=1 -DHAVE_TOLOWER\=1 -DHAVE_FSEEKO\=1 -DHAVE_CTYPE_H\=1 -DHAVE_LIMITS_H\=1 -DHAVE_STDLIB_H\=1 -DHAVE_ST
RING_H\=1 -DHAVE_WCTYPE_H\=1 -o x86_64-linux/kwajc.o -c ./libmspack/mspack//kwajc.c
gcc -fexceptions -O -fno-omit-frame-pointer -fno-strict-aliasing -fPIC -I./libmspack/mspack/ -D_FILE_OFFSET_BITS\=64 -DHAVE_MEMCMP\=1 -DHAVE_TOWLOWER\=1 -DHAVE_TOLOWER\=1 -DHAVE_FSEEKO\=1 -DHAVE_CTYPE_H\=1 -DHAVE_LIMITS_H\=1 -DHAVE_STDLIB_H\=1 -DHAVE_ST
RING_H\=1 -DHAVE_WCTYPE_H\=1 -o x86_64-linux/kwajd.o -c ./libmspack/mspack//kwajd.c
gcc -fexceptions -O -fno-omit-frame-pointer -fno-strict-aliasing -fPIC -I./libmspack/mspack/ -D_FILE_OFFSET_BITS\=64 -DHAVE_MEMCMP\=1 -DHAVE_TOWLOWER\=1 -DHAVE_TOLOWER\=1 -DHAVE_FSEEKO\=1 -DHAVE_CTYPE_H\=1 -DHAVE_LIMITS_H\=1 -DHAVE_STDLIB_H\=1 -DHAVE_ST
RING_H\=1 -DHAVE_WCTYPE_H\=1 -o x86_64-linux/litc.o -c ./libmspack/mspack//litc.c
gcc -fexceptions -O -fno-omit-frame-pointer -fno-strict-aliasing -fPIC -I./libmspack/mspack/ -D_FILE_OFFSET_BITS\=64 -DHAVE_MEMCMP\=1 -DHAVE_TOWLOWER\=1 -DHAVE_TOLOWER\=1 -DHAVE_FSEEKO\=1 -DHAVE_CTYPE_H\=1 -DHAVE_LIMITS_H\=1 -DHAVE_STDLIB_H\=1 -DHAVE_ST
RING_H\=1 -DHAVE_WCTYPE_H\=1 -o x86_64-linux/litd.o -c ./libmspack/mspack//litd.c
gcc -fexceptions -O -fno-omit-frame-pointer -fno-strict-aliasing -fPIC -I./libmspack/mspack/ -D_FILE_OFFSET_BITS\=64 -DHAVE_MEMCMP\=1 -DHAVE_TOWLOWER\=1 -DHAVE_TOLOWER\=1 -DHAVE_FSEEKO\=1 -DHAVE_CTYPE_H\=1 -DHAVE_LIMITS_H\=1 -DHAVE_STDLIB_H\=1 -DHAVE_ST
RING_H\=1 -DHAVE_WCTYPE_H\=1 -o x86_64-linux/lzssd.o -c ./libmspack/mspack//lzssd.c
gcc -fexceptions -O -fno-omit-frame-pointer -fno-strict-aliasing -fPIC -I./libmspack/mspack/ -D_FILE_OFFSET_BITS\=64 -DHAVE_MEMCMP\=1 -DHAVE_TOWLOWER\=1 -DHAVE_TOLOWER\=1 -DHAVE_FSEEKO\=1 -DHAVE_CTYPE_H\=1 -DHAVE_LIMITS_H\=1 -DHAVE_STDLIB_H\=1 -DHAVE_ST
RING_H\=1 -DHAVE_WCTYPE_H\=1 -o x86_64-linux/lzxc.o -c ./libmspack/mspack//lzxc.c
gcc -fexceptions -O -fno-omit-frame-pointer -fno-strict-aliasing -fPIC -I./libmspack/mspack/ -D_FILE_OFFSET_BITS\=64 -DHAVE_MEMCMP\=1 -DHAVE_TOWLOWER\=1 -DHAVE_TOLOWER\=1 -DHAVE_FSEEKO\=1 -DHAVE_CTYPE_H\=1 -DHAVE_LIMITS_H\=1 -DHAVE_STDLIB_H\=1 -DHAVE_ST
RING_H\=1 -DHAVE_WCTYPE_H\=1 -o x86_64-linux/lzxd.o -c ./libmspack/mspack//lzxd.c
gcc -fexceptions -O -fno-omit-frame-pointer -fno-strict-aliasing -fPIC -I./libmspack/mspack/ -D_FILE_OFFSET_BITS\=64 -DHAVE_MEMCMP\=1 -DHAVE_TOWLOWER\=1 -DHAVE_TOLOWER\=1 -DHAVE_FSEEKO\=1 -DHAVE_CTYPE_H\=1 -DHAVE_LIMITS_H\=1 -DHAVE_STDLIB_H\=1 -DHAVE_ST
RING_H\=1 -DHAVE_WCTYPE_H\=1 -o x86_64-linux/mszipc.o -c ./libmspack/mspack//mszipc.c
gcc -fexceptions -O -fno-omit-frame-pointer -fno-strict-aliasing -fPIC -I./libmspack/mspack/ -D_FILE_OFFSET_BITS\=64 -DHAVE_MEMCMP\=1 -DHAVE_TOWLOWER\=1 -DHAVE_TOLOWER\=1 -DHAVE_FSEEKO\=1 -DHAVE_CTYPE_H\=1 -DHAVE_LIMITS_H\=1 -DHAVE_STDLIB_H\=1 -DHAVE_ST
RING_H\=1 -DHAVE_WCTYPE_H\=1 -o x86_64-linux/mszipd.o -c ./libmspack/mspack//mszipd.c
gcc -fexceptions -O -fno-omit-frame-pointer -fno-strict-aliasing -fPIC -I./libmspack/mspack/ -D_FILE_OFFSET_BITS\=64 -DHAVE_MEMCMP\=1 -DHAVE_TOWLOWER\=1 -DHAVE_TOLOWER\=1 -DHAVE_FSEEKO\=1 -DHAVE_CTYPE_H\=1 -DHAVE_LIMITS_H\=1 -DHAVE_STDLIB_H\=1 -DHAVE_ST
RING_H\=1 -DHAVE_WCTYPE_H\=1 -o x86_64-linux/oabc.o -c ./libmspack/mspack//oabc.c
gcc -fexceptions -O -fno-omit-frame-pointer -fno-strict-aliasing -fPIC -I./libmspack/mspack/ -D_FILE_OFFSET_BITS\=64 -DHAVE_MEMCMP\=1 -DHAVE_TOWLOWER\=1 -DHAVE_TOLOWER\=1 -DHAVE_FSEEKO\=1 -DHAVE_CTYPE_H\=1 -DHAVE_LIMITS_H\=1 -DHAVE_STDLIB_H\=1 -DHAVE_ST
RING_H\=1 -DHAVE_WCTYPE_H\=1 -o x86_64-linux/oabd.o -c ./libmspack/mspack//oabd.c
gcc -fexceptions -O -fno-omit-frame-pointer -fno-strict-aliasing -fPIC -I./libmspack/mspack/ -D_FILE_OFFSET_BITS\=64 -DHAVE_MEMCMP\=1 -DHAVE_TOWLOWER\=1 -DHAVE_TOLOWER\=1 -DHAVE_FSEEKO\=1 -DHAVE_CTYPE_H\=1 -DHAVE_LIMITS_H\=1 -DHAVE_STDLIB_H\=1 -DHAVE_ST
RING_H\=1 -DHAVE_WCTYPE_H\=1 -o x86_64-linux/qtmd.o -c ./libmspack/mspack//qtmd.c
gcc -fexceptions -O -fno-omit-frame-pointer -fno-strict-aliasing -fPIC -I./libmspack/mspack/ -D_FILE_OFFSET_BITS\=64 -DHAVE_MEMCMP\=1 -DHAVE_TOWLOWER\=1 -DHAVE_TOLOWER\=1 -DHAVE_FSEEKO\=1 -DHAVE_CTYPE_H\=1 -DHAVE_LIMITS_H\=1 -DHAVE_STDLIB_H\=1 -DHAVE_ST
RING_H\=1 -DHAVE_WCTYPE_H\=1 -o x86_64-linux/system.o -c ./libmspack/mspack//system.c
gcc -fexceptions -O -fno-omit-frame-pointer -fno-strict-aliasing -fPIC -I./libmspack/mspack/ -D_FILE_OFFSET_BITS\=64 -DHAVE_MEMCMP\=1 -DHAVE_TOWLOWER\=1 -DHAVE_TOLOWER\=1 -DHAVE_FSEEKO\=1 -DHAVE_CTYPE_H\=1 -DHAVE_LIMITS_H\=1 -DHAVE_STDLIB_H\=1 -DHAVE_ST
RING_H\=1 -DHAVE_WCTYPE_H\=1 -o x86_64-linux/szddc.o -c ./libmspack/mspack//szddc.c
gcc -fexceptions -O -fno-omit-frame-pointer -fno-strict-aliasing -fPIC -I./libmspack/mspack/ -D_FILE_OFFSET_BITS\=64 -DHAVE_MEMCMP\=1 -DHAVE_TOWLOWER\=1 -DHAVE_TOLOWER\=1 -DHAVE_FSEEKO\=1 -DHAVE_CTYPE_H\=1 -DHAVE_LIMITS_H\=1 -DHAVE_STDLIB_H\=1 -DHAVE_ST
RING_H\=1 -DHAVE_WCTYPE_H\=1 -o x86_64-linux/szddd.o -c ./libmspack/mspack//szddd.c
gcc -shared -o x86_64-linux/libmspack.so x86_64-linux/cabc.o x86_64-linux/cabd.o x86_64-linux/chmc.o x86_64-linux/chmd.o x86_64-linux/crc32.o x86_64-linux/hlpc.o x86_64-linux/hlpd.o x86_64-linux/kwajc.o x86_64-linux/kwajd.o x86_64-linux/litc.o x86_64-li
nux/litd.o x86_64-linux/lzssd.o x86_64-linux/lzxc.o x86_64-linux/lzxd.o x86_64-linux/mszipc.o x86_64-linux/mszipd.o x86_64-linux/oabc.o x86_64-linux/oabd.o x86_64-linux/qtmd.o x86_64-linux/system.o x86_64-linux/szddc.o x86_64-linux/szddd.o -fexceptions

You can see full rebuild

maxirmx commented 9 months ago

IMHO you cannot just include some precompiled files to your gem and expect that Ruby build system would not recompile them. However, there are two approaches that work: 1) it is possible to have rakefile to "look around" and reuse binary files if rake logic decides that metioned binary files are suitable Example: seven_zip_ruby (https://rubygems.org/gems/seven-zip)
This approach works but I would call it very tricky 2) it is possible to implement canonical Ruby extension approach and have binary packages for some or all target environments Example (if you need it at all) https://rubygems.org/gems/emf2svg image

maxirmx commented 9 months ago

I would be happy to suggest a PR but do not want to suggest radical changes to the make system of your gem without prior discussion Thank you

davispuh commented 9 months ago

Actually it used to work fine in past. Anyway I just released ffi-compiler v2.3.0 so now it won't recompile. If you want to force rebuild (instead of using prebuilt) then pass --build-all flag like this gem install libmspack -- --build-all. You can see implementation here davispuh/ffi-compiler@592f1ba64125cabccb8d2059bdd178ecba86c611 and it's quite simple.

Regarding those other approaches:

  1. Idea is correct but implementation is overly complicated for no good reason. Also if you want it to be more robust you actually would need to compare ABIs but I haven't seen anyone do that.
  2. Makes sense if binaries are big but otherwise seems overkill since can bundle all in single gem
maxirmx commented 9 months ago

Thank you. It still recompiles but it is good. Otherwise x86_64-linux library would be used in musl environment and it probably won't work there.

davispuh commented 9 months ago

It shouldn't recompile but it's expected to still see Building native extensions. This could take a while... since ffi-compiler is invoked unconditionally.

I wonder how your libmspack-0.11.0/ext dir looks like