search

kwilczynski / ruby-magic

Simple interface to libmagic for Ruby Programming Language
Apache License 2.0
27 stars 8 forks source link

Extension compilation fails on Mac +RVM because of `sed` command failure #14

Closed pluff closed 2 years ago

pluff commented 3 years ago

While building native extensions ruby-magic fails while compiling docs.

Failing command:

Making all in doc
sed -e s@__CSECTION__@1@g \
        -e s@__FSECTION__@5@g \
        -e s@__VERSION__@5.39@g \
        -e s@__MAGIC__@/Users/shutin/.rvm/gems/ruby-2.7.2@gitlab-ee/gems/ruby-magic-0.3.2/ports/x86_64-apple-darwin19.6.0/libmagic/5.39/share/misc/magic@g ./file.man > file.1
sed: 1: "s@__MAGIC__@/Users/shut ...": bad flag in substitute command: 't'

Failure reason:

MAGIC path contains @ sign in the path and sed thinks it's a delimiter for regexp. RVM uses @ sign for marking gemsets

Possible solutions

  1. Escape path @ sign before passing to sed expression
  2. Use different delimiter e.g. |

Full log output:

gem install ruby-magic

Building native extensions. This could take a while...
ERROR:  Error installing ruby-magic:
    ERROR: Failed to build gem native extension.

    current directory: /Users/shutin/.rvm/gems/ruby-2.7.2@gitlab-ee/gems/ruby-magic-0.3.2/ext/magic
/Users/shutin/.rvm/rubies/ruby-2.7.2/bin/ruby -I /Users/shutin/.rvm/rubies/ruby-2.7.2/lib/ruby/2.7.0 -r ./siteconf20210405-20897-znlqwm.rb extconf.rb
Building ruby-magic using packaged libraries.
Static linking is enabled.
Cross build is disabled.
Using mini_portile version 2.5.0
Downloading file-5.39.tar.gz (100%)
Extracting file-5.39.tar.gz into tmp/x86_64-apple-darwin19.6.0/ports/libmagic/5.39... OK
Running 'configure' for libmagic 5.39... OK
Running 'compile' for libmagic 5.39... ERROR, review '/Users/shutin/.rvm/gems/ruby-2.7.2@gitlab-ee/gems/ruby-magic-0.3.2/ext/magic/tmp/x86_64-apple-darwin19.6.0/ports/libmagic/5.39/compile.log' to see what happened. Last lines are:
========================================================================
libtool: compile:  gcc -DHAVE_CONFIG_H -I. -I.. -DMAGIC=\"/Users/shutin/.rvm/gems/ruby-2.7.2@gitlab-ee/gems/ruby-magic-0.3.2/ports/x86_64-apple-darwin19.6.0/libmagic/5.39/share/misc/magic\" -fvisibility=hidden -Wall -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wmissing-declarations -Wredundant-decls -Wnested-externs -Wsign-compare -Wreturn-type -Wswitch -Wshadow -Wcast-qual -Wwrite-strings -Wextra -Wunused-parameter -Wformat=2 -fPIC -c readcdf.c  -fno-common -DPIC -o readcdf.o
/bin/sh ../libtool  --tag=CC   --mode=link gcc -fvisibility=hidden -Wall -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith        -Wmissing-declarations -Wredundant-decls -Wnested-externs        -Wsign-compare -Wreturn-type -Wswitch -Wshadow        -Wcast-qual -Wwrite-strings -Wextra -Wunused-parameter -Wformat=2 -fPIC -no-undefined -version-info 1:0:0  -o libmagic.la -rpath /Users/shutin/.rvm/gems/ruby-2.7.2@gitlab-ee/gems/ruby-magic-0.3.2/ports/x86_64-apple-darwin19.6.0/libmagic/5.39/lib buffer.lo magic.lo apprentice.lo softmagic.lo ascmagic.lo encoding.lo compress.lo is_csv.lo is_json.lo is_tar.lo readelf.lo print.lo fsmagic.lo funcs.lo apptype.lo der.lo cdf.lo cdf_time.lo readcdf.lo   -llzma -lbz2 -lz
libtool: link: ar cru .libs/libmagic.a  buffer.o magic.o apprentice.o softmagic.o ascmagic.o encoding.o compress.o is_csv.o is_json.o is_tar.o readelf.o print.o fsmagic.o funcs.o apptype.o der.o cdf.o cdf_time.o readcdf.o
libtool: link: ranlib .libs/libmagic.a
libtool: link: ( cd ".libs" && rm -f "libmagic.la" && ln -s "../libmagic.la" "libmagic.la" )
/bin/sh ../libtool  --tag=CC   --mode=link gcc -fvisibility=hidden -Wall -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith        -Wmissing-declarations -Wredundant-decls -Wnested-externs        -Wsign-compare -Wreturn-type -Wswitch -Wshadow        -Wcast-qual -Wwrite-strings -Wextra -Wunused-parameter -Wformat=2 -fPIC   -o file file.o seccomp.o libmagic.la -llzma -lbz2 -lz
libtool: link: gcc -fvisibility=hidden -Wall -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wmissing-declarations -Wredundant-decls -Wnested-externs -Wsign-compare -Wreturn-type -Wswitch -Wshadow -Wcast-qual -Wwrite-strings -Wextra -Wunused-parameter -Wformat=2 -fPIC -o file file.o seccomp.o  ./.libs/libmagic.a -llzma -lbz2 -lz
Making all in magic
../src/file -C -m magic
Making all in tests
make[2]: Nothing to be done for `all'.
Making all in doc
sed -e s@__CSECTION__@1@g \
        -e s@__FSECTION__@5@g \
        -e s@__VERSION__@5.39@g \
        -e s@__MAGIC__@/Users/shutin/.rvm/gems/ruby-2.7.2@gitlab-ee/gems/ruby-magic-0.3.2/ports/x86_64-apple-darwin19.6.0/libmagic/5.39/share/misc/magic@g ./file.man > file.1
sed: 1: "s@__MAGIC__@/Users/shut ...": bad flag in substitute command: 't'
make[2]: *** [file.1] Error 1
make[1]: *** [all-recursive] Error 1
make: *** [all] Error 2
========================================================================
*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of necessary
libraries and/or headers.  Check the mkmf.log file for more details.  You may
need configuration options.

Provided configuration options:
    --with-opt-dir
    --with-opt-include
    --without-opt-include=${opt-dir}/include
    --with-opt-lib
    --without-opt-lib=${opt-dir}/lib
    --with-make-prog
    --without-make-prog
    --srcdir=.
    --curdir
    --ruby=/Users/shutin/.rvm/rubies/ruby-2.7.2/bin/$(RUBY_BASE_NAME)
    --help
    --clean
    --enable-system-libraries
    --disable-system-libraries
    --use-system-libraries
    --enable-static
    --disable-static
    --enable-cross-build
    --disable-cross-build
/Users/shutin/.rvm/gems/ruby-2.7.2@gitlab-ee/gems/mini_portile2-2.5.0/lib/mini_portile2/mini_portile.rb:405:in `block in execute': Failed to complete compile task (RuntimeError)
    from /Users/shutin/.rvm/gems/ruby-2.7.2@gitlab-ee/gems/mini_portile2-2.5.0/lib/mini_portile2/mini_portile.rb:376:in `chdir'
    from /Users/shutin/.rvm/gems/ruby-2.7.2@gitlab-ee/gems/mini_portile2-2.5.0/lib/mini_portile2/mini_portile.rb:376:in `execute'
    from /Users/shutin/.rvm/gems/ruby-2.7.2@gitlab-ee/gems/mini_portile2-2.5.0/lib/mini_portile2/mini_portile.rb:115:in `compile'
    from /Users/shutin/.rvm/gems/ruby-2.7.2@gitlab-ee/gems/mini_portile2-2.5.0/lib/mini_portile2/mini_portile.rb:154:in `cook'
    from extconf.rb:124:in `block in process_recipe'
    from extconf.rb:79:in `tap'
    from extconf.rb:79:in `process_recipe'
    from extconf.rb:237:in `<main>'

extconf failed, exit code 1
kwilczynski commented 3 years ago

Hi @pluff, thank you for letting us know, and sorry for the issues!

Good catch! And thank you for detailed issue report! Much appreciated.

I need to look into this, especially how to sensibly fix this, since this is within the libmagic we rely on. Aside of the list of suggestions you provided (thank you!), perhaps there is a way to pass a flag to autotools so that the manual pages et al are simply not built - hardly useful in a case of trying to install and use a Ruby Gem.

In the meantime, perhaps this would unblock you (using Homebrew as an example*):

$ brew install libmagic
$ gem install ruby-magic -- --use-system-libraries

* MacPorts and other similar solutions would also work fine to install libmagic.

This would build ruby-magic without using the vendored libmagic and simply link against one that exist in your runtime environment. I hope this would help in the interim.

Again, sorry for issues!

I am adding @zoulasc for visibility as perhaps Christos would like us to open a bug report against libmagic.

Krzysztof

pluff commented 3 years ago

@kwilczynski thanks for quick reply! Temporary workaround with separate libmagic installation works 👍

kwilczynski commented 3 years ago

Hi @pluff,

@kwilczynski thanks for quick reply! Temporary workaround with separate libmagic installation works 👍

I am glad it worked fine!

Krzysztof

kwilczynski commented 3 years ago

A simple yet crude solution to the problem here would be to not build tests, documentation and the Python bindings. We could add a simple patch to be applied to the vendored libmagic at build time, for example:

diff --git a/Makefile.am b/Makefile.am
index 2ab67ed..c7fafeb 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -2,7 +2,7 @@ ACLOCAL_AMFLAGS = -I m4

 EXTRA_DIST = MAINT

-SUBDIRS = src magic tests doc python
+SUBDIRS = src magic

 # This variable must have 'exec' in its name, in order to be installed
 # by 'install-exec' target (instead of default 'install-data')
diff --git a/Makefile.in b/Makefile.in
index e19b15e..dcb5ce6 100644
--- a/Makefile.in
+++ b/Makefile.in
@@ -358,7 +358,7 @@ top_builddir = @top_builddir@
 top_srcdir = @top_srcdir@
 ACLOCAL_AMFLAGS = -I m4
 EXTRA_DIST = MAINT
-SUBDIRS = src magic tests doc python
+SUBDIRS = src magic

 # This variable must have 'exec' in its name, in order to be installed
 # by 'install-exec' target (instead of default 'install-data')

Technically, we don't need to run tests or build anything else aside, thus removing the extra sub-directories might not be a bad idea in the end.

kwilczynski commented 3 years ago

Hi @pluff,

I added two changes:

That I believe would solve this issue in a simple manner - at the moment libmagic does not offer an option (which would be useful) to opt-out from building tests, documentation (manual pages) and Python bindings (not something we would use) that one could pass to autotools, perhaps in the future, provided that @zoulasc would be OK with this.

I tested this locally, and it seem to be doing the right thing, so to speak.

Again, apologies for the troubles!

Krzysztof

stanhu commented 3 years ago

@kwilczynski Thanks for the fixes! This should be fixed with v0.4.0.

pluff commented 2 years ago

@stanhu @kwilczynski this issue is back with ruby-magic-0.5.3

Building ruby-magic using packaged libraries.
Static linking is enabled.
Cross build is disabled.
Using mini_portile version 2.6.1
---------- IMPORTANT NOTICE ----------
Building Ruby Magic with a packaged version of libmagic-5.39.
Configuration options: --disable-silent-rules --disable-dependency-tracking --enable-fsect-man5 --disable-shared --enable-static CFLAGS\=-fPIC
Downloading file-5.39.tar.gz ( 10%)
Downloading file-5.39.tar.gz ( 22%)
Downloading file-5.39.tar.gz ( 34%)
Downloading file-5.39.tar.gz ( 44%)
Downloading file-5.39.tar.gz ( 56%)
Downloading file-5.39.tar.gz ( 68%)
Downloading file-5.39.tar.gz ( 78%)
Downloading file-5.39.tar.gz ( 90%)
Downloading file-5.39.tar.gz (100%)
Extracting file-5.39.tar.gz into tmp/x86_64-apple-darwin21/ports/libmagic/5.39... OK
Running 'configure' for libmagic 5.39... OK
Running 'compile' for libmagic 5.39... ERROR, review '/Users/shutin/.rvm/gems/ruby-2.7.5@gitlab-ee/gems/ruby-magic-0.5.3/ext/magic/tmp/x86_64-apple-darwin21/ports/libmagic/5.39/compile.log' to see what happened. Last lines are:
========================================================================
libtool: compile:  gcc -DHAVE_CONFIG_H -I. -I.. -DMAGIC=\"/Users/shutin/.rvm/gems/ruby-2.7.5@gitlab-ee/gems/ruby-magic-0.5.3/ports/x86_64-apple-darwin21/libmagic/5.39/share/misc/magic\" -fvisibility=hidden -Wall -Wstrict-prototypes -Wmissing-prototypes
-Wpointer-arith -Wmissing-declarations -Wredundant-decls -Wnested-externs -Wsign-compare -Wreturn-type -Wswitch -Wshadow -Wcast-qual -Wwrite-strings -Wextra -Wunused-parameter -Wformat=2 -fPIC -c readcdf.c  -fno-common -DPIC -o readcdf.o
/bin/sh ../libtool  --tag=CC   --mode=link gcc -fvisibility=hidden -Wall -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith        -Wmissing-declarations -Wredundant-decls -Wnested-externs        -Wsign-compare -Wreturn-type -Wswitch -Wshadow
-Wcast-qual -Wwrite-strings -Wextra -Wunused-parameter -Wformat=2 -fPIC -no-undefined -version-info 1:0:0  -o libmagic.la -rpath /Users/shutin/.rvm/gems/ruby-2.7.5@gitlab-ee/gems/ruby-magic-0.5.3/ports/x86_64-apple-darwin21/libmagic/5.39/lib buffer.lo
magic.lo apprentice.lo softmagic.lo ascmagic.lo encoding.lo compress.lo is_csv.lo is_json.lo is_tar.lo readelf.lo print.lo fsmagic.lo funcs.lo apptype.lo der.lo cdf.lo cdf_time.lo readcdf.lo   -llzma -lbz2 -lz
libtool: link: ar cru .libs/libmagic.a  buffer.o magic.o apprentice.o softmagic.o ascmagic.o encoding.o compress.o is_csv.o is_json.o is_tar.o readelf.o print.o fsmagic.o funcs.o apptype.o der.o cdf.o cdf_time.o readcdf.o
libtool: link: ranlib .libs/libmagic.a
libtool: link: ( cd ".libs" && rm -f "libmagic.la" && ln -s "../libmagic.la" "libmagic.la" )
/bin/sh ../libtool  --tag=CC   --mode=link gcc -fvisibility=hidden -Wall -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith        -Wmissing-declarations -Wredundant-decls -Wnested-externs        -Wsign-compare -Wreturn-type -Wswitch -Wshadow
-Wcast-qual -Wwrite-strings -Wextra -Wunused-parameter -Wformat=2 -fPIC   -o file file.o seccomp.o libmagic.la -llzma -lbz2 -lz
libtool: link: gcc -fvisibility=hidden -Wall -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wmissing-declarations -Wredundant-decls -Wnested-externs -Wsign-compare -Wreturn-type -Wswitch -Wshadow -Wcast-qual -Wwrite-strings -Wextra
-Wunused-parameter -Wformat=2 -fPIC -o file file.o seccomp.o  ./.libs/libmagic.a -llzma -lbz2 -lz
Making all in magic
../src/file -C -m magic
Making all in tests
make[2]: Nothing to be done for `all'.
Making all in doc
sed -e s@__CSECTION__@1@g \
        -e s@__FSECTION__@5@g \
        -e s@__VERSION__@5.39@g \
        -e s@__MAGIC__@/Users/shutin/.rvm/gems/ruby-2.7.5@gitlab-ee/gems/ruby-magic-0.5.3/ports/x86_64-apple-darwin21/libmagic/5.39/share/misc/magic@g ./file.man > file.1
sed: 1: "s@__MAGIC__@/Users/shut ...": bad flag in substitute command: 't'
make[2]: *** [file.1] Error 1
make[1]: *** [all-recursive] Error 1
make: *** [all] Error 2
========================================================================
*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of necessary
libraries and/or headers.  Check the mkmf.log file for more details.  You may
need configuration options.

Provided configuration options:
    --with-opt-dir
    --with-opt-include
    --without-opt-include=${opt-dir}/include
    --with-opt-lib
    --without-opt-lib=${opt-dir}/lib
    --with-make-prog
    --without-make-prog
    --srcdir=.
    --curdir
    --ruby=/Users/shutin/.rvm/rubies/ruby-2.7.5/bin/$(RUBY_BASE_NAME)
    --help
    --clean
    --enable-system-libraries
    --disable-system-libraries
    --use-system-libraries
    --enable-static
    --disable-static
    --enable-cross-build
    --disable-cross-build
/Users/shutin/.rvm/gems/ruby-2.7.5@gitlab-ee/gems/mini_portile2-2.6.1/lib/mini_portile2/mini_portile.rb:443:in `block in execute': Failed to complete compile task (RuntimeError)
    from /Users/shutin/.rvm/gems/ruby-2.7.5@gitlab-ee/gems/mini_portile2-2.6.1/lib/mini_portile2/mini_portile.rb:409:in `chdir'
    from /Users/shutin/.rvm/gems/ruby-2.7.5@gitlab-ee/gems/mini_portile2-2.6.1/lib/mini_portile2/mini_portile.rb:409:in `execute'
    from /Users/shutin/.rvm/gems/ruby-2.7.5@gitlab-ee/gems/mini_portile2-2.6.1/lib/mini_portile2/mini_portile.rb:137:in `compile'
    from /Users/shutin/.rvm/gems/ruby-2.7.5@gitlab-ee/gems/mini_portile2-2.6.1/lib/mini_portile2/mini_portile.rb:180:in `cook'
    from extconf.rb:163:in `block in process_recipe'
    from extconf.rb:76:in `tap'
    from extconf.rb:76:in `process_recipe'
    from extconf.rb:287:in `<main>'

extconf failed, exit code 1

This workaround still works so just for you to be aware of the problem

stanhu commented 2 years ago

Somehow the doc and tests folders are still being built. I'm not seeing the patch being applied. This is what I see in my bundle exec rake compile output:

---------- IMPORTANT NOTICE ----------
Building Ruby Magic with a packaged version of libmagic-5.39.
Configuration options: --disable-silent-rules --disable-dependency-tracking --enable-fsect-man5 --disable-shared --enable-static CFLAGS\=-fPIC
The following patches are being applied:
  - 0001-Don-t-attempt-to-build-tests-documentation-and-Python-bindings.patch

@pluff Would you be able to check your /Users/shutin/.rvm/gems/ruby-2.7.5@gitlab-ee/gems/ruby-magic-0.5.3/patches/libmagic directory to see if there are any files? I wonder if PACKAGE_ROOT_DIR is not right because of rvm.

MAGIC path contains @ sign in the path and sed thinks it's a delimiter for regexp. RVM uses @ sign for marking gemsets

I'm not sure I follow. Is rvm intercepting the sed call here, or is this a problem with BSD sed vs GNU sed? I think I'm using BSD sed and a local compile of libmagic works fine with this.

kwilczynski commented 2 years ago

Hi @pluff,

Thank you for re-opening this and sorry for troubles!

Let me check if the patch gets applied correctly and, if not, then I will devise a different fix. Bear with me!

@stanhu thank you for diving into this! Much appreciated!

Krzysztof

kwilczynski commented 2 years ago

Hi @pluff,

Sorry for the late reply!

After having a closer look, I noticed that one of our Rake tasks, namely the build task coming from Bundler (see: bundler/lib/bundler/gem_helper.rb#L86-L95)) does not properly handle the additional dependencies that are otherwise needed when installing the Ruby Gem with an option to use bundled libmagic over the system-wide installation. In other words, the patches and other artefacts haven't been included as part of the Ruby Gem itself, thus when installing, the _miniportile2 driven installation has nothing to apply, as no patches are there to be applied.

The Bundler task invoked after gem build or bundle exec gem build simply wraps around the gem command, which in turn relies on the information provided in the gemspec file, and since we don't have any notion of bundling (or including, if you wish) build artefacts there (the gemspec file that is), thus nothing has been included as the side effects of this are twofold:

For a few releases since the version 0.4.0 we've been having this issue. You can see the difference in the size of the resulting Ruby Gem as per:

Screenshot from 2022-01-20 23-42-43

This is definitely not on purpose and I sincerely apologise (and THANK YOU, by the way, for letting us know) that we haven't noticed this to be a problem in time. The increase in traffic to the libmagic source code stored in an S3 compatible bucket, where we mirror the source code, was also a dead giveaway, as per:

Screenshot from 2022-01-21 01-39-28

Now, I will try to work with kind people behind Rubygems to see if we could, perhaps, replace our already released Ruby Gems with a proper build without having to release a completely new version, especially since GitLab has been released recently and it would be rather troublesome to deal with a point release, etc.

Krzysztof

stanhu commented 2 years ago

Oh, sorry about that. I doubt we can simply replace the files. It should be easy enough to update the version GitLab is using.

kwilczynski commented 2 years ago

Hi @stanhu,

Oh, sorry about that. I doubt we can simply replace the files. It should be easy enough to update the version GitLab is using.

No worries! I should have noticed that something was afoot with the Gem size. I also need to think of testing everything under RVM in the future, plus I will see how to address this properly on the libmagic side so that we can disable documentation and Python bindings from being built using an appropriate switch passed to autotools. The same goes for building and running tests (which we also disable in the patch, perhaps unnecessarily).

Re: GitLab - do you think this is doable? Otherwise, folks using RVM (which seems to be primarily affected, especially when someone uses gemsets there) would have to apply the workaround (install using a system-wide version of libmagic) or update their Gemfile. A good thing is that the binary packages now available in a few mainstream distributions aren't affected by this - at least not that I could see.

I am asking as I am not sure how turbulent/distracting GitLab releases might be (even a minor patch release) since it's a widely used large project.

Krzysztof

stanhu commented 2 years ago

@kwilczynski Not hard. Could you release a new version? bundle exec rake gem and gem build ruby-magic.gemspec don't seem to be building the package with the .tar.gz.

kwilczynski commented 2 years ago

Hi @stanhu,

@kwilczynski Not hard. Could you release a new version? bundle exec rake gem and gem build ruby-magic.gemspec don't seem to be building the package with the .tar.gz.

Done! As per https://github.com/kwilczynski/ruby-magic/issues/28#issuecomment-1023763179. This version contains patches and the libmagic source code bundled correctly. It should, hopefully, be OK for you to bump it on the GitLab's side.

Krzysztof

kwilczynski commented 2 years ago

Hi @pluff, I am going to close this issue.

Hopefully, it will not manifest itself again. That said, I updated the libmagic version to 5.42 in the upcoming release and also made sure that the vendored patch we apply is up-to-date.

That said, I need to work with the upstream project to see if there is a better solution than us always vendoring a patch.

Feel free to re-open if needed.