Closed lucaskanashiro closed 2 years ago
kwilczynski
commented
2 years ago Hi @lucaskanashiro,
Sorry for troubles you are having!
How are you building this Gem on Debian? Are you building it using the bundled libmagic or linking against the system-wide one?
Krzysztof
lucaskanashiro
commented
2 years ago Hi @kwilczynski ,
Thanks for the very quick response, appreciated! Per the Debian Policy, we need to build against libmagic installed from the system, coming from the Debian archive.
kwilczynski
commented
2 years ago Hi @lucaskanashiro,
Sorry for the delay!
I spun a Debian 11 minimal installation in a virtual machine to test and attempt to reproduce the problem, but sadly I can't seem to stumble into the same problem with the test failing. Everything I used was installed from the main repositories (no Ruby version manager of any sort was used), so that Ruby and libmagic are as shipping with Debian at the moment.
This is what I've done:
kwilczynski@debian11:~/ruby-magic$ lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description: Debian GNU/Linux 11 (bullseye)
Release: 11
Codename: bullseye
kwilczynski@debian11:~/ruby-magic$ apt show ruby bundler libmagic1
Package: ruby
Version: 1:2.7+2
Priority: optional
Section: ruby
Source: ruby-defaults
Maintainer: Debian Ruby Team <pkg-ruby-extras-maintainers@lists.alioth.debian.org>
Installed-Size: 35.8 kB
Provides: irb, rdoc, rubygems
Depends: ruby2.7, ruby-rubygems
Suggests: ri, ruby-dev
Conflicts: ruby-activesupport-2.3, ruby-activesupport-3.2
Breaks: apt-listbugs (<< 0.1.6), rbenv (<= 0.4.0-1), ruby-bootsnap (<< 1.4.6-1), ruby-debian (<< 0.3.8+b3), ruby-switch (<= 0.1.0)
Replaces: irb, rdoc, rubygems
Homepage: https://www.ruby-lang.org/
Tag: devel::interpreter, devel::lang:ruby, devel::runtime,
implemented-in::ruby, interface::commandline, role::dummy,
role::metapackage, role::program, works-with::software:source
Download-Size: 11.7 kB
APT-Manual-Installed: yes
APT-Sources: http://ftp.pl.debian.org/debian bullseye/main amd64 Packages
Description: Interpreter of object-oriented scripting language Ruby (default version)
Ruby is the interpreted scripting language for quick and easy
object-oriented programming. It has many features to process text
files and to do system management tasks (as in perl). It is simple,
straight-forward, and extensible.
.
This package is a dependency package, which depends on Debian's default Ruby
version (currently v2.7).
Package: bundler
Version: 2.2.5-2
Priority: optional
Section: ruby
Source: rubygems (3.2.5-2)
Maintainer: Debian Ruby Team <pkg-ruby-extras-maintainers@lists.alioth.debian.org>
Installed-Size: 76.8 kB
Depends: ruby-bundler (= 2.2.5-2)
Recommends: build-essential, less, ruby-dev, sudo
Homepage: https://rubygems.org
Tag: admin::configuring, admin::install, devel::buildtools, devel::lang:ruby,
devel::library, devel::packaging, implemented-in::ruby,
interface::commandline, protocol::http, role::devel-lib, role::program,
scope::utility, use::configuring, use::downloading,
works-with-format::TODO, works-with::software:package,
works-with::software:source
Download-Size: 65.6 kB
APT-Sources: http://ftp.pl.debian.org/debian bullseye/main amd64 Packages
Description: Manage Ruby application dependencies
Bundler manages a Ruby application's dependencies through its entire
life, across many machines, systematically and repeatably.
.
This is a dependency package which depends on a full Ruby development
environment, it is targeted at application developers.
Package: libmagic1
Version: 1:5.39-3
Priority: optional
Section: libs
Source: file
Maintainer: Christoph Biedl <debian.axhn@manchmal.in-ulm.de>
Installed-Size: 256 kB
Depends: libbz2-1.0, libc6 (>= 2.15), zlib1g (>= 1:1.1.4), libmagic-mgc (= 1:5.39-3)
Suggests: file
Homepage: https://www.darwinsys.com/file/
Tag: role::shared-lib
Download-Size: 126 kB
APT-Manual-Installed: no
APT-Sources: http://ftp.pl.debian.org/debian bullseye/main amd64 Packages
Description: Recognize the type of data in a file using "magic" numbers - library
This library can be used to classify files according to magic number
tests. It implements the core functionality of the file command.
kwilczynski@debian11:~/ruby-magic$ gem env
RubyGems Environment:
- RUBYGEMS VERSION: 3.2.5
- RUBY VERSION: 2.7.4 (2021-07-07 patchlevel 191) [x86_64-linux-gnu]
- INSTALLATION DIRECTORY: /var/lib/gems/2.7.0
- USER INSTALLATION DIRECTORY: /home/kwilczynski/.gem/ruby/2.7.0
- RUBY EXECUTABLE: /usr/bin/ruby2.7
- GIT EXECUTABLE: /usr/bin/git
- EXECUTABLE DIRECTORY: /usr/local/bin
- SPEC CACHE DIRECTORY: /home/kwilczynski/.local/share/gem/specs
- SYSTEM CONFIGURATION DIRECTORY: /etc
- RUBYGEMS PLATFORMS:
- ruby
- x86_64-linux
- GEM PATHS:
- /var/lib/gems/2.7.0
- /home/kwilczynski/.gem/ruby/2.7.0
- /usr/local/lib/ruby/gems/2.7.0
- /usr/lib/ruby/gems/2.7.0
- /usr/lib/x86_64-linux-gnu/ruby/gems/2.7.0
- /usr/share/rubygems-integration/2.7.0
- /usr/share/rubygems-integration/all
- /usr/lib/x86_64-linux-gnu/rubygems-integration/2.7.0
- GEM CONFIGURATION:
- :update_sources => true
- :verbose => true
- :backtrace => false
- :bulk_threshold => 1000
- REMOTE SOURCES:
- https://rubygems.org/
- SHELL PATH:
- /usr/local/bin
- /bin
- /usr/local/games
- /usr/games
kwilczynski@debian11:~/ruby-magic$ bundle config
Settings are listed in order of priority. The top value will be used.
path
Set for your local app (/home/kwilczynski/ruby-magic/.bundle/config): "/home/kwilczynski/.gem"
kwilczynski@debian11:~/ruby-magic$ ruby -v
ruby 2.7.4p191 (2021-07-07 revision a21a3b7d23) [x86_64-linux-gnu]
kwilczynski@debian11:~/ruby-magic$ rake clobber
kwilczynski@debian11:~/ruby-magic$ rake compile -- --use-system-libraries
Building ruby-magic using system libraries.
checking for ruby.h... yes
checking for -lpthread... yes
checking for -lrt... yes
checking for -ldl... yes
checking for -lcrypt... yes
checking for rb_thread_call_without_gvl()... yes
checking for rb_thread_blocking_region()... no
checking for magic.h... yes
checking for -lmagic... yes
checking for magic_getpath()... yes
checking for magic_getflags()... yes
checking for utime.h... yes
checking for sys/types.h... yes
checking for sys/time.h... yes
checking for utime()... yes
checking for utimes()... yes
creating extconf.h
creating Makefile
compiling ../../../../ext/magic/functions.c
In file included from /usr/include/ruby-2.7.0/ruby.h:33,
from ../../../../ext/magic/common.h:32,
from ../../../../ext/magic/functions.h:8,
from ../../../../ext/magic/functions.c:5:
/usr/include/ruby-2.7.0/ruby/ruby.h: In function 'rb_scan_args_set':
/usr/include/ruby-2.7.0/ruby/ruby.h:2592:40: warning: conversion to 'long unsigned int' from 'int' may change the sign of the result [-Wsign-conversion]
2592 | memcpy(ptr, argv, sizeof(VALUE)*argc);
| ^
/usr/include/ruby-2.7.0/ruby/ruby.h:1763:54: note: in definition of macro 'memcpy'
1763 | #define memcpy(p1,p2,n) ruby_nonempty_memcpy(p1, p2, n)
| ^
compiling ../../../../ext/magic/ruby-magic.c
In file included from /usr/include/ruby-2.7.0/ruby.h:33,
from ../../../../ext/magic/common.h:32,
from ../../../../ext/magic/ruby-magic.h:8,
from ../../../../ext/magic/ruby-magic.c:5:
/usr/include/ruby-2.7.0/ruby/ruby.h: In function 'rb_scan_args_set':
/usr/include/ruby-2.7.0/ruby/ruby.h:2592:40: warning: conversion to 'long unsigned int' from 'int' may change the sign of the result [-Wsign-conversion]
2592 | memcpy(ptr, argv, sizeof(VALUE)*argc);
| ^
/usr/include/ruby-2.7.0/ruby/ruby.h:1763:54: note: in definition of macro 'memcpy'
1763 | #define memcpy(p1,p2,n) ruby_nonempty_memcpy(p1, p2, n)
| ^
linking shared-object magic/magic.so
kwilczynski@debian11:~/ruby-magic$ ldd lib/magic/magic.so
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
linux-vdso.so.1 (0x00007ffddf6d9000)
libruby-2.7.so.2.7 => /lib/x86_64-linux-gnu/libruby-2.7.so.2.7 (0x00007f7d253f8000)
libmagic.so.1 => /lib/x86_64-linux-gnu/libmagic.so.1 (0x00007f7d253cd000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f7d253ab000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f7d251e6000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f7d251db000)
libgmp.so.10 => /lib/x86_64-linux-gnu/libgmp.so.10 (0x00007f7d2515a000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f7d25152000)
libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007f7d25117000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f7d24fd3000)
/lib64/ld-linux-x86-64.so.2 (0x00007f7d2576b000)
libbz2.so.1.0 => /lib/x86_64-linux-gnu/libbz2.so.1.0 (0x00007f7d24fc0000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f7d24fa3000)
kwilczynski@debian11:~/ruby-magic$ chrpath lib/magic/magic.so
lib/magic/magic.so: no rpath or runpath tag found.
kwilczynski@debian11:~/ruby-magic$ rake test
Loaded suite /home/kwilczynski/.gem/ruby/2.7.0/gems/rake-13.0.6/lib/rake/rake_test_loader
Started
......................................................................................................................................O
==================================================================================================================================================================================================================================================
Omission: Platform supports fork [test_magic_singleton_do_not_auto_load_global_no_fork(MagicTest)]
/home/kwilczynski/ruby-magic/test/test_magic.rb:785:in `test_magic_singleton_do_not_auto_load_global_no_fork'
==================================================================================================================================================================================================================================================
...........................
Finished in 0.053067688 seconds.
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
162 tests, 237 assertions, 0 failures, 0 errors, 0 pendings, 1 omissions, 0 notifications
100% passed
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
3052.71 tests/s, 4465.99 assertions/s
kwilczynski@debian11:~/ruby-magic$ RUBYLIB=./lib irb
irb(main):001:0> require 'magic'
=> true
irb(main):002:0> m = Magic.new
=> #<Magic:0x000055f544e1eee0 @flags=0, @paths=["/etc/magic", "/usr/share/misc/magic"]>
irb(main):003:0> m.paths
=> ["/etc/magic", "/usr/share/misc/magic"]
irb(main):004:0> m.file '/etc/hosts'
=> "ASCII text"
irb(main):005:0> m.file '/dev/null'
=> "character special (1/3)"
irb(main):006:0>
kwilczynski@debian11:~/ruby-magic$ Let me know if there is something you would like me to do so that I can reliably reproduce this on my side.
Krzysztof
lucaskanashiro
commented
2 years ago Thanks for trying this out, but with our tooling to build ruby gems it is failing, look at the logs of our CI pipeline:
https://salsa.debian.org/ruby-team/ruby-ruby-magic-static/-/jobs/2138963
I'd tell you to test with Debian unstable and also use the source package we have staged in our Gitlab instance. With the steps below you will be able to reproduce the failure that I mentioned using docker:
$ docker run -it debian:unstable /bin/bash
# apt update && apt upgrade -y
# apt install git git-buildpackage devscripts -y
# gbp clone https://salsa.debian.org/ruby-team/ruby-ruby-magic-static.git && cd ruby-ruby-magic-static/
# apt install gem2deb ruby-mini-portile2 libmagic-dev -y
# gbp buildpackage --git-builder=debuild -us -ucHi @lucaskanashiro,
Thanks for trying this out, but with our tooling to build ruby gems it is failing, look at the logs of our CI pipeline:
https://salsa.debian.org/ruby-team/ruby-ruby-magic-static/-/jobs/2138963 [...]
Thank you for sending this over! This is very helpful, and from looking through the logs I can see that you aren't building the Gem against system-wide installation of libmagic, as per:
https://salsa.debian.org/ruby-team/ruby-ruby-magic-static/-/jobs/2138963#L1397
Excerpt from the build log:
---------- IMPORTANT NOTICE ----------
Building Ruby Magic with a packaged version of libmagic-5.39.
Configuration options: --disable-silent-rules --disable-dependency-tracking --enable-fsect-man5 --disable-shared --enable-static CFLAGS\=-g\ -O2\ -ffile-prefix-map\=/builds/ruby-team/ruby-ruby-magic-static/debian/output/source_dir\=.\ -fstack-protector-strong\ -Wformat\ -Werror\=format-security\ -fPIC
Downloading file-5.39.tar.gz (100%)
Extracting file-5.39.tar.gz into tmp/x86_64-linux-gnu/ports/libmagic/5.39... OK
Running 'configure' for libmagic 5.39... OK
Running 'compile' for libmagic 5.39... OK
Running 'install' for libmagic 5.39... OK
Activating libmagic 5.39 (from /builds/ruby-team/ruby-ruby-magic-static/debian/output/source_dir/debian/ruby-ruby-magic-static/usr/lib/x86_64-linux-gnu/rubygems-integration/3.0.0/gems/ruby-magic-0.5.0/ports/x86_64-linux-gnu/libmagic/5.39)...This is because of the following:
https://salsa.debian.org/ruby-team/ruby-ruby-magic-static/-/blob/master/debian/rules#L5
The current:
export RB_MAGIC_USE_SYSTEM_LIBRARIES = trueWould need to be updated to:
export MAGIC_USE_SYSTEM_LIBRARIES = trueAs we removed the RB_ prefix some time ago. Try if this works and things build fine after you update the rules file.
Once the build against system-wide libmagic will work, then as part of it, it would enable a little bit of code that ensures that flags are preserved between certain operations. This will then fix the failing test.
I will update changelog retrospectively and ensure that new releases are also released with proper changelog. This one is on me to fix. Apologies!
Krzysztof
lucaskanashiro
commented
2 years ago @kwilczynski thank you! Renaming the variable in debian/rules fixed the issue indeed. Sorry for bothering you with that. And thanks again for the support here.
kwilczynski
commented
2 years ago Hi @lucaskanashiro,
@kwilczynski thank you! Renaming the variable in
debian/rulesfixed the issue indeed. Sorry for bothering you with that.
Not a problem! Bother all you want in the future - I am here to help.
And thanks again for the support here.
Thank you for working on packaging this Gem! Having said that, I had a look at how it's being build, and I believe that you don't need this patch:
When you are building using system-wide libraries, the rpath will not be set as the if-statement branch will never really run this code. Similarly this following patch would, or rather should, not be needed as this is only ever executed when the Gem is built against the bundled libmagic:
This makes me worried that something is not working correctly.
Also a question about this patch:
How did this break for you? Perhaps this is something I need to look into.
Krzysztof
kwilczynski
commented
2 years ago Hi @lucaskanashiro,
Forgive me, just one more question: why does the Debian package have the -static suffix? Is this mainly because of how the repository was originally named when @stanhu worked on it and a GitLab fix, or because you folks are building it against statically built Ruby and such?
Krzysztof
dleidert
commented
2 years ago @kwilczynski, answering on behalf of the Debian team: I think this has historical reasons and we need to change the package name. Other distros have renamed it already.
kwilczynski
commented
2 years ago Hi @dleidert,
Thank you for getting in touch! I see, no worries than! What do you think about some of the other questions above?
Also, let me know if you need anything updated, changed, fixed... to make your and Debian's team life easier. And thank you again for taking the time to package this Gem! I appreciate it!
By the way, I had a peek at the control file and I wonder if the package ought to add more dependencies? See https://gitlab.com/gitlab-org/ruby-magic/-/merge_requests/1#note_537310311 for reference, and do let me know what do you think.
Krzysztof
lucaskanashiro
commented
2 years ago @kwilczynski I checked the first two patches you mentioned and I agree they are not needed anymore, I removed them in the git repository and this change will land in the next upload to the Debian archive:
Regarding the third patch, this is needed because when we build the extension it is installed in another directory, if we use require_relative we will get the following error when building the package:
File does not exist: /<<PKGBUILDDIR>>/debian/ruby-ruby-magic-static/usr/lib/x86_64-linux-gnu/rubygems-integration/2.7.0/gems/ruby-magic-0.5.0/lib/magic/magicYour question about the package name is what @dleidert said, we can plan to rename it to ruby-ruby-magic.
And finally, your concern about the dependencies is valid but all the libraries it is linked to are always available in a Debian minimal system (they are required), so no need to explicit list them as build dependencies.
stanhu
commented
2 years ago When you are building using system-wide libraries, the rpath will not be set as the if-statement branch will never really run this code. Similarly this following patch would, or rather should, not be needed as this is only ever executed when the Gem is built against the bundled libmagic:
I see that @pravi filed https://gitlab.com/gitlab-org/ruby-magic/-/issues/3. The to_s was needed because pkg_config('libmagic', 'libs --static') returned nothing, which it seems to be on my Ubuntu 20.04 system:
$ pkg-config libmagic libs --static
$This could happen if pkg-config didn't find the .pc file. Maybe we should keep this to_s in case?
stanhu
commented
2 years ago On the other hand, we shouldn't be getting a blank entry if we're extracted the tarball and pointed to the path specifically. Perhaps https://gitlab.com/gitlab-org/ruby-magic/-/issues/3 is outdated now.
kwilczynski
commented
2 years ago Hi @lucaskanashiro,
Thank you for all the details and apologies for late reply!
[...]
Regarding the third patch, this is needed because when we build the extension it is installed in another directory, if we use
require_relativewe will get the following error when building the package:File does not exist: /<<PKGBUILDDIR>>/debian/ruby-ruby-magic-static/usr/lib/x86_64-linux-gnu/rubygems-integration/2.7.0/gems/ruby-magic-0.5.0/lib/magic/magic
Ah, OK. I need to look into this. Perhaps require_relative is not needed there and can be replaced.
Your question about the package name is what @dleidert said, we can plan to rename it to
ruby-ruby-magic.
Oh! Nice!
And finally, your concern about the dependencies is valid but all the libraries it is linked to are always available in a Debian minimal system (they are required), so no need to explicit list them as build dependencies.
OK. Makes sense, since you are also building libmagic that the Gem will use.
Thank you again for getting back to me with the answers and for packaging the Gem!
Krzysztof
kwilczynski
commented
2 years ago Hi @stanhu,
I appreciate you looked into this! Thank you!
[...]
This could happen if
pkg-configdidn't find the.pcfile. Maybe we should keep thisto_sin case?
[...]
On the other hand, we shouldn't be getting a blank entry if we're extracted the tarball and pointed to the path specifically. Perhaps https://gitlab.com/gitlab-org/ruby-magic/-/issues/3 is outdated now.
Just doing simple installation I wasn't able to reproduce this with the way how our installation is now working. We can, however, play it safe, so to speak, and either just turn it into a string (as per the change done by @pravi), or perhaps even issue a warning about being unable to resolve build dependencies using pkg-config.
Thoughts?
Krzysztof
stanhu
commented
2 years ago Just doing simple installation I wasn't able to reproduce this with the way how our installation is now working. We can, however, play it safe, so to speak, and either just turn it into a string (as per the change done by @pravi), or perhaps even issue a warning about being unable to resolve build dependencies using
pkg-config.
@kwilczynski As I said in https://gitlab.com/gitlab-org/ruby-magic/-/issues/3#note_724465221, I realized that if pkg-config is returning empty values, the static library compilation should fail since the requisite libraries aren't there. I don't know how this worked if that were happening.
kwilczynski
commented
2 years ago Hi @stanhu,
Just doing simple installation I wasn't able to reproduce this with the way how our installation is now working. We can, however, play it safe, so to speak, and either just turn it into a string (as per the change done by @pravi), or perhaps even issue a warning about being unable to resolve build dependencies using
pkg-config.@kwilczynski As I said in https://gitlab.com/gitlab-org/ruby-magic/-/issues/3#note_724465221, I realized that if
pkg-configis returning empty values, the static library compilation should fail since the requisite libraries aren't there. I don't know how this worked if that were happening.
Ah, sorry! I haven't seen your comment on GitLab, apologies. And indeed, this would have broke quite some time ago for us, or other uses (excluding Debian folks).
OK. Nothing to do here then, and we can fix this should this be a problem ever again.
Thank you!
Krzysztof
stanhu
commented
2 years ago I actually ran into this problem while testing ruby-magic on another platform. The issue in my case was that pkg-config was not installed, so the pkg_config in extconf.rb failed to return any output.
We might want to consider what Nokogiri does: https://github.com/sparklemotion/nokogiri/blob/3814fdff251b0f6d07fe9ab4992c1896ab7b4ead/ext/nokogiri/extconf.rb#L220-L251
Alternatively, we could disable static compilation or just use the default flags:
-L/path/to/build/libmagic/5.39/lib -lmagic -lzHi @stanhu,
I actually ran into this problem while testing ruby-magic on another platform. The issue in my case was that
pkg-configwas not installed, so thepkg_configinextconf.rbfailed to return any output.
Ah not good! But a good thing you confirmed this being an issue. I apologise for the troubles!
We might want to consider what Nokogiri does: https://github.com/sparklemotion/nokogiri/blob/3814fdff251b0f6d07fe9ab4992c1896ab7b4ead/ext/nokogiri/extconf.rb#L220-L251
We can try, indeed. I also hope that @flavorjones won't mind us borrowing some more of his excellent work!
Alternatively, we could disable static compilation or just use the default flags:
-L/path/to/build/libmagic/5.39/lib -lmagic -lz
That would be the "Plan B", so to speak, if adopting the solution used by Nokogiri will prove to be problematic.
Krzysztof
After importing version
0.5.0to Debian, we got the following test failure:I noticed this commit made a change to this test, and before that there is this discussion in this PR about this very same issue. It seems it is still true and the previous code is needed. If we move back to content of this commit the test is fixed.