search

kwilczynski / ruby-magic

Simple interface to libmagic for Ruby Programming Language
Apache License 2.0
27 stars 8 forks source link

Build by vendoring libmagic #3

Closed stanhu closed 3 years ago

stanhu commented 3 years ago

This commit downloads the file library from http://www.darwinsys.com/file/, builds and installs the files into lib/ext, and links the C extension against it.

At the moment, this isn't staticallly linked because I ran into some linker issues on macOS, but it could be done later if desired. The file binary is present and could be removed as well.

Closes #1

kwilczynski commented 3 years ago

Hi @stanhu, thank you for this!

This is very nice and a step in the right direction. The static linking issues is something I also run in the past - there is issues with linker, plus mkmf also stops being able to detect whether functions (symbols) are available, so the following check using have_func() would stop working correctly:

https://github.com/kwilczynski/ruby-magic/blob/6c110aaf93298e166475bbea4542d139ccf6916f/ext/magic/extconf.rb#L126-L136

But, I haven't looked into this since, apologies!

Krzysztof

stanhu commented 3 years ago

Talk about a ton of builds! https://travis-ci.org/github/kwilczynski/ruby-magic/builds/764335565

I wonder why some of the builds are timing out; could it be Travis or the FTP site throttling connections?

stanhu commented 3 years ago

Oh, I guess we can clean up .travis-ci.yml if we don't need to test every version.

kwilczynski commented 3 years ago

Hi @stanhu,

Talk about a ton of builds! travis-ci.org/github/kwilczynski/ruby-magic/builds/764335565

Yes, sorry about this!

Since in the past I had to rely on supporting the version of libmagic that was present in the environment where ruby-magic could have been installed (and libmagic has some interesting issues in some of the older version) I was testing against versions that seemed to be most widely available.

I wonder why some of the builds are timing out; could it be Travis or the FTP site throttling connections?

Could be both! I have odd issues with the free version of Travis CI in the past, but also the FTP server that hosts the libmagic has some limits I think (or is generally slow).

This is why I used a small build script that attempts to fetch libmagic source from multiple mirrors, and once it succeeds at least once, then next time it would leverage Travis CI build cache - which is not something that we have the benefit of when downloading source each time now.

Krzysztof

kwilczynski commented 3 years ago

Hi @stanhu,

Oh, I guess we can clean up .travis-ci.yml if we don't need to test every version.

Definitely. It's not really needed any more - perhaps we will revisit this in the future once moving to something like the mini_portile Ruby Gem, where people could pass a command-line switch to use libraries present already in the system (so we can test this scenario correctly too), but for the time being no issue with removing the extra versions and the test matrix.

I also need to think how to bring testing with both GCC and Clang back. :)

Krzysztof

stanhu commented 3 years ago

We could still have a matrix test with different compilers. I think setting the CC variable would do that.

I think Nokogiri (https://github.com/sparklemotion/nokogiri/blob/main/ext/nokogiri/extconf.rb) may be a good example of allowing users to use the system libraries.

kwilczynski commented 3 years ago

Hi @stanhu,

We could still have a matrix test with different compilers. I think setting the CC variable would do that.

We can do that and directly use CC - it should work - or keep it as I did it in the past.

I think Nokogiri (https://github.com/sparklemotion/nokogiri/blob/main/ext/nokogiri/extconf.rb) may be a good example of allowing users to use the system libraries.

I was looking at Nokogori as an example of how it's done - it seems few other projects adopted similar approach (some even based on Nokogiri's way of handling options, etc.), and I wanted (when I was initially thinking of vendoring libmagic) to keep things consistent.

Krzysztof

kwilczynski commented 3 years ago

Hi @stanhu,

I uploaded libmagic source code to an S3 bucket to solve the timeout during download issues, as per:

But since I noticed you found a mirror. :)

Krzysztof

kwilczynski commented 3 years ago

Hi @stanhu,

You should be able to merge at your convenience when you are happy with the changes. I added you as collaborator.

Thank you again for help!

Krzysztof

stanhu commented 3 years ago

@kwilczynski Thanks! I'm playing around with mini_portile2, but those improvements can come later.

stanhu commented 3 years ago

@kwilczynski Thanks for your help! Could you add me to RubyGems as well? https://rubygems.org/profiles/stanhu

kwilczynski commented 3 years ago

Hi @stanhu,

@kwilczynski Thanks for your help! Could you add me to RubyGems as well? rubygems.org/profiles/stanhu

Done! You should receive invite shortly.

Krzysztof

jrochkind commented 3 years ago

I wonder why some of the builds are timing out; could it be Travis or the FTP site throttling connections?

Travis doesn't really offer free support to open source anymore, although they are having trouble being transparent about saying so, instead making it complicated to come to that determination.

https://www.jeffgeerling.com/blog/2020/travis-cis-new-pricing-plan-threw-wrench-my-open-source-works

Github actions is one alternative.

olearycrew commented 3 years ago

Another alternative is GitLab CI/CD 😏