sparklemotion / nokogiri

Nokogiri (鋸) makes it easy and painless to work with XML and HTML from Ruby.
https://nokogiri.org/
MIT License
6.15k stars 897 forks source link

Compile issue on OS X Mavericks 10.9.5 #1206

Closed baburdick closed 9 years ago

baburdick commented 9 years ago

Despite saying that it's using a special bundled libxml2, the process fails to find it. Here is the interesting excerpt from ~/.rvm/gems/ruby-2.1.2@my_service/extensions/x86_64-darwin-13/2.1.0-static/nokogiri-1.6.5/mkmf.log. Suspicious lines (there is no "travis" user on my system):

ld: warning: directory not found for option '-L/Users/travis/.sm/pkg/active/lib'
ld: library not found for -llibxml2
conftest.c:15:27: error: too few arguments to function call, single argument 'cur' was not specified
int t(void) { xmlParseDoc(); return 0; }
              ~~~~~~~~~~~ ^
/Users/bruce/.rvm/gems/ruby-2.1.2@my_service/gems/nokogiri-1.6.5/ports/x86_64-apple-darwin13.1.0/libxml2/2.9.2/include/libxml2/libxml/parser.h:841:11: note: 'xmlParseDoc' declared here
XMLPUBFUN xmlDocPtr XMLCALL
          ^
have_library: checking for xmlParseDoc() in -llibxml2... -------------------- no

"gcc -o conftest -I/Users/bruce/.rvm/rubies/ruby-2.1.2/include/ruby-2.1.0/x86_64-darwin13.0 -I/Users/bruce/.rvm/rubies/ruby-2.1.2/include/ruby-2.1.0/ruby/backward -I/Users/bruce/.rvm/rubies/ruby-2.1.2/include/ruby-2.1.0 -I. -I/Users/bruce/.rvm/gems/ruby-2.1.2@my_service/gems/nokogiri-1.6.5/ports/x86_64-apple-darwin13.1.0/libxslt/1.1.28/include -I/Users/bruce/.rvm/gems/ruby-2.1.2@my_service/gems/nokogiri-1.6.5/ports/x86_64-apple-darwin13.1.0/libxml2/2.9.2/include/libxml2 -I/Users/bruce/.rvm/gems/ruby-2.1.2@my_service/gems/nokogiri-1.6.5/ports/x86_64-apple-darwin13.1.0/libxml2/2.9.2/include/libxml2 -D_XOPEN_SOURCE -D_DARWIN_C_SOURCE -D_DARWIN_UNLIMITED_SELECT -D_REENTRANT   -DNOKOGIRI_LIBXML2_PATH\=\"/Users/bruce/.rvm/gems/ruby-2.1.2@my_service/gems/nokogiri-1.6.5/ports/x86_64-apple-darwin13.1.0/libxml2/2.9.2\" -DNOKOGIRI_LIBXML2_PATCHES\=\"0001-Revert-Missing-initialization-for-the-catalog-module.patch\ 0002-Fix-missing-entities-after-CVE-2014-3660-fix.patch\" -DNOKOGIRI_LIBXSLT_PATH\=\"/Users/bruce/.rvm/gems/ruby-2.1.2@my_service/gems/nokogiri-1.6.5/ports/x86_64-apple-darwin13.1.0/libxslt/1.1.28\" -DNOKOGIRI_LIBXSLT_PATCHES\=\"0001-Adding-doc-update-related-to-1.1.28.patch\ 0002-Fix-a-couple-of-places-where-f-printf-parameters-wer.patch\ 0003-Initialize-pseudo-random-number-generator-with-curre.patch\ 0004-EXSLT-function-str-replace-is-broken-as-is.patch\ 0006-Fix-str-padding-to-work-with-UTF-8-strings.patch\ 0007-Separate-function-for-predicate-matching-in-patterns.patch\ 0008-Fix-direct-pattern-matching.patch\ 0009-Fix-certain-patterns-with-predicates.patch\ 0010-Fix-handling-of-UTF-8-strings-in-EXSLT-crypto-module.patch\ 0013-Memory-leak-in-xsltCompileIdKeyPattern-error-path.patch\ 0014-Fix-for-bug-436589.patch\ 0015-Fix-mkdir-for-mingw.patch\" -O3 -I/Users/travis/.sm/pkg/active/include -fPIC -mmacosx-version-min=10.6 -pipe  -Wall -Wcast-qual -Wwrite-strings -Wconversion -Wmissing-noreturn -Winline -DNOKOGIRI_USE_PACKAGED_LIBRARIES conftest.c  -L. -L/Users/bruce/.rvm/rubies/ruby-2.1.2/lib -L/Users/bruce/.rvm/gems/ruby-2.1.2@my_service/gems/nokogiri-1.6.5/ports/x86_64-apple-darwin13.1.0/libxml2/2.9.2/lib -L/Users/bruce/.rvm/gems/ruby-2.1.2@my_service/gems/nokogiri-1.6.5/ports/x86_64-apple-darwin13.1.0/libxslt/1.1.28/lib -L. -L/Users/travis/.sm/pkg/active/lib -fPIC -Bstatic -fstack-protector   -arch x86_64  /Users/bruce/.rvm/gems/ruby-2.1.2@my_service/gems/nokogiri-1.6.5/ports/x86_64-apple-darwin13.1.0/libxslt/1.1.28/lib/libexslt.a -lm -liconv -lpthread -lz /Users/bruce/.rvm/gems/ruby-2.1.2@my_service/gems/nokogiri-1.6.5/ports/x86_64-apple-darwin13.1.0/libxml2/2.9.2/lib/libxml2.a /Users/bruce/.rvm/gems/ruby-2.1.2@my_service/gems/nokogiri-1.6.5/ports/x86_64-apple-darwin13.1.0/libxslt/1.1.28/lib/libxslt.a -lm -liconv -lpthread -lz /Users/bruce/.rvm/gems/ruby-2.1.2@my_service/gems/nokogiri-1.6.5/ports/x86_64-apple-darwin13.1.0/libxml2/2.9.2/lib/libxml2.a -llzma -lruby-static -framework CoreFoundation -llibxml2 /Users/bruce/.rvm/gems/ruby-2.1.2@my_service/gems/nokogiri-1.6.5/ports/x86_64-apple-darwin13.1.0/libxslt/1.1.28/lib/libexslt.a -lm -liconv -lpthread -lz /Users/bruce/.rvm/gems/ruby-2.1.2@my_service/gems/nokogiri-1.6.5/ports/x86_64-apple-darwin13.1.0/libxml2/2.9.2/lib/libxml2.a /Users/bruce/.rvm/gems/ruby-2.1.2@my_service/gems/nokogiri-1.6.5/ports/x86_64-apple-darwin13.1.0/libxslt/1.1.28/lib/libxslt.a -lm -liconv -lpthread -lz /Users/bruce/.rvm/gems/ruby-2.1.2@my_service/gems/nokogiri-1.6.5/ports/x86_64-apple-darwin13.1.0/libxml2/2.9.2/lib/libxml2.a -llzma -lpthread -ldl -lobjc  "
ld: warning: directory not found for option '-L/Users/travis/.sm/pkg/active/lib'
ld: library not found for -llibxml2
clang: error: linker command failed with exit code 1 (use -v to see invocation)
checked program was:
/* begin */
 1: #include "ruby.h"
 2: 
 3: #include <libxml/parser.h>
 4: 
 5: /*top*/
 6: extern int t(void);
 7: int main(int argc, char **argv)
 8: {
 9:   if (argc > 1000000) {
10:     printf("%p", &t);
11:   }
12: 
13:   return 0;
14: }
15: int t(void) { void ((*volatile p)()); p = (void ((*)()))xmlParseDoc; return 0; }
/* end */

"gcc -o conftest -I/Users/bruce/.rvm/rubies/ruby-2.1.2/include/ruby-2.1.0/x86_64-darwin13.0 -I/Users/bruce/.rvm/rubies/ruby-2.1.2/include/ruby-2.1.0/ruby/backward -I/Users/bruce/.rvm/rubies/ruby-2.1.2/include/ruby-2.1.0 -I. -I/Users/bruce/.rvm/gems/ruby-2.1.2@my_service/gems/nokogiri-1.6.5/ports/x86_64-apple-darwin13.1.0/libxslt/1.1.28/include -I/Users/bruce/.rvm/gems/ruby-2.1.2@my_service/gems/nokogiri-1.6.5/ports/x86_64-apple-darwin13.1.0/libxml2/2.9.2/include/libxml2 -I/Users/bruce/.rvm/gems/ruby-2.1.2@my_service/gems/nokogiri-1.6.5/ports/x86_64-apple-darwin13.1.0/libxml2/2.9.2/include/libxml2 -D_XOPEN_SOURCE -D_DARWIN_C_SOURCE -D_DARWIN_UNLIMITED_SELECT -D_REENTRANT   -DNOKOGIRI_LIBXML2_PATH\=\"/Users/bruce/.rvm/gems/ruby-2.1.2@my_service/gems/nokogiri-1.6.5/ports/x86_64-apple-darwin13.1.0/libxml2/2.9.2\" -DNOKOGIRI_LIBXML2_PATCHES\=\"0001-Revert-Missing-initialization-for-the-catalog-module.patch\ 0002-Fix-missing-entities-after-CVE-2014-3660-fix.patch\" -DNOKOGIRI_LIBXSLT_PATH\=\"/Users/bruce/.rvm/gems/ruby-2.1.2@my_service/gems/nokogiri-1.6.5/ports/x86_64-apple-darwin13.1.0/libxslt/1.1.28\" -DNOKOGIRI_LIBXSLT_PATCHES\=\"0001-Adding-doc-update-related-to-1.1.28.patch\ 0002-Fix-a-couple-of-places-where-f-printf-parameters-wer.patch\ 0003-Initialize-pseudo-random-number-generator-with-curre.patch\ 0004-EXSLT-function-str-replace-is-broken-as-is.patch\ 0006-Fix-str-padding-to-work-with-UTF-8-strings.patch\ 0007-Separate-function-for-predicate-matching-in-patterns.patch\ 0008-Fix-direct-pattern-matching.patch\ 0009-Fix-certain-patterns-with-predicates.patch\ 0010-Fix-handling-of-UTF-8-strings-in-EXSLT-crypto-module.patch\ 0013-Memory-leak-in-xsltCompileIdKeyPattern-error-path.patch\ 0014-Fix-for-bug-436589.patch\ 0015-Fix-mkdir-for-mingw.patch\" -O3 -I/Users/travis/.sm/pkg/active/include -fPIC -mmacosx-version-min=10.6 -pipe  -Wall -Wcast-qual -Wwrite-strings -Wconversion -Wmissing-noreturn -Winline -DNOKOGIRI_USE_PACKAGED_LIBRARIES conftest.c  -L. -L/Users/bruce/.rvm/rubies/ruby-2.1.2/lib -L/Users/bruce/.rvm/gems/ruby-2.1.2@my_service/gems/nokogiri-1.6.5/ports/x86_64-apple-darwin13.1.0/libxml2/2.9.2/lib -L/Users/bruce/.rvm/gems/ruby-2.1.2@my_service/gems/nokogiri-1.6.5/ports/x86_64-apple-darwin13.1.0/libxslt/1.1.28/lib -L. -L/Users/travis/.sm/pkg/active/lib -fPIC -Bstatic -fstack-protector   -arch x86_64  /Users/bruce/.rvm/gems/ruby-2.1.2@my_service/gems/nokogiri-1.6.5/ports/x86_64-apple-darwin13.1.0/libxslt/1.1.28/lib/libexslt.a -lm -liconv -lpthread -lz /Users/bruce/.rvm/gems/ruby-2.1.2@my_service/gems/nokogiri-1.6.5/ports/x86_64-apple-darwin13.1.0/libxml2/2.9.2/lib/libxml2.a /Users/bruce/.rvm/gems/ruby-2.1.2@my_service/gems/nokogiri-1.6.5/ports/x86_64-apple-darwin13.1.0/libxslt/1.1.28/lib/libxslt.a -lm -liconv -lpthread -lz /Users/bruce/.rvm/gems/ruby-2.1.2@my_service/gems/nokogiri-1.6.5/ports/x86_64-apple-darwin13.1.0/libxml2/2.9.2/lib/libxml2.a -llzma -lruby-static -framework CoreFoundation -llibxml2 /Users/bruce/.rvm/gems/ruby-2.1.2@my_service/gems/nokogiri-1.6.5/ports/x86_64-apple-darwin13.1.0/libxslt/1.1.28/lib/libexslt.a -lm -liconv -lpthread -lz /Users/bruce/.rvm/gems/ruby-2.1.2@my_service/gems/nokogiri-1.6.5/ports/x86_64-apple-darwin13.1.0/libxml2/2.9.2/lib/libxml2.a /Users/bruce/.rvm/gems/ruby-2.1.2@my_service/gems/nokogiri-1.6.5/ports/x86_64-apple-darwin13.1.0/libxslt/1.1.28/lib/libxslt.a -lm -liconv -lpthread -lz /Users/bruce/.rvm/gems/ruby-2.1.2@my_service/gems/nokogiri-1.6.5/ports/x86_64-apple-darwin13.1.0/libxml2/2.9.2/lib/libxml2.a -llzma -lpthread -ldl -lobjc  "
conftest.c:15:27: error: too few arguments to function call, single argument 'cur' was not specified
int t(void) { xmlParseDoc(); return 0; }
              ~~~~~~~~~~~ ^
/Users/bruce/.rvm/gems/ruby-2.1.2@my_service/gems/nokogiri-1.6.5/ports/x86_64-apple-darwin13.1.0/libxml2/2.9.2/include/libxml2/libxml/parser.h:841:11: note: 'xmlParseDoc' declared here
XMLPUBFUN xmlDocPtr XMLCALL
          ^
1 error generated.
checked program was:
/* begin */
 1: #include "ruby.h"
 2: 
 3: #include <libxml/parser.h>
 4: 
 5: /*top*/
 6: extern int t(void);
 7: int main(int argc, char **argv)
 8: {
 9:   if (argc > 1000000) {
10:     printf("%p", &t);
11:   }
12: 
13:   return 0;
14: }
15: int t(void) { xmlParseDoc(); return 0; }
/* end */

--------------------
ArloL commented 9 years ago

I have a similar issue in 10.10.1 and created a gist with the necessary information.

I already tried entirely removing and re-installing the ruby.

behrangsa commented 9 years ago

I have a similar issue. However for me, the mysterious directory is /Users/mpapis/.sm/pkg/active/lib and mpapis is @mpapis's username.

The full error log is here: https://gist.github.com/behrangsa/d95a9e6e48d718fd6b94

In short, here are the list of errors:

System Details

OS: Mavericks 10.9.5 XCode: 6.1.1 Package Manager: MacPorts Ruby: 2.1.5 Ruby Manager: RVM xcode-select: /Library/Developer/CommandLineTools Command Line Tools version: 6.1

behrangsa commented 9 years ago

For me, setting the NOKOGIRI_USE_SYSTEM_LIBRARIES environment variable fixed the issue:

export NOKOGIRI_USE_SYSTEM_LIBRARIES=1
gem install nokogiri
mpapis commented 9 years ago

the warning with /Users/[mpapis|travis] is just a warning - it is not cause of any problems, you will find the same warning in successful build logs

behrangsa commented 9 years ago

@mpapis, I realized that, but it still doesn't seem correct for the gem to expect to find something in /Users/mpapis or /Users/travis.

mpapis commented 9 years ago

@behrangsa its limitation of remembering compilation flags by ruby, when the binary rubies are compiled by me or on travis they require to be given paths to statically linked libraries (*.a objects) - this path is then remembered by ruby and added to every gem compilation, as you can see this is not an error as only warning is issued, if you feel this requires fixing open a bug for MRI asking to provide separate flags for compilation that are not remembered ... or maybe allow filtering the remembered flags (I would also cut many of the -w* flags)

behrangsa commented 9 years ago

@mpapis, I didn't know that. If this is something common to gems with native extensions, etc. and how MRI works and it is harmless, then I have no complaints. :smiley:

mpapis commented 9 years ago

it is standard and harmless thing, but ruby could allow filtering out this flags, this could be done as part of the filtration process that already happens in preprocessing in miniruby - but I do not think it would be considered as a ticket without pull request / patch to address this problem.

jrep commented 9 years ago

export NOKOGIRI_USE_SYSTEM_LIBRARIES=1

But the nokogiri install insists on displaying scary warnings against doing this

For example, libxml2-2.9.0 and higher are currently known to be broken and thus unsupported by nokogiri, due to compatibility problems and XPath optimization bugs.

I'm beginning to suspect that the error I'm wrestling with

too few arguments to function call, single argument 'cur' was not specified

is one of those compatibility problems

flavorjones commented 9 years ago

@jrep Since a2ce84669758f0f7794839b3ece1b3da0f94e185, which is in Nokogiri 1.6.4, this warning has been toned down. This is the same version of Nokogiri that updated to libxml 2.9.2.

flavorjones commented 9 years ago

It's not clear to me that there's anything actionable in this issue. Please let me know in the next few days if something is still broken or you're still unhappy; otherwise I'll close after 2015-03-05.

cabo commented 9 years ago

The issue is very much alive, tl;dr: Can't build Nokogiri 1.6.6 on a modern OSX (10.9 or 10.10). I have exactly the issue documented in https://gist.github.com/ArloL/11c944436f5135de4347 -- nokogiri fails to find/understand the libxml2 it just built.

felipecsl commented 9 years ago

+1 same issue as described by @cabo here

felipecsl commented 9 years ago

ok was able to fix it by running

xcode-select --install
gem install nokogiri -v '1.6.6.2' -- --use-system-libraries --with-xml2-include=/usr/include/libxml2 --with-xml2- lib=/usr/lib/
dsedivec commented 9 years ago

I think I ran into the same problem as ArloL. In my case I suspect that it has something to do with my MacPorts iconv (in /opt/local/lib) vs. the system iconv (in /usr/lib). I get the feeling libxml2 is being linked against iconv from /opt/local/lib whereas libxslt is trying to pull in the system iconv from /usr/lib. I can't see why that would be, but in any case I was able to get Nokogiri 1.6.6.2 built with its bundled libxml2 and libxslt with LDFLAGS:

LDFLAGS=-L/opt/local/lib bundle install

Using Nokogiri's --with-iconv-dir=/opt/local did not fix this problem, by the way. It looked like -L/opt/local/lib was getting tacked on for the libxml2 build, but not the libxslt build.

I'm running Ruby 2.2.1 via rbenv, OS X 10.9.5.

flavorjones commented 9 years ago

@felipecsl - The fix you point out is documented at http://www.nokogiri.org/tutorials/installing_nokogiri.html#mac_os_x

@cabo - Does this fix work for you? If not, have you followed advice documented at http://www.nokogiri.org/tutorials/installing_nokogiri.html#mac_os_x ?

@dsedivec - Have you followed advice documented at http://www.nokogiri.org/tutorials/installing_nokogiri.html#mac_os_x ?

I apologize for asking this question multiple times, but if the docs are incomplete or don't work for you, that's EXTREMELY valuable information; and it's not clear from this thread who's aware of (and following) the advice that @zenspider has taken the time to document there.

Unless I hear back from someone on this thread that the documentation is being followed and does not address your situation, I intend to close this issue later this week.

dsedivec commented 9 years ago

@dsedivec - Have you followed advice documented at http://www.nokogiri.org/tutorials/installing_nokogiri.html#mac_os_x ?

I think so? Let me address each point in that document:

homebrew 0.9.5+

I'm not using Homebrew.

“I’m on a virginal Yosemite installation.”

I'm not on Yosemite, I'm on Mavericks.

“I see error messages about libiconv.”

I don't, like @ArloL I get:

libxml2 is missing.  Please locate mkmf.log to investigate how it is failing.

However, as I said in my previous comment, I think the problem really was with iconv. Nonetheless, I did run xcode-select --install as advised there. It had no effect, building Nokogiri still failed in the same way. I didn't run the gem commands since I never got Nokogiri installed in the first place, and I was using Bundler to install it. The brew command was not relevant to me either since I don't use Homebrew.

Make sure ruby is compiled with the latest clang compiler.

I don't really know what compiler was used for Ruby, but I prefer to believe it was what I get when I type clang, which is:

$ clang --version
Apple LLVM version 6.0 (clang-600.0.57) (based on LLVM 3.5svn)
Target: x86_64-apple-darwin13.4.0
Thread model: posix

That is the latest, to the best of my knowledge.

Ruby is no longer dependent upon gcc-4.2.

I don't think mine is? I guess I don't know how to check.

Binary gems and ruby really should be compiled with the same compiler/environment.

I think they are? I'm compiling Ruby and Nokogiri back-to-back on the same OS X system.

If you have multiple versions of xcode installed, make sure you use the right xcode-select.

I believe I only have one version of Xcode installed.

fxtentacle commented 9 years ago

Thanks @dsedivec for the solution :) I also had the error

conftest.c:7:23: error: too few arguments to function call, single argument 'cur' was not specified int t() { xmlParseDoc(); return 0; }

and

LDFLAGS=-L/opt/local/lib gem install nokogiri -v '1.6.6.2'

worked :)

cabo commented 9 years ago

Yes. The specific problem is that the test is broken.

conftest.c:15:27: error: too few arguments to function call, single argument 'cur' was not specified
int t(void) { xmlParseDoc(); return 0; }
              ~~~~~~~~~~~ ^

That makes extconf think that there is no libxml2.

This bug is somewhat masked by the many ways in which prerequisites can be broken, so there are indeed other ways to get similar errors.

But you still need to fix this particular bug.

In extconf.rb, you need to call have_func with the necessary arguments for xmlParseDoc().

pgm commented 9 years ago

I'll offer my experience since I was just struggling with this as well. I was investigating the error "conftest.c:15:27: error: too few arguments..." as well and I discovered that even if I fixed it to compile successfully, it would fail to link because the locally built libxml2 library was compiled for the wrong arch. That lead me to suspect that the wrong compiler was being used for those libraries.

(I use macports, not homebrew, so I had ignored the instructions about homebrew which mention perhaps needing to uninstall gcc.)

In anycase, I dropped macports' bin directory from my path and tried again, and finally it installed cleanly.

So, perhaps a note warning macport users as well that gcc might be picked up is worthwhile.

(Or even better would be if the install script could somehow warn that the wrong compiler was being picked up when building xml2. Could potentially avoid a lot of confusion.)

moribellamy commented 9 years ago

Got this error message when running

sudo gem install rails

on OSX Yosemite. I use MacPorts. I used the fix in @felipecsl 's post to install nokogiri, then the rest of the rails installation succeeded for me.

cabo commented 9 years ago

Just had another course of 21 students run into this issue.

Increasing the robustness of this installation process should be very, very, very high on the priorities of this project.

flavorjones commented 9 years ago

Summoning @zenspider for assistance.

zenspider commented 9 years ago

@cabo a claim of 21 datapoints w/ no data is just as bad as a claim of 1 datapoint with no data. It's just 21x sadder. I can't act on anything if I don't have details to reproduce the issue. As you can see here, nokogiri installs fine for my setup, so there's little I can do for you w/o more info:

http://www.zenspider.com/~ryan/nokogiri.mov

I forgot to include this in the video:

clang --version
Apple LLVM version 6.1.0 (clang-602.0.53) (based on LLVM 3.6.0svn)
Target: x86_64-apple-darwin14.5.0
Thread model: posix

@flavorjones I'm seriously tempted to suggest you lock this issue down so we get more people filing new (hopefully useful) bugs rather than "me too" votes.

ETA: sorry. I thought this was the OTHER (other other) "Can't build on osx" ticket. I haven't seen this one yet. I withdraw my request to lock until I've reviewed the data (if any) provided.

zenspider commented 9 years ago

@pgm good info. thanks. Totally curious question: Why are you using macports? Is there any compelling reason to stay on it these days other than inertia?

@flavorjones I will say that I am STILL having the problem where nokogiri won't install via bundler but installs fine via rubygems. I don't know if that is conflating this issue or not. Trying to get help in #bundler is like pulling teeth.

zenspider commented 9 years ago

Is this problem entirely isolated to macports? Are there any homebrew users having issues with this ticket?

cabo commented 9 years ago

@zenspider You will get continue to get tons of me-too reports until you first increase the diagnostic value of the failure message. Right now, we are all riding in Kernighan's car: The failure can be a symptom of one of maybe half of dozen of problems, and serious sleuthing is required to find out which one.

pgm commented 9 years ago

@zenspider Yes, my use of macports is probably just inertia. A long time ago, I used fink, and then after a few years moved to macports. Seems like homebrew has become the dominant packaging so I'm using that moving forward.

So my only real issue was that I couldn't tell that the root cause was a macports compiler in my path. The error I did get led me astray.

On Mon, Aug 31, 2015 at 4:38 PM, Ryan Davis notifications@github.com wrote:

@pgm https://github.com/pgm good info. thanks. Totally curious question: Why are you using macports? Is there any compelling reason to stay on it these days other than inertia?

@flavorjones https://github.com/flavorjones I will say that I am STILL having the problem where nokogiri won't install via bundler but installs fine via rubygems. I don't know if that is conflating this issue or not. Trying to get help in #bundler is like pulling teeth.

— Reply to this email directly or view it on GitHub https://github.com/sparklemotion/nokogiri/issues/1206#issuecomment-136492485 .

cabo commented 9 years ago

@pgm there are still quite a few ports that are only on macports (or are not really useful in their homebrew incarnations). I can't blame students when they are using macports to get those ports, and I think nokogiri would be ill-advised to make a concept out of not installing on those machines.

baburdick commented 9 years ago

Very much agreed with @cabo, here. I use MacPorts and Fink because I know them well and because they continue to work (MacPorts more so than Fink). I respect Homebrew. But it doesn't have what I want and need. I don't view it as a clear improvement over MacPorts. It's simply a different approach among several.

jcayzac commented 9 years ago
Package Manager: MacPorts

One thing I've noticed is that pkg-config --libs libxml2 on my system returns flags with non-absolute paths, which might be an issue when using NOKOGIRI_USE_SYSTEM_LIBRARIES, since that's MacPorts' pkg-config and the build system might think the lib is installed at standard locations, not under /opt/local)

zenspider commented 9 years ago

@pgm if you've diagnosed and fixed this while using macports, do you think you can send me a PR against the doco with tips on how to deal with it?

zenspider commented 9 years ago

@cabo I'm getting the impression that you're not interested in contributing towards this ticket. 3 comments in 24 hours with zero data to help further the cause. Can you please either contribute data from your 21 students that might help us diagnose this problem or refrain from diluting the signal?

zenspider commented 9 years ago

@jcayzac can you tell me what type pkg-config responds with? and provide the output for your pkg-config line from above?

cabo commented 9 years ago

@zenspider The data I do have is much earlier in this thread. I don't have new data from the course I'm running right now because people quickly resorted to the (somewhat questionable) workarounds documented over the last couple of years in a number of places, such as, http://stackoverflow.com/questions/19643153/error-to-install-nokogiri-on-osx-10-9-maverick

cabo commented 9 years ago

One data point from the laptop of a student that had trouble installing:

$ type pkg-config
pkg-config is /opt/local/bin/pkg-config
$ pkg-config --libs libxml2
Package libxml2 was not found in the pkg-config search path.
Perhaps you should add the directory containing `libxml2.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libxml2' found

This is just one data point, but the plot thickens around Macports. (Note that the above output from pkg-config doesn't mean that libxml2 isn't installed -- I get the same output, but sure have libxml2 2.9.2 installed in macports.)

jcayzac commented 9 years ago

@zenspider hmm never mind, I'm getting different results now: the linker flags have the right -L (although the pkg-config file is called libxml-2.0).

$ type pkg-config
pkg-config is hashed (/opt/local/bin/pkg-config)
$ port installed libxml2
The following ports are currently installed:
  libxml2 @2.9.2_2+universal (active)
$ pkg-config --libs libxml2
Package libxml2 was not found in the pkg-config search path.
Perhaps you should add the directory containing `libxml2.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libxml2' found
$ pkg-config --libs libxml-2.0
-L/opt/local/lib -lxml2

Note I'm on Yosemite, so this might not be relevant to the issue the OP reported.

cabo commented 9 years ago

@jcayzac good point. From the laptop of the student:

$ port installed libxml2
Warning: port definitions are more than two weeks old, consider updating them by running 'port selfupdate'.
The following ports are currently installed:
  libxml2 @2.9.2_2 (active)
$ pkg-config --libs libxml-2.0
-L/opt/local/lib -lxml2
flavorjones commented 9 years ago

Hi all,

It's still not clear to me how we can reproduce this issue. If the nokogiri core team can reproduce it, I promise we'll fix it, but based on my reading of the above thread, @zenspider is stymied.

@cabo - Some of your above info indicates that this might correlate with macports. Has anyone confirmed that? Has anyone seen this problem when not using macports?

I'd also like to caution everyone on this thread to check the loaded language. Let's be constructive with each other and work towards reproducing the issue and finding a solution.

jrep commented 9 years ago

I have seen this failure while using Homebrew (see my report above), but it was a long time ago and I've upgraded everything. I have tried, several times, to reproduce it with current everything, and I have not been able.

cabo commented 9 years ago

@flavorjones maybe that's the wrong question. Given that the test appears to be wrong, we're curious why it still goes through in many cases. But maybe that's not so important; you could fix the test anyway and that might solve it for all if us?

cabo commented 9 years ago

Ah. If I fix the xmlParseDoc test, I then run into the libiconv problem. One aggravating circumstance of this cluster of problems is that, from the command line, you cannot see which of the several problems you've run into; you really have to examine the mkmf.log. I'll continue to investigate this until I get it running with macports installed ahead of homebrew. Just as an indication of progress, here is the first fixlet for extconf.rb:

-  "xml2"  => ['xmlParseDoc',            'libxml/parser.h'],
+  "xml2"  => ['xmlParseDoc(NULL)',            'libxml/parser.h'],
cabo commented 9 years ago

@flavorjones I'm indeed focusing on the situation with MacPorts installed, specifically with the MacPorts bin directory ahead of the HomeBrew bin directory in ENV['PATH']. I'm running into a bit of a problem with the libiconv selection, which stops me from fixing the rest of the installation tests (have_func): I don't know how to make the tests select the same libiconv that MiniPortile selects for the libxml2/libxslt builds. So the libxml2 tests currently always fail on missing symbols from libiconv. The workaround from @dsedivec (set LDFLAGS=-L/opt/local/lib in the environment) does not help me here, as the installation tests still use -liconv, without any reference to /opt/local/lib, and that seems to point to /usr/lib.

rbbicknell commented 9 years ago

Hi all, glad I found this thread and that is still being extended at this late date. This issue appeared for me suddenly yesterday. Heretofore, I have happily compiled Nokogiri 1.6.x in the usual way with Bundler numerous times. I can show I last successfully compiled 1.6.6.2 into a Rails 4.2/Ruby 2.1.5 app on 2015-07-15, and that was the last time, until yesterday, I had attempted to do so. FWIW, Nokogiri 1.6.3.1 continues to install successfully in a Rails 3.2/Ruby 1.9.3 app. I also use MacPorts.

I do not know what changed. Possibly Xcode 6.2, but I don't remember when that update dropped and whether it was after my last known successful build in July. RVM is a recent update, but the ruby 2.1.5 was built 2014-12-22.

Here's my environment rundown:

Mac OS X Mavericks 10.9.5
RVM 1.26.11 (each app has a separate gemset)
Ruby 2.1.5p273 (2014-11-13 revision 48405) [x86_64-darwin14.0]
MacPorts 2.3.3
Xcode 6.2
Xcode command line tools 6.2 (6C131e)

xcode-select -p
/Applications/Xcode.app/Contents/Developer

gcc -v
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/usr/include/c++/4.2.1
Apple LLVM version 6.0 (clang-600.0.57) (based on LLVM 3.5svn)
Target: x86_64-apple-darwin13.4.0
Thread model: posix

clang -v
Apple LLVM version 6.0 (clang-600.0.57) (based on LLVM 3.5svn)
Target: x86_64-apple-darwin13.4.0
Thread model: posix

I looked at the advice documented at http://www.nokogiri.org/tutorials/installing_nokogiri.html#mac_os_x, however that is not applicable; I don't use HomeBrew, I'm not on Yosemite, I don't have errors about libiconv. The particular error I have is as follows:

Activating libxslt 1.1.28 (from /Users/bruce/.rvm/gems/ruby-2.1.5@apiclient-gemset-test/gems/nokogiri-1.6.6.2/ports/x86_64-apple-darwin14.0.0/libxslt/1.1.28)...
checking for main() in -llzma... yes
checking for xmlParseDoc() in libxml/parser.h... no
checking for xmlParseDoc() in -lxml2... no
checking for xmlParseDoc() in -llibxml2... no
-----
libxml2 is missing.  Please locate mkmf.log to investigate how it is failing.
-----
*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of necessary
libraries and/or headers.  Check the mkmf.log file for more details.  You may
need configuration options.

Which is the same as reported earlier in the thread. Looking in mkmf.log, in the section labeled have_func: checking for xmlParseDoc() in libxml/parser.h...

There is this:

[gcc -o conftest -- rest of invocation omitted for brevity]
Undefined symbols for architecture x86_64:
  "_libiconv", referenced from:
      _xmlIconvWrapper in libxml2.a(encoding.o)
  "_libiconv_close", referenced from:
      _xmlFindCharEncodingHandler in libxml2.a(encoding.o)
      _xmlCharEncCloseFunc in libxml2.a(encoding.o)
  "_libiconv_open", referenced from:
      _xmlFindCharEncodingHandler in libxml2.a(encoding.o)
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
checked program was:
/* begin */
 1: #include "ruby.h"
 2: 
 3: #include <libxml/parser.h>
 4: 
 5: /*top*/
 6: extern int t(void);
 7: int main(int argc, char **argv)
 8: {
 9:   if (argc > 1000000) {
10:     printf("%p", &t);
11:   }
12: 
13:   return 0;
14: }
15: int t(void) { void ((*volatile p)()); p = (void ((*)()))xmlParseDoc; return 0; }
/* end */

In the next invocation, this:

[gcc -o conftest -- rest of invocation omitted for brevity]
conftest.c:15:27: error: too few arguments to function call, single argument 'cur' was not specified
int t(void) { xmlParseDoc(); return 0; }
              ~~~~~~~~~~~ ^
/Users/bruce/.rvm/gems/ruby-2.1.5@apiclient-gemset-test/gems/nokogiri-1.6.6.2/ports/x86_64-apple-darwin14.0.0/libxml2/2.9.2/include/libxml2/libxml/parser.h:841:11: note: 'xmlParseDoc' declared here
XMLPUBFUN xmlDocPtr XMLCALL
          ^
1 error generated.
checked program was:
/* begin */
 1: #include "ruby.h"
 2: 
 3: #include <libxml/parser.h>
 4: 
 5: /*top*/
 6: extern int t(void);
 7: int main(int argc, char **argv)
 8: {
 9:   if (argc > 1000000) {
10:     printf("%p", &t);
11:   }
12: 
13:   return 0;
14: }
15: int t(void) { xmlParseDoc(); return 0; }
/* end */

The same errors are reported in the section have_library: checking for xmlParseDoc() in -lxml2... and have_library: checking for xmlParseDoc() in -llibxml2... Again, the same issues as shown earlier in the thread.

Trying gem install nokogiri -- --use-system-libraries failed to impress

Trying Nokogiri 1.6.5 produced the same result.

I do not have libxml2 installed using MacPorts

anna:apiclient$ pkg-config --libs libxml-2.0
Package libxml-2.0 was not found in the pkg-config search path.
Perhaps you should add the directory containing `libxml-2.0.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libxml-2.0' found

The workaround from @dsedivec export LDFLAGS=-L/opt/local/lib does work for me, thank goodness. Although when doing so, I don't get the usual IMPORTANT NOTICE: Building Nokogiri with a packaged version of libxml2-2.9.2

I'll check with a co-worker who has a similar configuration but on Yosemite tomorrow to see if she can continue to build Nokogiri.

Thank you all for continuing to look into this and for all your help.

cabo commented 9 years ago

@rbbicknell Do you have an easy way to find out what you installed into MacPorts since July? (e.g., feeding an appropriate prefix of ls -tr of /opt/local/bin into port provides?)

rbbicknell commented 9 years ago

I can safely say I haven't port installed anything lately. The last thing I installed was git +svn on 2014-12-29. I do run port selfupdate and port upgrade outdated from time to time, the last being on 2015-09-01. Here's a list of active ports, all of which where updated on Sep 1:

anna:~$ port installed requested | grep active
  apple-gcc42 @5666.3_15 (active)
  autoconf @2.69_5 (active)
  automake @1.15_1 (active)
  curl-ca-bundle @7.44.0_0 (active)
  git @2.5.0_0+credential_osxkeychain+doc+pcre+perl5_16+python27+svn (active)
  libksba @1.3.3_0 (active)
  libtool @2.4.6_1 (active)
  libyaml @0.1.6_1 (active)
  openssl @1.0.2d_0 (active)
  pkgconfig @0.28_0 (active)
  readline @6.3.003_1 (active)
  sqlite3 @3.8.11.1_0 (active)

I think I recall that I had to tell RVM to build ruby 2 using a MacPorts compiler on a Yosemite box, but I don't have those notes here. I don't have any notes about doing anything unusual when I built Ruby on this box.

dlech commented 9 years ago

I hope I am not just being a "me too". I ran into this same issue, but I don't use MacPorts, so I thought it worth mentioning. I was trying to install on OS X 10.10.5 using the built-in ruby and XCode 7. I do have homebrew installed.

gerda:ev3dev.github.io david$ ruby --version
ruby 2.0.0p481 (2014-05-08 revision 45883) [universal.x86_64-darwin14]
gerda:ev3dev.github.io david$ clang --version
Apple LLVM version 7.0.0 (clang-700.0.72)
Target: x86_64-apple-darwin14.5.0

I could see in the successful install (.mov) from @zenspider, he was using ruby 2.2, so I installed rbenv using the instructions from here. After that, I was able to get through the installation without error.

So maybe the trick to reproducing this issue is to not use rbenv or rvm.

cabo commented 9 years ago

@dlech Be careful with the "same issue" -- there are a number of issues that manifest themselves in the same symptom. Can you pastebin the mkmf.log from the failed attempt?

dlech commented 9 years ago

Sure: https://gist.github.com/dlech/7203f1be7695033a31b3

cabo commented 9 years ago

@dlech The include files from the newly built libxml2 are not found by the rest of Nokogiri's build process. Looking at the log, I'm wondering what /BuildRoot is.