Error compiling vg on MacOS with gcc 9, due to failing dependency build (raptor).

[yakneens]

Please describe:

1. What you were trying to do Build on a Mac with GCC 9.

2. What you wanted to happen Git clone and compile

3. What actually happened Error compiling a dependency

4. What data and command line to use to make the problem recur, if applicable make CC=gcc-9 CXX=g++-9

siakhnin@mac-korbel21-5:~/tools/vg$ make CC=gcc-9 CXX=g++-9
cd deps/raptor/build && cmake .. && rm -f src/turtle_parser.c && rm -f src/turtle_lexer.c && make turtle_lexer_tgt && make -f src/CMakeFiles/raptor2.dir/build.make src/turtle_lexer.c && sed -i.bak '/yycleanup/d' src/turtle_lexer.c && /Applications/Xcode.app/Contents/Developer/usr/bin/make 2>&1 | python /Users/siakhnin/tools/vg/scripts/filter-noisy-assembler-warnings.py && cp src/libraptor2.a /Users/siakhnin/tools/vg/lib && mkdir -p /Users/siakhnin/tools/vg/include/raptor2 && cp src/*.h /Users/siakhnin/tools/vg/include/raptor2
-- 
################################################################
Raptor Configuration Summary
################################################################

    http://librdf.org/raptor/

    Configured on host  
      host OS                   Darwin
      host architecture         x86_64

    General flags:
      CC                        /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc
      CXX (not used)            
      CFLAGS                    
      CXXFLAGS                  
      LDFLAGS                   -L/usr/local/opt/llvm/lib -Wl,-rpath,/usr/local/opt/llvm/lib

    Installation prefix:        /usr/local

    Dependencies (packages marked with *** are required):

  *** Perl                        /usr/bin/perl
  *** Bison                       /usr/local/opt/bison/bin/bison
  *** Flex                        /usr/bin/flex
  *** libxml2 (lib)               /usr/lib/libxml2.dylib
  *** libxml2 (include)           /usr/include/libxml2
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/siakhnin/tools/vg/deps/raptor
[100%] Built target turtle_flex_tgt
[100%] Built target turtle_lexer_tgt
Generating turtle_lexer.c
Built target turtle_tables_tgt
Generating turtle_parser.c
Generating turtle_parser.h
Error renaming from "/Users/siakhnin/tools/vg/deps/raptor/build/src/turtle_parser.tab.h" to "/Users/siakhnin/tools/vg/deps/raptor/build/src/turtle_parser.h": No such file or directory
make[3]: *** [src/turtle_parser.h] Error 1
make[2]: *** [src/CMakeFiles/turtle_parser_tgt.dir/all] Error 2
make[1]: *** [all] Error 2
make: *** [lib/libraptor2.a] Error 2

[ekg]

And we only need raptor for an application that should probably be factored into its own repository (semantic translation of variation graphs).

[adamnovak]

Hm. It looks like your Bison is not putting its output files where they should be as *.tab.c/.h (i.e. not running in Yacc mode).

Did you install your Bison in a particularly special way? /usr/local/opt/bison/bin/bison looks like a bit of a strange place for it.

I just made a PR #2295 that might fix this, by telling Bison to explicitly act like Yacc and make the .tab files.

Can you test it @llevar? It should be something like:

git fetch origin
git checkout origin/fix-raptor
git submodule update --init --recursive
cd deps/raptor
rm -Rf build
git checkout build
cd ../..
make

[yakneens]

@adamnovak, thank you for looking at this. I tried your patch, but am still getting the same error.

As to your question about bison, I'm pretty sure that I installed it via homebrew.

siakhnin@mac-korbel21-5:/usr/local/opt/bison$ cat INSTALL_RECEIPT.json 
{"homebrew_version":"2.1.3-13-g2d445ae","used_options":[],"unused_options":[],"built_as_bottle":true,"poured_from_bottle":true,"installed_as_dependency":false,"installed_on_request":true,"changed_files":["bin/yacc","ChangeLog","INSTALL_RECEIPT.json","NEWS","share/doc/bison/NEWS"],"time":1559579259,"source_modified_time":1558503937,"HEAD":null,"stdlib":null,"compiler":"clang","aliases":["bison@3.4"],"runtime_dependencies":[],"source":{"path":"/usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/bison.rb","tap":"homebrew/core","spec":"stable","versions":{"stable":"3.4.1","devel":"","head":"","version_scheme":0}}}

I forget why it ended up at that particular path, to be honest. I think I tried building raptor separately from vg, to isolate the issue, and then it was having problems finding the updated bison, as Mac OS has a very old version that can't be upgraded.

Let me know if you have any other suggestions, and thanks again for looking at the issue.

[adamnovak]

Hm.

The only thing I could suggest is to try manually running bison on one of the Raptor .y files and seeing if it makes the .tab.c/.h files properly.

[adamnovak]

OK, I've installed GCC 9.1 and Bison 3.3.2 (the latest from Macports) alongside Clang 8.0 on a Mac. I've set GCC as the default compiler and Macports Clang as the default Clang. I'm not able to reproduce this issue, although I did hack up the VG build system a little to make it show the right compiler to Raptor. I've updated my PR.

[adamnovak]

@llevar Can you test current vg master again? I've made some changes to it, and I have it compiling fine on GCC 9.1 and Bison 3.3.2. I made some changes to how vg invokes the raptor build that might improve things.

[yakneens]

Thanks for try to address this guys. I'm currently on vacation so won't be able to test until next week. Will try as soon as I am back.

On Mon, 10 Jun 2019, 19:22 Adam Novak, notifications@github.com wrote:

@llevar https://github.com/llevar Can you test current vg master again? I've made some changes to it, and I have it compiling fine on GCC 9.1 and Bison 3.3.2. I made some changes to how vg invokes the raptor build that might improve things.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/vgteam/vg/issues/2292?email_source=notifications&email_token=ABMVGB6E5EIQMUGRHZ42T6LPZ2EVZA5CNFSM4HS2KCZKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXKRE4Q#issuecomment-500503154, or mute the thread https://github.com/notifications/unsubscribe-auth/ABMVGB7GK43UH6WDNZPIU43PZ2EVZANCNFSM4HS2KCZA .

[matt-shenton]

Hi VG team,

I seem to have hit a similar problem installing on Macos Catalina 10.15.6

gcc (Macports gcc9 9.3.0_1) 9.3.0 bison (GNU Bison) 3.7 (also from macports, at /opt/local/bin/bison)

I tried to build as directed with . ./source_me.sh && make

Many thanks in advance for your help Best

Matt Shenton

output below:

Scanning dependencies of target turtle_tables_tgt
[  1%] Generating turtle_parser.tab.c
/Users/mashus096/vg/deps/raptor/src/turtle_parser.y:118.1-29: warning: deprecated directive: `%name-prefix "turtle_parser_"', use `%define api.prefix {turtle_parser_}' [-Wdeprecated]
  118 | %name-prefix "turtle_parser_"
      | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      | %define api.prefix {turtle_parser_}
/Users/mashus096/vg/deps/raptor/src/turtle_parser.y: warning: fix-its can be applied.  Rerun with option '--update'. [-Wother]
[  1%] Built target turtle_tables_tgt
Scanning dependencies of target turtle_parser_tgt
[  2%] Generating turtle_parser.c
[  3%] Generating turtle_parser.h
[  3%] Built target turtle_parser_tgt
Scanning dependencies of target parsedate_tables_tgt
[  3%] Generating parsedate.tab.c
/Users/mashus096/vg/deps/raptor/src/parsedate.y:167.1-32: warning: deprecated directive: `%name-prefix "raptor_parsedate_"', use `%define api.prefix {raptor_parsedate_}' [-Wdeprecated]
  167 | %name-prefix "raptor_parsedate_"
      | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      | %define api.prefix {raptor_parsedate_}
/Users/mashus096/vg/deps/raptor/src/parsedate.y: warning: fix-its can be applied.  Rerun with option '--update'. [-Wother]
[  3%] Built target parsedate_tables_tgt
Scanning dependencies of target parsedate_tgt
[  4%] Generating parsedate.c
[  5%] Generating parsedate.h
[  5%] Built target parsedate_tgt
[  6%] Built target turtle_flex_tgt
[  6%] Built target turtle_lexer_tgt
Scanning dependencies of target raptor2
[  6%] Building C object src/CMakeFiles/raptor2.dir/raptor_avltree.c.o
[  7%] Building C object src/CMakeFiles/raptor2.dir/raptor_concepts.c.o
[  8%] Building C object src/CMakeFiles/raptor2.dir/raptor_escaped.c.o
[  9%] Building C object src/CMakeFiles/raptor2.dir/raptor_general.c.o
[ 10%] Building C object src/CMakeFiles/raptor2.dir/raptor_iostream.c.o
[ 11%] Building C object src/CMakeFiles/raptor2.dir/raptor_json_writer.c.o
[ 12%] Building C object src/CMakeFiles/raptor2.dir/raptor_locator.c.o
[ 13%] Building C object src/CMakeFiles/raptor2.dir/raptor_log.c.o
[ 14%] Building C object src/CMakeFiles/raptor2.dir/raptor_memstr.c.o
[ 14%] Building C object src/CMakeFiles/raptor2.dir/raptor_namespace.c.o
[ 15%] Building C object src/CMakeFiles/raptor2.dir/raptor_option.c.o
[ 16%] Building C object src/CMakeFiles/raptor2.dir/raptor_parse.c.o
[ 17%] Building C object src/CMakeFiles/raptor2.dir/raptor_qname.c.o
[ 18%] Building C object src/CMakeFiles/raptor2.dir/raptor_rfc2396.c.o
[ 19%] Building C object src/CMakeFiles/raptor2.dir/raptor_sax2.c.o
[ 20%] Building C object src/CMakeFiles/raptor2.dir/raptor_sequence.c.o
[ 21%] Building C object src/CMakeFiles/raptor2.dir/raptor_serialize.c.o
[ 22%] Building C object src/CMakeFiles/raptor2.dir/raptor_set.c.o
[ 22%] Building C object src/CMakeFiles/raptor2.dir/raptor_statement.c.o
[ 23%] Building C object src/CMakeFiles/raptor2.dir/raptor_stringbuffer.c.o
[ 24%] Building C object src/CMakeFiles/raptor2.dir/raptor_syntax_description.c.o
[ 25%] Building C object src/CMakeFiles/raptor2.dir/raptor_term.c.o
[ 26%] Building C object src/CMakeFiles/raptor2.dir/raptor_turtle_writer.c.o
[ 27%] Building C object src/CMakeFiles/raptor2.dir/raptor_unicode.c.o
[ 28%] Building C object src/CMakeFiles/raptor2.dir/raptor_uri.c.o
[ 29%] Building C object src/CMakeFiles/raptor2.dir/raptor_www.c.o
[ 29%] Building C object src/CMakeFiles/raptor2.dir/raptor_xml.c.o
[ 30%] Building C object src/CMakeFiles/raptor2.dir/raptor_xml_writer.c.o
[ 31%] Building C object src/CMakeFiles/raptor2.dir/snprintf.c.o
[ 32%] Building C object src/CMakeFiles/raptor2.dir/sort_r.c.o
[ 33%] Building C object src/CMakeFiles/raptor2.dir/turtle_common.c.o
[ 34%] Building C object src/CMakeFiles/raptor2.dir/ntriples_parse.c.o
[ 35%] Building C object src/CMakeFiles/raptor2.dir/raptor_ntriples.c.o
[ 36%] Building C object src/CMakeFiles/raptor2.dir/turtle_lexer.c.o
[ 37%] Building C object src/CMakeFiles/raptor2.dir/turtle_parser.c.o
turtle_parser.c:176:10: fatal error: turtle_parser.tab.h: No such file or directory
  176 | #include "turtle_parser.tab.h"
      |          ^~~~~~~~~~~~~~~~~~~~~
compilation terminated.
make[3]: *** [src/CMakeFiles/raptor2.dir/turtle_parser.c.o] Error 1
make[2]: *** [src/CMakeFiles/raptor2.dir/all] Error 2
make[1]: *** [all] Error 2
make: *** [lib/libraptor2.a] Error 2

[adamnovak]

@matt-shenton We haven't been trying to support GNU GCC on Mac for a while (only Apple Clang), although I don't see how that would really cause this. But if you do try to continue on with GNU GCC, you're going to need to make sure that your system-installed Protobuf library is linked against the C++ standard library that GNU GCC links with, or it won't be able to be used by vg.

It sounds like Raptor might just not build right on new Bison?

Maybe the new Bison can update the .y files automatically?

/Users/mashus096/vg/deps/raptor/src/turtle_parser.y: warning: fix-its can be applied.  Rerun with option '--update'. [-Wother]

I'm a little worried that the changes that we need for Bison 3.7 might not quite work right on Bison 3.0, which we still need to support.

Bison 3.7 still supports the -y option to run in YACC mode, right?

The underlying problem is that nobody's maintaining raptor anymore, and we're trying to build it from source on a bunch of machines. What we really should do is rip out the vendored raptor, and demand the libraptor2-dev Debian package or the raptor2 Macports package or the raptor Homebrew package. It seems to be in distros; we should try and make it the distros' problem to get raptor to build.

[adamnovak]

OK, this is why we maybe can't just use distribution Raptor:

$ ldd /usr/lib/x86_64-linux-gnu/libraptor2.so
    linux-vdso.so.1 (0x00007ffde5d8c000)
    libcurl-gnutls.so.4 => /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4 (0x00007fb5f6b90000)
    libxslt.so.1 => /usr/lib/x86_64-linux-gnu/libxslt.so.1 (0x00007fb5f6953000)
    libxml2.so.2 => /usr/lib/x86_64-linux-gnu/libxml2.so.2 (0x00007fb5f6592000)
    libyajl.so.2 => /usr/lib/x86_64-linux-gnu/libyajl.so.2 (0x00007fb5f6388000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fb5f6169000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fb5f5d78000)
    libnghttp2.so.14 => /usr/lib/x86_64-linux-gnu/libnghttp2.so.14 (0x00007fb5f5b53000)
    libidn2.so.0 => /usr/lib/x86_64-linux-gnu/libidn2.so.0 (0x00007fb5f5936000)
    librtmp.so.1 => /usr/lib/x86_64-linux-gnu/librtmp.so.1 (0x00007fb5f571a000)
    libpsl.so.5 => /usr/lib/x86_64-linux-gnu/libpsl.so.5 (0x00007fb5f550c000)
    libnettle.so.6 => /usr/lib/x86_64-linux-gnu/libnettle.so.6 (0x00007fb5f52d6000)
    libgnutls.so.30 => /usr/lib/x86_64-linux-gnu/libgnutls.so.30 (0x00007fb5f4f70000)
    libgssapi_krb5.so.2 => /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2 (0x00007fb5f4d25000)
    libldap_r-2.4.so.2 => /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2 (0x00007fb5f4ad3000)
    liblber-2.4.so.2 => /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2 (0x00007fb5f48c5000)
    libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fb5f46a8000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fb5f430a000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fb5f4106000)
    libicuuc.so.60 => /usr/lib/x86_64-linux-gnu/libicuuc.so.60 (0x00007fb5f3d4e000)
    liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007fb5f3b28000)
    /lib64/ld-linux-x86-64.so.2 (0x00007fb5f706a000)
    libunistring.so.2 => /usr/lib/x86_64-linux-gnu/libunistring.so.2 (0x00007fb5f37aa000)
    libhogweed.so.4 => /usr/lib/x86_64-linux-gnu/libhogweed.so.4 (0x00007fb5f3576000)
    libgmp.so.10 => /usr/lib/x86_64-linux-gnu/libgmp.so.10 (0x00007fb5f32f5000)
    libp11-kit.so.0 => /usr/lib/x86_64-linux-gnu/libp11-kit.so.0 (0x00007fb5f2fc6000)
    libtasn1.so.6 => /usr/lib/x86_64-linux-gnu/libtasn1.so.6 (0x00007fb5f2db3000)
    libkrb5.so.3 => /usr/lib/x86_64-linux-gnu/libkrb5.so.3 (0x00007fb5f2add000)
    libk5crypto.so.3 => /usr/lib/x86_64-linux-gnu/libk5crypto.so.3 (0x00007fb5f28ab000)
    libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 (0x00007fb5f26a7000)
    libkrb5support.so.0 => /usr/lib/x86_64-linux-gnu/libkrb5support.so.0 (0x00007fb5f249c000)
    libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007fb5f2281000)
    libsasl2.so.2 => /usr/lib/x86_64-linux-gnu/libsasl2.so.2 (0x00007fb5f2066000)
    libgssapi.so.3 => /usr/lib/x86_64-linux-gnu/libgssapi.so.3 (0x00007fb5f1e25000)
    libicudata.so.60 => /usr/lib/x86_64-linux-gnu/libicudata.so.60 (0x00007fb5f027c000)
    libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fb5efef3000)
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fb5efcdb000)
    libffi.so.6 => /usr/lib/x86_64-linux-gnu/libffi.so.6 (0x00007fb5efad3000)
    libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1 (0x00007fb5ef8cf000)
    libheimntlm.so.0 => /usr/lib/x86_64-linux-gnu/libheimntlm.so.0 (0x00007fb5ef6c6000)
    libkrb5.so.26 => /usr/lib/x86_64-linux-gnu/libkrb5.so.26 (0x00007fb5ef439000)
    libasn1.so.8 => /usr/lib/x86_64-linux-gnu/libasn1.so.8 (0x00007fb5ef197000)
    libhcrypto.so.4 => /usr/lib/x86_64-linux-gnu/libhcrypto.so.4 (0x00007fb5eef61000)
    libroken.so.18 => /usr/lib/x86_64-linux-gnu/libroken.so.18 (0x00007fb5eed4b000)
    libwind.so.0 => /usr/lib/x86_64-linux-gnu/libwind.so.0 (0x00007fb5eeb22000)
    libheimbase.so.1 => /usr/lib/x86_64-linux-gnu/libheimbase.so.1 (0x00007fb5ee913000)
    libhx509.so.5 => /usr/lib/x86_64-linux-gnu/libhx509.so.5 (0x00007fb5ee6c9000)
    libsqlite3.so.0 => /usr/lib/x86_64-linux-gnu/libsqlite3.so.0 (0x00007fb5ee3c0000)
    libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007fb5ee188000)

Ubuntu ships a libraptor2.a, but it built Raptor to link against a screen full of libraries, including e.g. libcurl and a bunch of DNS stuff that I don't think is very happy linking statically, and doesn't expose, as far as I can tell, a pkg-config or anything you might be able to use to get the -l flags you would need to actually use the libraptor2.a that they provide. Even if we did collect them all manually, and all those libraries themselves had static versions available, we'd run into trouble if we ever tried to do the static build on a system that built raptor in a slightly different configuration.

@JervenBolleman Is it feasible to sever the Raptor dependency and keep support for Turtle files? Is there another, smaller library we could use? Can we hand-code a parser? Can we pull out Turtle/vg interconversion into another repo that can just use libvgio and libhandlegraph and system libraptor2 and doesn't need to be able to build statically linked for single-binary distribution?

[JervenBolleman]

TL;DR; Losing reading turtle is ok, losing writing turtle would make me very sad ;(

@adamnovak it was a requirement from @ekg that we should be able to roundtrip all VG formats. Now I think reading turtle is not needed as it is not generic turtle that can be read, only the specific output as generated by VG writer in that specific order.

As a note it would be nice if you would at least use the same libhandlegraph .a between all VG projects and that would be extern/C friendly.

[adamnovak]

We definitely would be able to write turtle with a hand-coded writer. But writing a format we can't read, or that we can only read a restricted subset of that only we can generate, sounds like a bad idea.

We definitely want to use a consistent libhandlegraph in all projects. If we're not it's because we haven't gotten aroudn to updating submodule pointers.

But it's in C++ and is probably always going to need to link against the C++ standard library. We've been thinking about coming up with a C wrapper interface so you can use it from C or other languages that can't directly bind C++, but we haven't actually designed one yet.

I think the way to go on this issue for now might be using system Raptor on Mac (where we don't have to worry about static linking) and vendored-in Raptor on Linux (where the environment seems to be a bit more consistent).

On 7/29/20, JervenBolleman notifications@github.com wrote:

TL;DR; Losing reading turtle is ok, losing writing turtle would make me very sad ;(

@adamnovak it was a requirement from @ekg that we should be able to roundtrip all VG formats. Now I think reading turtle is not needed as it is not generic turtle that can be read, only the specific output as generated by VG writer in that specific order.

As a note it would be nice if you would at least use the same libhandlegraph .a between all VG projects and that would be extern/C friendly.

-- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: https://github.com/vgteam/vg/issues/2292#issuecomment-665798048

[matt-shenton]

Thanks a lot for looking at it.

I'm going to try and use docker for the moment.

Best wishes Matt