Build fails when using Bison 3.2

[jasonboggess]

Attempted to install via pip on macos 10.14 using python 3.7 and pip3 version 18.0

Got the following error:

    /private/var/folders/_9/fz23mhs14q135p8kh6f4dg980000gn/T/pip-install-g5ph25wz/yaramod/build/src/yaramod/yy/yy_parser.hpp:738:102: error: too many arguments provided to function-like macro invocation
          basic_symbol (typename Base::kind_type t, YY_RVREF (std::pair<nonstd::optional<std::uint64_t>, nonstd::optional<std::uint64_t>>) v, YY_RVREF (location_type) l);

[s3rvac]

Hi. Could you please try building yaramod and its Python bindings manually by following these two steps?

1. First, ensure that you have Flex >= 2.6 and Bison >= 3.0 and set proper paths (ensure that they exist on your system):

   # Apple ships old Flex & Bison, so Homebrew versions should be used.
   export CMAKE_INCLUDE_PATH="/usr/local/opt/flex/include"
   export CMAKE_LIBRARY_PATH="/usr/local/opt/flex/lib;/usr/local/opt/bison/lib"
   export PATH="/usr/local/opt/flex/bin:/usr/local/opt/bison/bin:$PATH"

2. Clone the repo and run the build:

   git clone https://github.com/avast-tl/yaramod.git
   cd yaramod
   mkdir build
   cd build
   cmake -DYARAMOD_PYTHON=ON ..
   make

If the build fails, please post the output here. If the build passes, could you please try running pip install [..] again but after setting the above variables?

BTW, which of the following two commands did you run?

pip install git+https://github.com/avast-tl/yaramod.git
pip install yaramod

[jasonboggess]

Hi! Thanks for the super fast reply. I did the above steps (cloned the repo and tried to build manually) and still got the same errors.

Used homebrew to install flex and bison.

Here are what versions I'm using and what my variables ended up to be:

$ flex --version
flex 2.6.4
$ bison --version
bison (GNU Bison) 3.2
Written by Robert Corbett and Richard Stallman.

Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
$ echo $CMAKE_INCLUDE_PATH
/usr/local/opt/flex/include
$ echo $CMAKE_LIBRARY_PATH
/usr/local/opt/flex/lib;/usr/local/opt/bison/lib
$ cmake --version
cmake version 3.12.3

CMake suite maintained and supported by Kitware (kitware.com/cmake).
$ make --version
GNU Make 3.81
Copyright (C) 2006  Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.

This program built for i386-apple-darwin11.3.0
$ gcc --version
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/usr/include/c++/4.2.1
Apple LLVM version 10.0.0 (clang-1000.11.45.5)
Target: x86_64-apple-darwin18.0.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

And when running make:

$ make
[  3%] Building CXX object src/CMakeFiles/yaramod.dir/builder/yara_file_builder.cpp.o
In file included from yaramod/src/builder/yara_file_builder.cpp:8:
In file included from yaramod/include/yaramod/parser/parser_driver.h:13:
In file included from yaramod/include/yaramod/parser/lexer.h:13:
yaramod/build/src/yaramod/yy/yy_parser.hpp:738:102: error: too many arguments provided to function-like macro invocation
      basic_symbol (typename Base::kind_type t, YY_RVREF (std::pair<nonstd::optional<std::uint64_t>, nonstd::optional<std::uint64_t>>) v, YY_RVREF (location_type) l);
                                                                                                     ^
yaramod/build/src/yaramod/yy/yy_parser.hpp:80:10: note: macro 'YY_RVREF' defined here
# define YY_RVREF(Type)    Type&&
         ^
yaramod/build/src/yaramod/yy/yy_parser.hpp:738:49: error: unknown type name 'YY_RVREF'
      basic_symbol (typename Base::kind_type t, YY_RVREF (std::pair<nonstd::optional<std::uint64_t>, nonstd::optional<std::uint64_t>>) v, YY_RVREF (location_type) l);
                                                ^
yaramod/build/src/yaramod/yy/yy_parser.hpp:1209:76: error: too many arguments provided to function-like macro invocation
    make_REGEXP_RANGE (YY_COPY (std::pair<nonstd::optional<std::uint64_t>, nonstd::optional<std::uint64_t>>) v, YY_COPY (location_type) l);
                                                                           ^
yaramod/build/src/yaramod/yy/yy_parser.hpp:81:10: note: macro 'YY_COPY' defined here
# define YY_COPY(Type)     Type
         ^
yaramod/build/src/yaramod/yy/yy_parser.hpp:1209:24: error: unknown type name 'YY_COPY'
    make_REGEXP_RANGE (YY_COPY (std::pair<nonstd::optional<std::uint64_t>, nonstd::optional<std::uint64_t>>) v, YY_COPY (location_type) l);
                       ^
yaramod/build/src/yaramod/yy/yy_parser.hpp:1632:128: error: too many arguments provided to function-like macro invocation
   Parser ::basic_symbol<Base>::basic_symbol (typename Base::kind_type t, YY_RVREF (std::pair<nonstd::optional<std::uint64_t>, nonstd::optional<std::uint64_t>>) v, YY_RVREF (location_type) l)
                                                                                                                               ^
yaramod/build/src/yaramod/yy/yy_parser.hpp:80:10: note: macro 'YY_RVREF' defined here
# define YY_RVREF(Type)    Type&&
         ^
yaramod/build/src/yaramod/yy/yy_parser.hpp:1632:75: error: unknown type name 'YY_RVREF'
   Parser ::basic_symbol<Base>::basic_symbol (typename Base::kind_type t, YY_RVREF (std::pair<nonstd::optional<std::uint64_t>, nonstd::optional<std::uint64_t>>) v, YY_RVREF (location_type) l)
                                                                          ^
yaramod/build/src/yaramod/yy/yy_parser.hpp:2717:84: error: too many arguments provided to function-like macro invocation
   Parser ::make_REGEXP_RANGE (YY_COPY (std::pair<nonstd::optional<std::uint64_t>, nonstd::optional<std::uint64_t>>) v, YY_COPY (location_type) l)
                                                                                   ^
yaramod/build/src/yaramod/yy/yy_parser.hpp:81:10: note: macro 'YY_COPY' defined here
# define YY_COPY(Type)     Type
         ^
yaramod/build/src/yaramod/yy/yy_parser.hpp:2717:32: error: unknown type name 'YY_COPY'
   Parser ::make_REGEXP_RANGE (YY_COPY (std::pair<nonstd::optional<std::uint64_t>, nonstd::optional<std::uint64_t>>) v, YY_COPY (location_type) l)
                               ^
yaramod/build/src/yaramod/yy/yy_parser.hpp:1484:7: error: no matching constructor for initialization of 'yaramod::yy::Parser::by_type'
    : Base (YY_MOVE (other))
      ^     ~~~~~~~~~~~~~~~
yaramod/build/src/yaramod/yy/yy_parser.hpp:2103:12: note: in instantiation of member function 'yaramod::yy::Parser::basic_symbol<yaramod::yy::Parser::by_type>::basic_symbol' requested here
    return symbol_type (token::END, YY_MOVE (l));
           ^
yaramod/build/src/yaramod/yy/yy_parser.hpp:2043:22: note: candidate constructor not viable: no known conversion from 'typename remove_reference<basic_symbol<by_type> &>::type' (aka 'yaramod::yy::Parser::basic_symbol<yaramod::yy::Parser::by_type>') to 'const yaramod::yy::Parser::by_type' for 1st argument
   Parser ::by_type::by_type (const by_type& other)
                     ^
yaramod/build/src/yaramod/yy/yy_parser.hpp:2048:22: note: candidate constructor not viable: no known conversion from 'typename remove_reference<basic_symbol<by_type> &>::type' (aka 'yaramod::yy::Parser::basic_symbol<yaramod::yy::Parser::by_type>') to 'yaramod::yy::Parser::token_type' (aka 'yaramod::yy::Parser::token::yytokentype') for 1st argument
   Parser ::by_type::by_type (token_type t)
                     ^
yaramod/build/src/yaramod/yy/yy_parser.hpp:2038:22: note: candidate constructor not viable: requires 0 arguments, but 1 was provided
   Parser ::by_type::by_type ()
                     ^
yaramod/build/src/yaramod/yy/yy_parser.hpp:1488:13: error: cannot initialize object parameter of type 'const yaramod::yy::Parser::by_type' with an expression of type 'yaramod::yy::Parser::basic_symbol<yaramod::yy::Parser::by_type>'
    switch (other.type_get ())
            ^~~~~
10 errors generated.
make[2]: *** [src/CMakeFiles/yaramod.dir/builder/yara_file_builder.cpp.o] Error 1
make[1]: *** [src/CMakeFiles/yaramod.dir/all] Error 2
make: *** [all] Error 2

FYI I ran:

pip3 install yaramod

[metthal]

I am not sure if it'll help because the errors are from bison generated parser, but I see some nonstd::optional in the output so I tried to update optional-lite to the newest version in new-optional-lite branch. Can you please try to compile this branch? Thanks.

[jasonboggess]

@metthal I deleted the build directory, and checked out your branch, but same errors.

[metthal]

It may have something to do with this. From what I see, you use bison 3.2 which was rolled out just few days ago so this might be some bug in the new release which showed itself only with our parser definition. If this is an option for you, could you please try to revert bison to 3.1 or 3.0.x? I'll try to upgrade to bison 3.2 on my machine as soon as I get to it. If it's really problem in the new release, we might need to file a bug there.

[s3rvac]

@metthal Good catch. The issue is really caused by Bison 3.2. I was able to reproduce it even on Ubuntu inside Docker via the following Dockerfile:

FROM ubuntu:bionic

RUN apt-get -y update && \
    apt-get install -y build-essential cmake git flex wget

RUN wget https://ftp.gnu.org/gnu/bison/bison-3.2.tar.gz && \
    tar xf bison-3.2.tar.gz && \
    cd bison-3.2 && \
    ./configure && \
    make -j$(nproc) && \
    make install

RUN git clone https://github.com/avast-tl/yaramod && \
    mkdir yaramod/build && \
    cd yaramod/build && \
    cmake .. && \
    make -j$(nproc)

When you run

$ docker build -t yaramod .

the build fails with

In file included from /yaramod/include/yaramod/parser/lexer.h:13:0,
                 from /yaramod/include/yaramod/parser/parser_driver.h:13,
                 from /yaramod/src/builder/yara_file_builder.cpp:8:
/yaramod/build/src/yaramod/yy/yy_parser.hpp:738:134: error: macro "YY_RVREF" passed 2 arguments, but takes just 1
       basic_symbol (typename Base::kind_type t, YY_RVREF (std::pair<nonstd::optional<std::uint64_t>, nonstd::optional<std::uint64_t>>) v, YY_RVREF (location_type) l);
                                                                                                                                      ^
[..]

[jasonboggess]

I manually installed 3.1.0 and 3.0.5 of bison and I still see the same issue. I made sure that bison --version was correct and that my env variables were set:

export CMAKE_INCLUDE_PATH="/usr/local/opt/flex/include"
export CMAKE_LIBRARY_PATH="/usr/local/opt/flex/lib;/usr/local/opt/bison/lib"
export PATH="/usr/local/opt/flex/bin:/usr/local/opt/bison/bin:$PATH"

[s3rvac]

This is weird. With 3.1.0 and 3.0.5, the build succeeds for me. It only fails with 3.2.0.

[s3rvac]

@jasonboggess Based on the output you have provided in the (now deleted) comment, it seems to me that you are running cmake/make from a build directory in which you have already run cmake/make (maybe with Bison 3.2?). Could you please try running cmake .. and make from a completely clean build directory? If you indeed have Bison 3.1, cmake .. should print the following line

-- Found BISON: /usr/local/opt/bison/bin (found version "3.1")

and the build should succeed.

[jasonboggess]

Yes, I think that was it. Thanks!

[s3rvac]

Great :+1: We will leave the issue open until we fix the build with Bison 3.2.

[metthal]

I've resolved the problem with bison 3.2. The should now be able to compile yaramod while having bison 3.2.

[slw07g]

I'm having the issue with bison 3.3 (the version homebrew installed)

[metthal]

What kind of issues? I see that some of the directives we use are now deprecated but it should work just fine despite the warnings. I'll fix the warnings, thanks for pointing them out, but you should still be able to compile and use it.

[slw07g]

$> cmake .
...
-- Found FLEX: /usr/local/homebrew/bin/flex (found version "2.6.4") 
-- Found BISON: /usr/local/homebrew/bin/bison (found version "3.3.2") 
...
$> make
...
$> cat external/src/yaramod-project-stamp/yaramod-project-build-err.log 
parser/yy/parser.y:7.1-5: invalid directive: `%code'
parser/yy/parser.y:7.7-14: syntax error, unexpected identifier
make[5]: *** [src/yaramod/yy/yy_parser.cpp] Error 1
make[4]: *** [src/CMakeFiles/yaramod.dir/all] Error 2
make[3]: *** [all] Error 2
~                               

[metthal]

This error reminds me of reports we had when pre-3 version of bison was used instead. Could you please paste here contents of external/src/yaramod-project-stamp/yaramod-project-configure-out.log?

[slw07g]

-- Configuring done
-- Generating done
-- Build files have been written to: /Users/<username>/retdec/external/src/yaramod-project-build

[metthal]

Is that everything? Because my looks like this

-- The CXX compiler identification is GNU 8.2.1
-- Check for working CXX compiler: /usr/bin/c++ 
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found FLEX: /usr/bin/flex (found version "2.6.1").
-- Found BISON: /usr/bin/bison (found version "3.0.4").
-- Configuring done
-- Generating done
-- Build files have been written to: /home/<username>/projects/retdec/build/external/src/yaramod-project-build

Could you please try to remove external/src/yaramod-project* completely and run cmake and then make one more time?

[slw07g]

Yes, that was everything in the file.

I was able to build yaramod-project succesfully after following your suggestion. (Perhaps that should be a part of make clean?)

The commands I ran:

rm -rf external/src/yaramod-project-*
cmake .
make