search

manticoresoftware / manticoresearch

Easy to use open source fast database for search | Good alternative to Elasticsearch now | Drop-in replacement for E in the ELK soon
https://manticoresearch.com
GNU General Public License v3.0
9.04k stars 507 forks source link

Docker - Alpine Linux - Compiling tar.gz - Patching RE2 /usr/bin/patch: unrecognized option: binary #128

Closed sshilko closed 6 years ago

sshilko commented 6 years ago

Are you in the right place?

Yes

For general questions, post a question to Forum or Slack

If you open a Github issue, it must be a bug or feature request and you need to provide the below information.

Describe the environment

Alpine Linux 3.7 (Docker)

Manticore Search version: 2.7.3 and 2.7.4

OS version: OSX host Build version: Github release (not github clone)

Describe the problem

Compiling from github tar.gz tarball has issues with RE2 Probably due to

https://github.com/manticoresoftware/manticoresearch/blob/master/cmake/GetRE2.cmake#L94 /usr/bin/patch -p1 --binary -i under Alpine

-- Unpack RE2 from /build/manticore/build/master.zip
-- Patching RE2
/usr/bin/patch: unrecognized option: binary
BusyBox v1.27.2 (2017-12-12 10:41:50 GMT) multi-call binary.

Usage: patch [OPTIONS] [ORIGFILE [PATCHFILE]]

    -p,--strip N        Strip N leading components from file names
    -i,--input DIFF     Read DIFF instead of stdin
    -R,--reverse        Reverse patch
    -N,--forward        Ignore already applied patches
    -E,--remove-empty-files Remove output files if they become empty

Description of the issue:

Steps to reproduce:


Step 4/7 : ADD https://github.com/manticoresoftware/manticoresearch/releases/download/2.7.4/manticore-2.7.4-181101-9496493-release.tar.gz /build/manticore.tar.gz
Downloading [==================================================>]  3.205MB/3.205MB
 ---> Using cache
 ---> 8418dd6706ca
Step 5/7 : RUN apk add --no-cache     cmake                  make                   g++                    mariadb-dev            mariadb-client
 ---> Using cache
 ---> de63bac33b3c
Step 6/7 : RUN cd /build &&      mkdir manticore &&      tar --strip-components=1 -xzf manticore.tar.gz -C manticore
 ---> Using cache
 ---> 3881f6cd4c9c
Step 7/7 : RUN cd /build/manticore      && mkdir -p build      && cd build      && cmake          -D CMAKE_BUILD_TYPE=Release          -D SPLIT_SYMBOLS=1          -D WITH_EXPAT=0          -D WITH_MYSQL=1          -D WITH_ODBC=0          -D WITH_PGSQL=0          -D WITH_RE2=ON          -D WITH_STEMMER=ON          -D DISABLE_TESTING=ON          -D CMAKE_INSTALL_PREFIX=/usr          -D CONFFILEDIR=/etc/sphinxsearch              -D SPHINX_TAG=release ..
 ---> Running in 5b1b8f7185c9
-- The C compiler identification is GNU 6.4.0
-- The CXX compiler identification is GNU 6.4.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Checking for endianess
-- Check if the system is big endian
-- Searching 16 bit integer
-- Looking for sys/types.h
-- Looking for sys/types.h - found
-- Looking for stdint.h
-- Looking for stdint.h - found
-- Looking for stddef.h
-- Looking for stddef.h - found
-- Check size of unsigned short
-- Check size of unsigned short - done
-- Using unsigned short
-- Check if the system is big endian - little endian
-- Checking for gcc version
-- Performing Test _RESULT
-- Performing Test _RESULT - Success
-- Could NOT find JEMALLOC (missing: JEMALLOC_LIBRARY)
-- Storing compiler version
-- Compiler is 6.4.0
-- Storing system name
-- Looking for threads library
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - found
-- Found Threads: TRUE
-- Looking for pthread_mutex_timedlock
-- Looking for pthread_mutex_timedlock - found
-- Checking for standard header files
-- Looking for 8 include files dlfcn.h, ..., float.h
-- Looking for 8 include files dlfcn.h, ..., float.h - found
-- Performing Test freeExists
-- Performing Test freeExists - Success
-- ANSI C header files - found
-- Checking if mincore has unsigned 3-rd param
-- Performing Test HAVE_UNSIGNED_MINCORE
-- Performing Test HAVE_UNSIGNED_MINCORE - Success
-- Checking for specific headers
-- Looking for execinfo.h
-- Looking for execinfo.h - not found
-- Looking for syslog.h
-- Looking for syslog.h - found
-- Looking for sys/eventfd.h
-- Looking for sys/eventfd.h - found
-- Looking for malloc.h
-- Looking for malloc.h - found
-- Looking for sys/prctl.h
-- Looking for sys/prctl.h - found
-- Checking for library functions
-- Looking for setsockopt
-- Looking for setsockopt - found
-- Looking for getaddrinfo
-- Looking for getaddrinfo - found
-- Looking for logf
-- Looking for logf - found
-- Looking for dlopen
-- Looking for dlopen - found
-- Looking for getaddrinfo_a
-- Looking for getaddrinfo_a - not found
-- Looking for getaddrinfo_a in anl
-- Looking for getaddrinfo_a in anl - not found
-- Looking for strnlen
-- Looking for strnlen - found
-- Looking for pread
-- Looking for pread - found
-- Looking for poll
-- Looking for poll - found
-- Looking for backtrace
-- Looking for backtrace - not found
-- Looking for backtrace_symbols
-- Looking for backtrace_symbols - not found
-- Looking for mremap
-- Looking for mremap - found
-- Looking for nanosleep
-- Looking for nanosleep - found
-- Looking for eventfd
-- Looking for eventfd - found
-- Looking for kqueue
-- Looking for kqueue - not found
-- Looking for getrlimit
-- Looking for getrlimit - found
-- Looking for setrlimit
-- Looking for setrlimit - found
-- Looking for epoll_ctl
-- Looking for epoll_ctl - found
-- Looking for clock_gettime
-- Looking for clock_gettime - found
-- Performing Test HAVE_F_SETLKW
-- Performing Test HAVE_F_SETLKW - Success
-- Checking for atomic function
-- Performing Test HAVE_SYNC_FETCH
-- Performing Test HAVE_SYNC_FETCH - Success
-- Checking for unaligned RAM access
-- Performing Test _UNALIGNED_RAM
-- Performing Test _UNALIGNED_RAM - Success
-- Checking for SO_REUSEPORT flag
-- Performing Test HAVE_SO_REUSEPORT
-- Performing Test HAVE_SO_REUSEPORT - Success
-- Checking for malloc_trim availability
-- Performing Test HAVE_MALLOC_TRIM
-- Performing Test HAVE_MALLOC_TRIM - Failed
-- Checking for malloc_stats availability
-- Performing Test HAVE_MALLOC_STATS
-- Performing Test HAVE_MALLOC_STATS - Failed
-- Checking for PTHREAD_RWLOCK_PREFER_WRITER_NONRECURSIVE_NP flag
-- Performing Test HAVE_RWLOCK_PREFER_WRITER
-- Performing Test HAVE_RWLOCK_PREFER_WRITER - Failed
-- Configuring the options...
-- Option DISABLE_MEMROUTINES
-- Option USE_SYSLOG
-- Option WITH_ZLIB
-- Option WITH_ODBC 0
-- ODBC_INCLUDE_DIRS is ODBC_INCLUDE_DIRS-NOTFOUND
-- ODBC_LIBRARY is ODBC_LIBRARY-NOTFOUND
-- ODBC found is FALSE
-- ODBC will be linked as
-- Option WITH_RLP
-- Option WITH_EXPAT 0
-- EXPAT will be linked as
-- Option WITH_ICONV
-- Option WITH_MYSQL 1
-- mysql_config was found /usr/bin/mysql_config
-- MySQL Include dir: /usr/include/mysql
-- MySQL Library    : mysqlclient;z;ssl;crypto
-- MySQL Library dir: /usr/lib
-- MySQL CXXFLAGS: -I/usr/include/mysql
-- MySQL Library Name   : libmysqlclient.so
-- MYSQL will be loaded dynamically in runtime as libmysqlclient.so.18
-- Option WITH_PGSQL 0
-- Could NOT find BISON (missing: BISON_EXECUTABLE)
-- Could NOT find FLEX (missing: FLEX_EXECUTABLE)
-- Option WITH_RE2 ON
-- Could NOT find RE2 (missing: RE2_INCLUDE_DIRS RE2_LIBRARY)
-- Downloading RE2
-- Unpack RE2 from /build/manticore/build/master.zip
-- Patching RE2
/usr/bin/patch: unrecognized option: binary
BusyBox v1.27.2 (2017-12-12 10:41:50 GMT) multi-call binary.

Usage: patch [OPTIONS] [ORIGFILE [PATCHFILE]]

    -p,--strip N        Strip N leading components from file names
    -i,--input DIFF     Read DIFF instead of stdin
    -R,--reverse        Reverse patch
    -N,--forward        Ignore already applied patches
    -E,--remove-empty-files Remove output files if they become empty
-- /build/manticore - source dir
-- Option WITH_STEMMER ON
-- Could NOT find STEMMER (missing: STEMMER_LIBRARY STEMMER_INCLUDE_DIR)
-- Downloading Stemmer
-- [download 3% complete]
adriannuta commented 6 years ago

Seems the busybox version doesn't support 'binary' option. Provide a Dockerfile to reproduce the issue.

As workaround you could install the gnu patch.

sshilko commented 6 years ago

Attached log & Dockerfile

log.txt Dockerfile.txt

klirichek commented 6 years ago

We use original source of RE2 from google (well, one concrete revision, m.b. quite ancient) and patch it with our own patch. For this we use one of two approaches: one is use git apply, another is use binutils patch. Even these two is quite over-engineering for such simple task as 'apply patch', but it works since you may not have 'git' and build directly from source tarball, say, on windows. '-binary' flag helps to avoid any issues with different cr/lf symbols in the code/ in the patch (otherwise they WILL arise on win). So, no other zoo of tools (like 'patch' from busybox vs 'patch' from binutils) is planned to support; these two are enough. So sorry, but you have either to patch the sources, either to install git or 'real' patch tool in the build env. OR as a variant - you can configure the build on another host where all the tools persists, set 'WITH_RE2_FORCE_STATIC' (and m.b. also 'WITH_STEMMER_FORCE_STATIC') and perform 'make tarball' then. It will produce your own tarball with all these things prepared, so when building on apline they will not need any patching (and also not need other tools like 'flex' and 'bison' btw.).

Just for reference: for building on alpine this basic docker is enogh:

FROM alpine

RUN apk add --no-cache \
    git \
    cmake \
    make \
    g++ \
    mariadb-dev \
    bison \
    flex-dev
sshilko commented 6 years ago

Okay, i can install my own gnu patch or use other -b compatible patch. For now compiling without RE2 support works too.

Updating documentation would fix issue for futute.Thx.