Open esabol opened 4 years ago
A clean rebuild of my 1.1.18 Docker image works just fine. My Docker image is based on Ubuntu Xenial (16.04).
My gearmand Docker image does not utilize PostgreSQL at all. Adding postgresql
to the apt-get -y install
line in my Dockerfile still results in the same error, presumably because pg_config
isn't in $PATH
.
When configuring 1.1.18, it just says:
checking for pg_config... no
configure: WARNING: does not exist or it is not an exectuable file
no
and it just keeps on configuring.
Found a workaround: Specify --with-postgresql=no
on the ./configure
line. That causes the check for pg_config
to be skipped.
But then the build still fails (ugh!) with errors, like this:
tests/sqlite_test.cc:42:21: fatal error: sqlite3.h: No such file or directory
compilation terminated.
Makefile:8054: recipe for target 'tests/t_sqlite-sqlite_test.o' failed
make[2]: *** [tests/t_sqlite-sqlite_test.o] Error 1
...
In file included from libgearman-server/plugins/queue/sqlite/queue.cc:50:0:
./libgearman-server/plugins/queue/sqlite/instance.hpp:42:21: fatal error: sqlite3.h: No such file or directory
compilation terminated.
Makefile:5639: recipe for target 'libgearman-server/plugins/queue/sqlite/libgearman_server_libgearman_server_la-queue.lo' failed
make[2]: *** [libgearman-server/plugins/queue/sqliteIn file included from libgearman-server/plugins/queue/sqlite/queue.cc:50:0:
./libgearman-server/plugins/queue/sqlite/instance.hpp:42:21: fatal error: sqlite3.h: No such file or directory
compilation terminated.
Makefile:5639: recipe for target 'libgearman-server/plugins/queue/sqlite/libgearman_server_libgearman_server_la-queue.lo' failed
make[2]: *** [libgearman-server/plugins/queue/sqlite/libgearman_server_libgearman_server_la-queue.lo] Error 1
/libgearman_server_libgearman_server_la-queue.lo] Error 1
Separate issue??
I tried adding --with-sqlite=no
to my ./configure
line, and it didn't help. I got this:
configure: WARNING: unrecognized options: --with-sqlite
Adding libsqlite3-dev
to my Dockerfile's apt-get -y install
line got the thing to fully compile and test, but this would appear to be another regression.
Any ideas as to what went wrong here??
I don't know, it still feels like there's some weird bug lurking in the m4 and we're hitting it. I'm thinking a bisect is in order to find out when this broke.
Looking at the diffs of 1.1.19's configure and 1.1.18's configure, and this diff looks like the reason for the sqlite issue:
@@ -19972,6 +19725,7 @@
{ $as_echo "$as_me:${as_lineno-$LINENO}: result: not found" >&5
$as_echo "not found" >&6; }
success="no"
+ WANT_SQLITE3="no"
fi
There are many, many diffs between the PostgreSQL portions of these configure scripts, too.
The 1.1.19 configure script claims it was created by the same version of Autoconf as 1.1.18's, so why did it result in a vastly different script?? How is that possible? I'm not sure how autoconf works, honestly. Can anyone spare a clue?
I don't know, it still feels like there's some weird bug lurking in the m4 and we're hitting it. I'm thinking a bisect is in order to find out when this broke.
I really doubt it. m4/ax_lib_sqlite3.m4 and m4/ax_lib_postgresql.m4 haven't changed in 8 years. And the stuff that's in those files is not what's in the 1.1.19's configure. Could something have overrode our m4 files?
I have a Dockerfile based on Ubuntu 16.04 which git clones the master branch, runs ./bootstrap -a
, ./configure --enable-ssl
, make
, and then make test
. It works flawlessly. I've got to run, but I'll post it soon. I also want to compare the configure files with the release version.
Ditto for Ubuntu 18.04 (using gcc-8/g++-8) and 14.04, by the way.
FROM ubuntu:16.04
MAINTAINER gearmand
ARG version=16.04
ARG GEARMAN_REPO=https://github.com/gearman/gearmand
LABEL version="${version}" description="Gearman SSL Job Server Image"
# Configure environment
ENV DEBIAN_FRONTEND=noninteractive \
TZ=America/New_York \
HOME=/root
# Install packages
RUN apt-get update \
&& apt-get -y upgrade \
&& apt-get -y install \
automake \
autoconf \
libtool \
make \
curl \
gcc \
g++ \
git \
gperf \
libboost-all-dev \
libevent-dev \
libhiredis-dev \
libssl-dev \
libtokyocabinet-dev \
python3-sphinx \
uuid-dev \
wget \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
# Retrieve the source code and bootstrap
RUN cd /tmp && git clone --depth 1 --branch master ${GEARMAN_REPO}.git
WORKDIR /tmp/gearmand
RUN ./bootstrap.sh -a
RUN ./configure --enable-ssl 2>&1 | tee ./configure.log
RUN make 2>&1 | tee ./build.log
RUN make test 2>&1 | tee ./test.log
The resulting configure file from the ./boostrap.sh -a
in the Docker image looks nothing like the configure file from the 1.1.19 release tarball. It's very similar to the configure file from the 1.1.18 release tarball with the changes mainly having to do with libevent2 and pthreads (as I would expect).
Essentially, I think the 1.1.19 release tarball wasn't packaged correctly....
@SpamapS : How does your release packaging procedure differ from the above Dockerfile?
The m4 files in the 1.1.19 release tarball do not match the m4 files in this repository.
Thanks Ed, I think I was rushing and ran make dist
outside of the clean VM that I was using to make the release, and my laptop is still producing weirdness in configure scripts.
I'll try making a 1.1.19.1 tarball from a clean VM. (and I'll test it before release this time!)
Please try 1.1.19.1, which should work fine.
I don't know, it still feels like there's some weird bug lurking in the m4 and we're hitting it. I'm thinking a bisect is in order to find out when this broke.
bootstrap force overriding of m4 files: https://github.com/gearman/gearmand/blob/a6bd168ca73bd29d0bf13ad946034957f9c86546/bootstrap.sh#L1096
Source build on Ubuntu 18.04 overrides 27 files:
autoreconf: running: aclocal --install --force -I m4
aclocal: overwriting 'm4/ax_add_am_macro.m4' with '/usr/share/aclocal/ax_add_am_macro.m4'
aclocal: overwriting 'm4/ax_am_jobserver.m4' with '/usr/share/aclocal/ax_am_jobserver.m4'
aclocal: overwriting 'm4/ax_am_macros.m4' with '/usr/share/aclocal/ax_am_macros.m4'
aclocal: overwriting 'm4/ax_append_compile_flags.m4' with '/usr/share/aclocal/ax_append_compile_flags.m4'
aclocal: overwriting 'm4/ax_append_flag.m4' with '/usr/share/aclocal/ax_append_flag.m4'
aclocal: overwriting 'm4/ax_append_link_flags.m4' with '/usr/share/aclocal/ax_append_link_flags.m4'
aclocal: overwriting 'm4/ax_append_to_file.m4' with '/usr/share/aclocal/ax_append_to_file.m4'
aclocal: overwriting 'm4/ax_boost_base.m4' with '/usr/share/aclocal/ax_boost_base.m4'
aclocal: overwriting 'm4/ax_boost_program_options.m4' with '/usr/share/aclocal/ax_boost_program_options.m4'
aclocal: overwriting 'm4/ax_check_compile_flag.m4' with '/usr/share/aclocal/ax_check_compile_flag.m4'
aclocal: overwriting 'm4/ax_check_link_flag.m4' with '/usr/share/aclocal/ax_check_link_flag.m4'
aclocal: overwriting 'm4/ax_check_openssl.m4' with '/usr/share/aclocal/ax_check_openssl.m4'
aclocal: overwriting 'm4/ax_compare_version.m4' with '/usr/share/aclocal/ax_compare_version.m4'
aclocal: overwriting 'm4/ax_compiler_vendor.m4' with '/usr/share/aclocal/ax_compiler_vendor.m4'
aclocal: overwriting 'm4/ax_compiler_version.m4' with '/usr/share/aclocal/ax_compiler_version.m4'
aclocal: overwriting 'm4/ax_count_cpus.m4' with '/usr/share/aclocal/ax_count_cpus.m4'
aclocal: overwriting 'm4/ax_file_escapes.m4' with '/usr/share/aclocal/ax_file_escapes.m4'
aclocal: overwriting 'm4/ax_generate_changelog.m4' with '/usr/share/aclocal/ax_generate_changelog.m4'
aclocal: overwriting 'm4/ax_lib_curl.m4' with '/usr/share/aclocal/ax_lib_curl.m4'
aclocal: overwriting 'm4/ax_lib_postgresql.m4' with '/usr/share/aclocal/ax_lib_postgresql.m4'
aclocal: overwriting 'm4/ax_lib_sqlite3.m4' with '/usr/share/aclocal/ax_lib_sqlite3.m4'
aclocal: overwriting 'm4/ax_path_generic.m4' with '/usr/share/aclocal/ax_path_generic.m4'
aclocal: overwriting 'm4/ax_print_to_file.m4' with '/usr/share/aclocal/ax_print_to_file.m4'
aclocal: overwriting 'm4/ax_pthread.m4' with '/usr/share/aclocal/ax_pthread.m4'
aclocal: overwriting 'm4/ax_restore_flags.m4' with '/usr/share/aclocal/ax_restore_flags.m4'
aclocal: overwriting 'm4/ax_save_flags.m4' with '/usr/share/aclocal/ax_save_flags.m4'
aclocal: overwriting 'm4/ax_with_prog.m4' with '/usr/share/aclocal/ax_with_prog.m4'
bootstrap force overriding of m4 files: ... Source build on Ubuntu 18.04 overrides 27 files:
Well, um, don’t do that?
Wouldn't it be good to put those update into repo?
Please try 1.1.19.1, which should work fine.
I've tested 1.1.19.1, and it compiled fine and did not exhibit the problems detailed earlier. make test
passed on the second try. t/multi_client
FAILed on the first try.
But the m4 files in the m4 directory still do not match the m4 files in this repo:
% wget https://github.com/gearman/gearmand/releases/download/1.1.19.1/gearmand-1.1.19.1.tar.gz
% tar xzvf gearmand-1.1.19.1.tar.gz
% cd gearmand-1.1.19.1/m4/
% diff -ur . ../../gearmand/m4/
Only in ../../gearmand/m4/: ax_cxx_compile_stdcxx_0x.m4
Only in ../../gearmand/m4/: ax_cxx_header_stdcxx_98.m4
Only in ../../gearmand/m4/: ax_switch_flags.m4
Only in ../../gearmand/m4/: lib-ld.m4
Only in ../../gearmand/m4/: lib-link.m4
Only in ../../gearmand/m4/: lib-prefix.m4
Only in .: libtool.m4
Only in .: lt~obsolete.m4
Only in .: ltoptions.m4
Only in .: ltsugar.m4
Only in .: ltversion.m4
Only in ../../gearmand/m4/: pandora_check_cxx_standard.m4
Only in ../../gearmand/m4/: pandora_clock_gettime.m4
Only in ../../gearmand/m4/: pandora_extensions.m4
Only in ../../gearmand/m4/: pandora_have_gcc_atomics.m4
Only in ../../gearmand/m4/: pandora_plugins.m4
Only in ../../gearmand/m4/: pandora_print_callstack.m4
Only in ../../gearmand/m4/: pandora_python3_devel.m4
Only in ../../gearmand/m4/: pandora_stack_direction.m4
Only in ../../gearmand/m4/: pandora_use_pipe.m4
Only in ../../gearmand/m4/: pandora_with_perl.m4
Only in ../../gearmand/m4/: pandora_with_python3.m4
Only in ../../gearmand/m4/: pandora_with_python.m4
Only in ../../gearmand/m4/: pandora_with_r.m4
Only in ../../gearmand/m4/: pandora_with_ruby.m4
Only in ../../gearmand/m4/: pkg.m4
Only in ../../gearmand/m4/: progtest.m4
Only in ../../gearmand/m4/: README.md
Clearly, the release procedure is different from I would expect. Could the release procedure be documented somewhere?
Wouldn't it be good to put those update into repo?
If they improve something, sure, but it should be done in a PR.
bootstrap force overriding of m4 files: ... Source build on Ubuntu 18.04 overrides 27 files:
My Ubuntu 18.04 build doesn't do this. I wonder what's different about your build? My aclocal on Ubuntu 18.04 is version 1.15.1.
FROM ubuntu:18.04
MAINTAINER gearmand
ARG version=18.04
ARG GEARMAN_REPO=https://github.com/gearman/gearmand
LABEL version="${version}" description="Gearman SSL Job Server Image"
# Configure environment
ENV DEBIAN_FRONTEND=noninteractive \
TZ=America/New_York \
HOME=/root
# Install packages
RUN apt-get update \
&& apt-get -y upgrade \
&& apt-get -y install \
automake \
autoconf \
libtool \
make \
curl \
gcc-8 \
g++-8 \
git \
gperf \
libboost-all-dev \
libevent-dev \
libhiredis-dev \
libssl-dev \
libtokyocabinet-dev \
tcsh \
python3-sphinx \
uuid-dev \
wget \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
# Retrieve the source code and bootstrap
RUN cd /tmp && git clone --depth 1 --branch master ${GEARMAN_REPO}.git
WORKDIR /tmp/gearmand
RUN ./bootstrap.sh -a
RUN env CC=gcc-8 CXX=g++-8 ./configure --enable-ssl 2>&1 | tee ./configure.log
RUN make 2>&1 | tee ./build.log
RUN make test 2>&1 | tee ./test.log
After running ./bootstrap.sh -a
on Ubuntu 18.04, the only m4 files that have changed are:
# cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.4 LTS"
# cp -ar m4 m4.old
# ./bootstrap.sh -a
`/usr/bin/libtoolize --copy --install --force'
libtoolize: found 'configure.ac'
libtoolize: export M4='/usr/bin/m4'
libtoolize: ac_aux_dir='build-aux'
libtoolize: ltdl mode=''
libtoolize: putting auxiliary files in AC_CONFIG_AUX_DIR, 'build-aux'.
libtoolize: cp -f /usr/share/libtool/build-aux/config.guess build-aux
libtoolize: cp -f /usr/share/libtool/build-aux/config.sub build-aux
libtoolize: cp -f /usr/share/libtool/build-aux/install-sh build-aux
libtoolize: cp -f /usr/share/libtool/build-aux/ltmain.sh build-aux
libtoolize: found 'Makefile.am'
libtoolize: aclocal_amflags=-I m4'
libtoolize: am_macro_dir='m4'
libtoolize: putting macros in AC_CONFIG_MACRO_DIRS, 'm4'.
libtoolize: cp -f /usr/share/aclocal/libtool.m4 m4
libtoolize: Not copying 'm4/ltargz.m4', libltdl not used.
libtoolize: Not copying 'm4/ltdl.m4', libltdl not used.
libtoolize: cp -f /usr/share/aclocal/ltoptions.m4 m4
libtoolize: cp -f /usr/share/aclocal/ltsugar.m4 m4
libtoolize: cp -f /usr/share/aclocal/ltversion.m4 m4
libtoolize: cp -f /usr/share/aclocal/lt~obsolete.m4 m4
libtoolize: found LT_INIT invocation
`/usr/bin/autoreconf --install --force --verbose'
autoreconf: Entering directory `.'
autoreconf: configure.ac: not using Gettext
autoreconf: running: aclocal --install --force -I m4
autoreconf: configure.ac: tracing
autoreconf: running: true --copy --force
autoreconf: running: /usr/bin/autoconf --force
autoreconf: running: /usr/bin/autoheader --force
autoreconf: running: automake --add-missing --copy --force-missing
configure.ac:19: installing 'build-aux/compile'
configure.ac:30: installing 'build-aux/missing'
Makefile.am: installing 'build-aux/depcomp'
parallel-tests: installing 'build-aux/test-driver'
autoreconf: Leaving directory `.'
# diff -ur m4.old m4
Only in m4: libtool.m4
Only in m4: ltoptions.m4
Only in m4: ltsugar.m4
Only in m4: ltversion.m4
Only in m4: lt~obsolete.m4
My Ubuntu 18.04 build doesn't do this. I wonder what's different about your build?
Please fix your Dockerfile
: remove building behind uuid-dev \
.
edit: I've the same Ubuntu release, kernel 4.15.0-76-generic
My aclocal on Ubuntu 18.04 is version 1.15.1.
The same in my environment.
Please fix your
Dockerfile
: remove building behinduuid-dev \
.
I'm sorry, but I don't understand what you are saying.
EDIT: Oh, OK. I see what you mean now. That was just an error in my copying and pasting. My Dockerfile does not have "building" there. I've edited the above post to remove it.
Quoting Ed Sabol (2020-02-19 13:42:26)
Please try 1.1.19.1, which should work fine.
I've tested 1.1.19.1, and it compiled fine and did not exhibit the problems detailed earlier. make test passed on the second try. t/multi_client FAILed on the first try.
But the m4 files in the m4 directory still do not match the m4 files in this repo: % wget https://github.com/gearman/gearmand/releases/download/1.1.19.1/gearmand-1 .1.19.1.tar.gz % tar xzvf gearmand-1.1.19.1.tar.gz % cd gearmand-1.1.19.1/m4/ % diff -ur . ../../gearmand/m4/ Only in ../../gearmand/m4/: ax_cxx_compile_stdcxx_0x.m4 Only in ../../gearmand/m4/: ax_cxx_header_stdcxx_98.m4 Only in ../../gearmand/m4/: ax_switch_flags.m4 Only in ../../gearmand/m4/: lib-ld.m4 Only in ../../gearmand/m4/: lib-link.m4 Only in ../../gearmand/m4/: lib-prefix.m4 Only in .: libtool.m4 Only in .: lt~obsolete.m4 Only in .: ltoptions.m4 Only in .: ltsugar.m4 Only in .: ltversion.m4 Only in ../../gearmand/m4/: pandora_check_cxx_standard.m4 Only in ../../gearmand/m4/: pandora_clock_gettime.m4 Only in ../../gearmand/m4/: pandora_extensions.m4 Only in ../../gearmand/m4/: pandora_have_gcc_atomics.m4 Only in ../../gearmand/m4/: pandora_plugins.m4 Only in ../../gearmand/m4/: pandora_print_callstack.m4 Only in ../../gearmand/m4/: pandora_python3_devel.m4 Only in ../../gearmand/m4/: pandora_stack_direction.m4 Only in ../../gearmand/m4/: pandora_use_pipe.m4 Only in ../../gearmand/m4/: pandora_with_perl.m4 Only in ../../gearmand/m4/: pandora_with_python3.m4 Only in ../../gearmand/m4/: pandora_with_python.m4 Only in ../../gearmand/m4/: pandora_with_r.m4 Only in ../../gearmand/m4/: pandora_with_ruby.m4 Only in ../../gearmand/m4/: pkg.m4 Only in ../../gearmand/m4/: progtest.m4 Only in ../../gearmand/m4/: README.md
Clearly, the release procedure is different from I would expect. Could the release procedure be documented somewhere?
Great idea, I'll try to get some cycles for a more complete writeup but here's the basics:
1) On a clean VM, clone gearmand
2) git tag -s {release} describing overall release changes
3) ./bootstrap.sh
4) assuming clean build and test seen there: make dist
-- this makes
the tarball
5) copy tarball to another dir, extract, ./configure && make -j && make
install
6) smoke test gearmand
7) gpg --armor --sign --detach {tarball}
8) Create release on github, upload tarball + signature
The reason this makes the m4 files different in the tarball is that it uses the versions packaged along with auto tools. I'm not a deep expert here (I'm pretty sure nobody actually is), but I think the approach @p-alik is taking to update them in the git repo is correct, as long as we can get it to build with all the stuff turned on.
Quoting Ed Sabol (2020-02-20 18:45:50)
bootstrap force overriding of m4 files: ... Source build on Ubuntu 18.04 overrides 27 files:
My Ubuntu 18.04 build doesn't do this. I wonder what's different about your build? My aclocal on Ubuntu 18.04 is version 1.15.1.
FROM ubuntu:18.04
MAINTAINER gearmand
ARG version=18.04 ARG GEARMAN_REPO=https://github.com/gearman/gearmand
LABEL version="${version}" description="Gearman SSL Job Server Image"
Configure environment
ENV DEBIAN_FRONTEND=noninteractive \ TZ=America/New_York \ HOME=/root
Install packages
RUN apt-get update \ && apt-get -y upgrade \ && apt-get -y install \ automake \ autoconf \ libtool \ make \ curl \ gcc-8 \ g++-8 \ git \ gperf \ libboost-all-dev \ libevent-dev \ libhiredis-dev \ libssl-dev \ libtokyocabinet-dev \ tcsh \ python3-sphinx \ uuid-dev \ wget \ && apt-get clean \ && rm -rf /var/lib/apt/lists/*
Retrieve the source code and bootstrap
RUN cd /tmp && git clone --depth 1 --branch master ${GEARMAN_REPO}.git WORKDIR /tmp/gearmand RUN ./bootstrap.sh -a
RUN env CC=gcc-8 CXX=g++-8 ./configure --enable-ssl 2>&1 | tee ./configure.log RUN make 2>&1 | tee ./build.log RUN make test 2>&1 | tee ./test.log
The Dockerfiles like the one above have been useful for me. Can you submit it as Dockerfile in the repo root?
here's the basics:
1) On a clean VM, clone gearmand 2) git tag -s {release} describing overall release changes 3) ./bootstrap.sh 4) assuming clean build and test seen there:
make dist
-- this makes the tarball 5) copy tarball to another dir, extract, ./configure && make -j && make install 6) smoke test gearmand 7) gpg --armor --sign --detach {tarball} 8) Create release on github, upload tarball + signature
The main difference I see from what you do and what I do (and what Travis CI does) is that you execute ./boostrap.sh
, but I execute ./bootstrap.sh -a
. Neither one results in modifications to the m4 files on my ubuntu:18.04 Docker image, so I'm still confused as to why you and @p-alik are getting different m4 files from me.
The reason this makes the m4 files different in the tarball is that it uses the versions packaged along with auto tools.
Sure, I get that, but I'm using the latest versions of aclocal and automake on Ubuntu 18.04... Do you have a different versions?
docker:/tmp/gearmand# apt-get -y install autoconf
Reading package lists... Done
Building dependency tree
Reading state information... Done
autoconf is already the newest version (2.69-11).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
docker:/tmp/gearmand# apt-get -y install automake
Reading package lists... Done
Building dependency tree
Reading state information... Done
automake is already the newest version (1:1.15.1-3ubuntu2).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
The Dockerfiles like the one above have been useful for me. Can you submit it as Dockerfile in the repo root?
Just the one Dockerfile or something like this?
docker/README
docker/ubuntu-18.04/Dockerfile
docker/ubuntu-16.04/Dockerfile
docker/ubuntu-14.04/Dockerfile
docker/fedora-31/Dockerfile
docker/centos-7/Dockerfile
docker/centos-7-i386/Dockerfile
I went to update the Docker image I use at work, and I encountered the following when building:
All I did was change the version number of the gearmand.tar.gz file from 1.1.18 to 1.1.19.