search

citusdata / citus

Distributed PostgreSQL as an extension
https://www.citusdata.com
GNU Affero General Public License v3.0
10.61k stars 671 forks source link

"make check" fails on my mac #373

Open begriffs opened 8 years ago

begriffs commented 8 years ago

Running make and make install succeed on my system but make check causes this error.

/Applications/Xcode.app/Contents/Developer/usr/bin/make -C src/test/regress check-full
'/usr/bin/perl' /Users/j/citus/citus/src/test/regress/pg_regress_multi.pl --pgxsdir="/usr/local/lib/postgresql/pgxs" --bindir="/usr/local/Cellar/postgresql/9.5.1/bin" --libdir="/usr/local/lib" --majorversion="9.5" --load-extension=citus \
    -- --inputdir=/Users/j/citus/citus/src/test/regress   --schedule=/Users/j/citus/citus/src/test/regress/multi_schedule
dyld: Symbol not found: __cg_jpeg_resync_to_restart
  Referenced from: /System/Library/Frameworks/ImageIO.framework/Versions/A/ImageIO
  Expected in: /usr/local/lib/libJPEG.dylib
 in /System/Library/Frameworks/ImageIO.framework/Versions/A/ImageIO

It happens because of the value of the library search path (libdir). As a first gross hack I can get make check to pass by hard-coding libdir and bindir in the regression test Makefile.

--- a/src/test/regress/Makefile
+++ b/src/test/regress/Makefile
@@ -14,7 +14,7 @@ endif
 ## Citus regression support
 ##
 MULTI_INSTALLDIR=$(CURDIR)/tmp_check/install
-pg_regress_multi_check = $(PERL) $(citus_abs_srcdir)/pg_regress_multi.pl --pgxsdir="$(pgxsdir)" --bindir="$(bindir)" --libdir="$(libdir)" --majorversion="$(MAJORVERSION)"
+pg_regress_multi_check = $(PERL) $(citus_abs_srcdir)/pg_regress_multi.pl --pgxsdir="$(pgxsdir)" --bindir="/usr/local/Cellar/postgresql/9.5.1/bin" --libdir="/usr/local/Cellar/postgresql/9.5.1/lib" --majorversion="$(MAJORVERSION)"
 MULTI_REGRESS_OPTS = --inputdir=$(citus_abs_srcdir) $(pg_regress_locale_flags)

 # XXX: Can't actually do useful testruns against install - $libdir

My next attempt is to set these variables with the configure script rather than hard-coding them:

make clean
./configure --libdir=/usr/local/Cellar/postgresql/9.5.1/lib --bindir=/usr/local/Cellar/postgresql/9.5.1/bin
make
make check

However this fails. I notice that the main makefile passes bindir to the regression test makefile but appears not to pass my custom libdir.

My system has had several versions of postgres on it at various times, so it may be fussier than most. Here are some other facts

anarazel commented 8 years ago

I do get the libdir configured when compiling postgres, which you don't seem to be. What PG are you compiling this against? What's the output of pg_config?

begriffs commented 8 years ago

I'm running postgres 9.5.1 installed via homebrew

pg_config:

BINDIR = /usr/local/Cellar/postgresql/9.5.1/bin
DOCDIR = /usr/local/Cellar/postgresql/9.5.1/share/doc/postgresql
HTMLDIR = /usr/local/Cellar/postgresql/9.5.1/share/doc/postgresql
INCLUDEDIR = /usr/local/Cellar/postgresql/9.5.1/include
PKGINCLUDEDIR = /usr/local/Cellar/postgresql/9.5.1/include
INCLUDEDIR-SERVER = /usr/local/Cellar/postgresql/9.5.1/include/server
LIBDIR = /usr/local/lib
PKGLIBDIR = /usr/local/lib/postgresql
LOCALEDIR = /usr/local/Cellar/postgresql/9.5.1/share/locale
MANDIR = /usr/local/Cellar/postgresql/9.5.1/share/man
SHAREDIR = /usr/local/share/postgresql
SYSCONFDIR = /usr/local/etc/postgresql
PGXS = /usr/local/lib/postgresql/pgxs/src/makefiles/pgxs.mk
CONFIGURE = '--disable-debug' '--prefix=/usr/local/Cellar/postgresql/9.5.1' '--datadir=/usr/local/share/postgresql' '--libdir=/usr/local/lib' '--sysconfdir=/usr/local/etc' '--docdir=/usr/local/Cellar/postgresql/9.5.1/share/doc/postgresql' '--enable-thread-safety' '--with-bonjour' '--with-gssapi' '--with-ldap' '--with-openssl' '--with-pam' '--with-libxml' '--with-libxslt' '--with-perl' '--with-tcl' '--with-tclconfig=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk/usr/lib' '--with-uuid=e2fs' 'CC=clang' 'LDFLAGS=-L/usr/local/opt/openssl/lib -L/usr/local/opt/readline/lib' 'CPPFLAGS=-I/usr/local/opt/openssl/include -I/usr/local/opt/readline/include'
CC = clang
CPPFLAGS = -I/usr/local/opt/openssl/include -I/usr/local/opt/readline/include -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk/usr/include/libxml2
CFLAGS = -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -Wno-unused-command-line-argument -O2
CFLAGS_SL =
LDFLAGS = -L../../../src/common -L/usr/local/opt/openssl/lib -L/usr/local/opt/readline/lib -Wl,-dead_strip_dylibs
LDFLAGS_EX =
LDFLAGS_SL =
LIBS = -lpgcommon -lpgport -lxslt -lxml2 -lpam -lssl -lcrypto -lgssapi_krb5 -lz -lreadline -lm
VERSION = PostgreSQL 9.5.1
anarazel commented 8 years ago

@begriffs Note that libdir in the pg_config output is /usr/local/lib, but you've adjusted it to --libdir="/usr/local/Cellar/postgresql/9.5.1/lib" in your workaround. So there really shouldn't be anything relevant in "/usr/local/Cellar/postgresql/9.5.1/lib" - what's the current contents?

begriffs commented 8 years ago

https://gist.github.com/begriffs/ac82bc5f7e5953616aba

anarazel commented 8 years ago

@begriffs So, I don't understand how you have any files in that directory at all; libdir and pglibdir pretty clearly are in /usr/local/lib and /usr/local/lib/postgresql respectively. Are files there as well? What are you passing to citus' ./configure?

begriffs commented 8 years ago

This is weird, but the following fixes my mac:

brew uninstall --force libjpeg libpng libtiff libgif giflib
begriffs commented 8 years ago

I'm attempting to determine why it works for me now and did not work earlier.

First @anarazel you were wondering why I used the path /usr/local/Cellar/postgresql/9.5.1/lib despite pg_config reporting LIBDIR=/usr/local/lib. It's because all directories in /usr/local/lib are symlinks to things in my homebrew "Cellar."

$ ls -lh /usr/local/lib/postgresql
lrwxr-xr-x  1 j  admin    41B Feb 12 23:48 /usr/local/lib/postgresql -> ../Cellar/postgresql/9.5.1/lib/postgresql

So interesting thing for debugging the problem: if I install libjpeg then I can get my system to break again.

brew install libjpeg
make clean
./configure
make
make install

Now make check fails like this

# make check

/Applications/Xcode.app/Contents/Developer/usr/bin/make -C src/backend/distributed/ all
make[1]: Nothing to be done for `all'.
/Applications/Xcode.app/Contents/Developer/usr/bin/make -C src/bin/csql/ all
make[1]: Nothing to be done for `all'.
/Applications/Xcode.app/Contents/Developer/usr/bin/make -C src/backend/distributed/ install
/bin/sh /usr/local/lib/postgresql/pgxs/src/makefiles/../../config/install-sh -c -d '/usr/local/lib/postgresql'
/bin/sh /usr/local/lib/postgresql/pgxs/src/makefiles/../../config/install-sh -c -d '/usr/local/share/postgresql/extension'
/bin/sh /usr/local/lib/postgresql/pgxs/src/makefiles/../../config/install-sh -c -d '/usr/local/share/postgresql/extension'
/bin/sh /usr/local/lib/postgresql/pgxs/src/makefiles/../../config/install-sh -c -d '/usr/local/Cellar/postgresql/9.5.1/bin'
/usr/bin/install -c -m 755  citus.so '/usr/local/lib/postgresql/citus.so'
/usr/bin/install -c -m 644 .//citus.control '/usr/local/share/postgresql/extension/'
/usr/bin/install -c -m 644  citus--5.0.sql '/usr/local/share/postgresql/extension/'
/usr/bin/install -c -m 755 .//../../bin/scripts/copy_to_distributed_table '/usr/local/Cellar/postgresql/9.5.1/bin/'
/Applications/Xcode.app/Contents/Developer/usr/bin/make -C src/bin/csql/ install
/bin/sh /usr/local/lib/postgresql/pgxs/src/makefiles/../../config/install-sh -c -d '/usr/local/Cellar/postgresql/9.5.1/bin'
/usr/bin/install -c  csql '/usr/local/Cellar/postgresql/9.5.1/bin'
/bin/sh /usr/local/lib/postgresql/pgxs/src/makefiles/../../config/install-sh -c -d '/usr/local/Cellar/postgresql/9.5.1/include/server/distributed/'
/usr/bin/install -c -m 644 src/include/citus_config.h '/usr/local/Cellar/postgresql/9.5.1/include/server/'
/usr/bin/install -c -m 644 /Users/j/citus/citus/./src/include/distributed/*.h '/usr/local/Cellar/postgresql/9.5.1/include/server/distributed/'
/Applications/Xcode.app/Contents/Developer/usr/bin/make -C src/test/regress check-full
'/usr/bin/perl' /Users/j/citus/citus/src/test/regress/pg_regress_multi.pl --pgxsdir="/usr/local/lib/postgresql/pgxs" --bindir="/usr/local/Cellar/postgresql/9.5.1/bin" --libdir="/usr/local/lib" --majorversion="9.5" --load-extension=citus \
    -- --inputdir=/Users/j/citus/citus/src/test/regress   --schedule=/Users/j/citus/citus/src/test/regress/multi_schedule
dyld: Symbol not found: __cg_jpeg_resync_to_restart
  Referenced from: /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libTIFF.dylib
  Expected in: /usr/local/lib/libJPEG.dylib
 in /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libTIFF.dylib
no data was returned by command ""/usr/local/Cellar/postgresql/9.5.1/bin/postgres" -V"
The program "postgres" is needed by initdb but was not found in the
same directory as "/usr/local/Cellar/postgresql/9.5.1/bin/initdb".
Check your installation.
Could not create master data directory at /Users/j/citus/citus/src/test/regress/pg_regress_multi.pl line 153.
Use of uninitialized value $serversAreShutdown in string eq at /Users/j/citus/citus/src/test/regress/pg_regress_multi.pl line 173.
make[1]: *** [check-multi] Error 25
make: *** [check] Error 2

@lfittl I'd like to try building Citus on your mac and see if the presence or absence of libjpeg affects your build. Just to see if it this is something specific to my machine.

begriffs commented 8 years ago

(Just a note for myself as I continue debugging)

I upgraded my homebrew postgres.

From same terminal

When running pg_config in the same terminal in which I ran the update it tells me LIBDIR = /usr/local/Cellar/postgresql/9.5.2/lib.

BINDIR = /usr/local/Cellar/postgresql/9.5.2/bin
DOCDIR = /usr/local/Cellar/postgresql/9.5.2/share/doc/postgresql
HTMLDIR = /usr/local/Cellar/postgresql/9.5.2/share/doc/postgresql
INCLUDEDIR = /usr/local/Cellar/postgresql/9.5.2/include
PKGINCLUDEDIR = /usr/local/Cellar/postgresql/9.5.2/include/postgresql
INCLUDEDIR-SERVER = /usr/local/Cellar/postgresql/9.5.2/include/postgresql/server
LIBDIR = /usr/local/Cellar/postgresql/9.5.2/lib
PKGLIBDIR = /usr/local/Cellar/postgresql/9.5.2/lib/postgresql
LOCALEDIR = /usr/local/Cellar/postgresql/9.5.2/share/locale
MANDIR = /usr/local/Cellar/postgresql/9.5.2/share/man
SHAREDIR = /usr/local/Cellar/postgresql/9.5.2/share/postgresql
SYSCONFDIR = /usr/local/Cellar/postgresql/9.5.2/etc/postgresql
PGXS = /usr/local/Cellar/postgresql/9.5.2/lib/postgresql/pgxs/src/makefiles/pgxs.mk
CONFIGURE = '--prefix=/Applications/Postgres.app/Contents/Versions/9.5' '--with-includes=/Applications/Postgres.app/Contents/Versions/9.5/include' '--with-libraries=/Applications/Postgres.app/Contents/Versions/9.5/lib' '--enable-thread-safety' '--with-openssl' '--with-gssapi' '--with-bonjour' '--with-libxml' '--with-libxslt' '--with-perl' '--with-python' '--with-readline' '--with-uuid=e2fs' 'CFLAGS= -mmacosx-version-min=10.7'
CC = gcc
CPPFLAGS = -I/Applications/Postgres.app/Contents/Versions/9.5/include/libxml2 -I/Applications/Postgres.app/Contents/Versions/9.5/include
CFLAGS = -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -Wno-unused-command-line-argument  -mmacosx-version-min=10.7
CFLAGS_SL =
LDFLAGS = -L../../../src/common -L/Applications/Postgres.app/Contents/Versions/9.5/lib -L/Applications/Postgres.app/Contents/Versions/9.5/lib -Wl,-dead_strip_dylibs
LDFLAGS_EX =
LDFLAGS_SL =
LIBS = -lpgcommon -lpgport -lxslt -lxml2 -lssl -lcrypto -lgssapi_krb5 -lz -lreadline -lm
VERSION = PostgreSQL 9.5.0

New terminal

Settings have changed, including LIBDIR.

BINDIR = /usr/local/Cellar/postgresql/9.5.2/bin
DOCDIR = /usr/local/Cellar/postgresql/9.5.2/share/doc/postgresql
HTMLDIR = /usr/local/Cellar/postgresql/9.5.2/share/doc/postgresql
INCLUDEDIR = /usr/local/Cellar/postgresql/9.5.2/include
PKGINCLUDEDIR = /usr/local/Cellar/postgresql/9.5.2/include
INCLUDEDIR-SERVER = /usr/local/Cellar/postgresql/9.5.2/include/server
LIBDIR = /usr/local/lib
PKGLIBDIR = /usr/local/lib/postgresql
LOCALEDIR = /usr/local/Cellar/postgresql/9.5.2/share/locale
MANDIR = /usr/local/Cellar/postgresql/9.5.2/share/man
SHAREDIR = /usr/local/share/postgresql
SYSCONFDIR = /usr/local/etc/postgresql
PGXS = /usr/local/lib/postgresql/pgxs/src/makefiles/pgxs.mk
CONFIGURE = '--disable-debug' '--prefix=/usr/local/Cellar/postgresql/9.5.2' '--datadir=/usr/local/share/postgresql' '--libdir=/usr/local/lib' '--sysconfdir=/usr/local/etc' '--docdir=/usr/local/Cellar/postgresql/9.5.2/share/doc/postgresql' '--enable-thread-safety' '--with-bonjour' '--with-gssapi' '--with-ldap' '--with-openssl' '--with-pam' '--with-libxml' '--with-libxslt' '--with-perl' '--with-tcl' '--with-tclconfig=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk/usr/lib' '--with-uuid=e2fs' 'CC=clang' 'LDFLAGS=-L/usr/local/opt/openssl/lib -L/usr/local/opt/readline/lib' 'CPPFLAGS=-I/usr/local/opt/openssl/include -I/usr/local/opt/readline/include'
CC = clang
CPPFLAGS = -I/usr/local/opt/openssl/include -I/usr/local/opt/readline/include -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk/usr/include/libxml2
CFLAGS = -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -Wno-unused-command-line-argument -O2
CFLAGS_SL = 
LDFLAGS = -L../../../src/common -L/usr/local/opt/openssl/lib -L/usr/local/opt/readline/lib -Wl,-dead_strip_dylibs
LDFLAGS_EX = 
LDFLAGS_SL = 
LIBS = -lpgcommon -lpgport -lxslt -lxml2 -lpam -lssl -lcrypto -lgssapi_krb5 -lz -lreadline -lm 
VERSION = PostgreSQL 9.5.2

Sadly make check does not work in either terminal.

ozgune commented 8 years ago

@begriffs -- I had two quick questions.

Is this still an issue for you? How likely do you think regular Citus users will run into this issue?

ozgune commented 8 years ago

@begriffs -- are you still running into this issue?

jasonmp85 commented 8 years ago

(I'm also curious, as I'd like to close this)

begriffs commented 8 years ago

I made a note for myself to check this on Tuesday when I'm back in the office.

jasonmp85 commented 8 years ago

I just checked myself; same issue (with libdir, libjpeg, etc.)

In particular, I noticed that removing the DYLD_LIBRARY_PATH from pg_regress_multi fixes things, but I no longer remember why that was necessary; maybe @anarazel can chime in?

jasonmp85 commented 8 years ago

Also, I believe this might be related to homebrew's PostgreSQL itself, which has problems I documented in two other issues:

Given that no customer (or even developer) has ever hit this, I'm kind of leaning to closing it as a known issue with several workarounds that we can discuss if a customer/would-be developer happens to hit it.

ozgune commented 6 years ago

@begriffs - What do you think? Does closing this issue sound good to you?

begriffs commented 6 years ago

Sure, it's an old issue that hasn't seemed to affect anyone else.

ozgune commented 6 years ago

@begriffs -- I just run into problems with make check and brew on my OS X. So I'm temporarily re-opening this issue. 😄

I followed the steps in our CONTRIBUTING.md and installed PostgreSQL from Homebrew. Over time, I ended up with two PG versions (9.5.1 and 9.6.3) on my machine. I recently downloaded Citus 7.0.3 and ran the regression tests.

44 regression tests failed with the following SSL related message: INSERT INTO events_reference_table SELECT * FROM events_table; +WARNING: connection error: localhost:57638 +DETAIL: could not send data to server: Socket is not connected +could not send SSL negotiation packet: Socket is not connected

@jasonmp85 -- I remember you mentioned that you were seeing SSL related errors in your regression tests.

I then asked @metdos to retry the same steps in his OS X. @metdos told me that he always build from source because he had run into issues with building from Homebrew's PostgreSQL. So, I uninstalled all Homebrew related Postgres versions and built from source. I then configured Citus with the following:

PG_CONFIG=/usr/local/pgsql/bin/pg_config ./configure

Next, I built and installed Citus. I ran the regression tests and they all pass now. @jasonmp85 -- This looks very similar to the issue you reported on connection tests. Could you check if that's the case?

@begriffs -- Should we guide our users towards building against PG source in CHANGELOG.md -- don't know, just asking?