Get the thing compiling

staab commented 4 years ago

For additional background on getting and installing libpq, check out this SO answer.

This is just a dumping place for my progress in compiling/linking my code with postgres. Unless otherwise specified, I'm using jpm test to trigger the errors.

After getting the right -I flag to include libpq-fe.h, I started getting:

creating static library build/pg.a...
running test/pg.janet ...
error: could not load native build/pg.so: dlopen(build/pg.so, 2): Symbol not found: _PQconnectdb
  Referenced from: build/pg.so
  Expected in: flat namespace
 in build/pg.so
  in native
  in <anonymous> [boot.janet] on line 1934, column 26
  in require [boot.janet] on line 1955, column 16
  in import* [boot.janet] on line 1967, column 15
  in _thunk [test/pg.janet] (tailcall) on line 1, column 1

I was able to get that error to go away by including all the c files from libpq:

(declare-native
 :name "pg"
 :source @["staab.pg/pg.c"
           "postgresql/src/interfaces/libpq/fe-auth-scram.c"
           "postgresql/src/interfaces/libpq/fe-auth.c"
           "postgresql/src/interfaces/libpq/fe-connect.c"
           "postgresql/src/interfaces/libpq/fe-exec.c"
           "postgresql/src/interfaces/libpq/fe-gssapi-common.c"
           "postgresql/src/interfaces/libpq/fe-lobj.c"
           "postgresql/src/interfaces/libpq/fe-misc.c"
           "postgresql/src/interfaces/libpq/fe-print.c"
           "postgresql/src/interfaces/libpq/fe-protocol2.c"
           "postgresql/src/interfaces/libpq/fe-protocol3.c"
           "postgresql/src/interfaces/libpq/fe-secure-common.c"
           "postgresql/src/interfaces/libpq/fe-secure-gssapi.c"
           "postgresql/src/interfaces/libpq/fe-secure-openssl.c"
           "postgresql/src/interfaces/libpq/fe-secure.c"
           "postgresql/src/interfaces/libpq/legacy-pqsignal.c"
           "postgresql/src/interfaces/libpq/libpq-events.c"
           "postgresql/src/interfaces/libpq/pqexpbuffer.c"
           "postgresql/src/interfaces/libpq/pthread-win32.c"
           "postgresql/src/interfaces/libpq/win32.c"]
 :cflags @["-Ipostgresql/src/interfaces/libpq"
           "-Ipostgresql/src/include"])

I then got:

compiling build/postgresql___src___interfaces___libpq___fe-connect.o...
postgresql/src/interfaces/libpq/fe-connect.c:32:10: fatal error: 'pg_config_paths.h' file not found
#include "pg_config_paths.h"
         ^~~~~~~~~~~~~~~~~~~
1 error generated.

Which I was able to get rid of by running cd postgresql/src/interfaces/libpq && make ../../../src/port/pg_config_paths.h and adding -Ipostgresql/src/port to my :cflags section.

After looking into it a bit more, that seemed like a bandaid, so I ended up installing the client libraries to postgresql/build, using:

cd postgresql
rm -rf build
make distclean
./configure --prefix=$(pwd)/build
make -C src/bin install
make -C src/include install
make -C src/interfaces install
make -C doc install

This let me to change my project.janet to:

(declare-native
 :name "pg"
 :source @["staab.pg/pg.c"
           "postgresql/src/interfaces/libpq/fe-auth-scram.c"
           "postgresql/src/interfaces/libpq/fe-auth.c"
           "postgresql/src/interfaces/libpq/fe-connect.c"
           "postgresql/src/interfaces/libpq/fe-exec.c"
           "postgresql/src/interfaces/libpq/fe-gssapi-common.c"
           "postgresql/src/interfaces/libpq/fe-lobj.c"
           "postgresql/src/interfaces/libpq/fe-misc.c"
           "postgresql/src/interfaces/libpq/fe-print.c"
           "postgresql/src/interfaces/libpq/fe-protocol2.c"
           "postgresql/src/interfaces/libpq/fe-protocol3.c"
           "postgresql/src/interfaces/libpq/fe-secure-common.c"
           "postgresql/src/interfaces/libpq/fe-secure-gssapi.c"
           "postgresql/src/interfaces/libpq/fe-secure-openssl.c"
           "postgresql/src/interfaces/libpq/fe-secure.c"
           "postgresql/src/interfaces/libpq/legacy-pqsignal.c"
           "postgresql/src/interfaces/libpq/libpq-events.c"
           "postgresql/src/interfaces/libpq/pqexpbuffer.c"
           "postgresql/src/interfaces/libpq/pthread-win32.c"
           "postgresql/src/interfaces/libpq/win32.c"]
 :cflags @["-Ipostgresql/build/include"
           "-Ipostgresql/build/include/libpq"
           "-Ipostgresql/build/include/internal"
           "-Ipostgresql/build/include/internal/libpq"
           "-Ipostgresql/build/include/server"
           "-Lpostgresql/build/lib"])

At this point, I'm now getting a bunch of errors having to do with types that aren't defined:

compiling build/postgresql___src___interfaces___libpq___fe-gssapi-common.o...
clang: warning: argument unused during compilation: '-Lpostgresql/build/lib' [-Wunused-command-line-argument]
postgresql/src/interfaces/libpq/fe-gssapi-common.c:26:6: error: unknown type name 'OM_uint32'; did you mean 'uint32'?
                                 OM_uint32 stat, int type)
                                 ^~~~~~~~~
                                 uint32
postgresql/build/include/internal/c.h:359:22: note: 'uint32' declared here
typedef unsigned int uint32;    /* == 32 bits */
                     ^
postgresql/src/interfaces/libpq/fe-gssapi-common.c:28:2: error: unknown type name 'OM_uint32'; did you mean 'uint32'?
        OM_uint32       lmin_s;
        ^~~~~~~~~
        uint32
postgresql/build/include/internal/c.h:359:22: note: 'uint32' declared here
typedef unsigned int uint32;    /* == 32 bits */
                     ^
postgresql/src/interfaces/libpq/fe-gssapi-common.c:29:2: error: use of undeclared identifier 'gss_buffer_desc'
        gss_buffer_desc lmsg;
        ^
postgresql/src/interfaces/libpq/fe-gssapi-common.c:30:2: error: unknown type name 'OM_uint32'; did you mean 'uint32'?
        OM_uint32       msg_ctx = 0;
        ^~~~~~~~~
        uint32
postgresql/build/include/internal/c.h:359:22: note: 'uint32' declared here
typedef unsigned int uint32;    /* == 32 bits */
                     ^
postgresql/src/interfaces/libpq/fe-gssapi-common.c:34:3: warning: implicit declaration of function
      'gss_display_status' is invalid in C99 [-Wimplicit-function-declaration]
                gss_display_status(&lmin_s, stat, type,
                ^
postgresql/src/interfaces/libpq/fe-gssapi-common.c:35:10: error: use of undeclared identifier 'GSS_C_NO_OID'
                                                   GSS_C_NO_OID, &msg_ctx, &lmsg);
                                                   ^
postgresql/src/interfaces/libpq/fe-gssapi-common.c:35:35: error: use of undeclared identifier 'lmsg'
                                                   GSS_C_NO_OID, &msg_ctx, &lmsg);

(several more lines)

If I search postgresql for e.g., OM_uint32, I get a lot of uses of it, but no definition, so I wonder if it's a preprocessor thing. Maybe I need to reference different c files, or use a -L option?

In summary, I have no idea what I'm doing. I'll keep this updated as I work on it.

staab commented 4 years ago

I got it! I switched to using pg_config to reference my installation of postgresql, as documented here. This means that everyone who installs this module will need postgresql installed and pg_config on the path, but that's ok for now.

staab commented 4 years ago

So that got it compiling on MacOS fine, but then I wanted to run Valgrind on the code, which doesn't currently run on Mojave (a sketchy version can be installed with brew install --HEAD https://raw.githubusercontent.com/sowson/valgrind/master/valgrind.rb but I'm not sure I trust the port). So, I added a dockerfile to try running valgrind on my library under linux. To do this, I installed libpq-dev using apt, and compiled my app, but when I went to run jpm test I got an error about an undefined symbol. Build/test output is below:

root@e36dfcf520b7:/var/app# jpm build
linking build/pg.so...
generating meta file build/pg.meta.janet...
compiling build/staab.pg___pg.static.o...
creating static library build/pg.a...
root@e36dfcf520b7:/var/app# jpm test
running test/pg.janet ...
error: could not load native build/pg.so: build/pg.so: undefined symbol: PQconnectdb
  in native
  in <anonymous> [boot.janet] on line 1942, column 26
  in require [boot.janet] on line 1963, column 16
  in import* [boot.janet] on line 1975, column 15
  in _thunk [test/pg.janet] (tailcall) on line 1, column 1

From everything I've read, my configuration should work (I am passing the correct includedir and libdir per pg_config, as well as -lpq). When I run nm -D /usr/lib/x86_64-linux-gnu/libpq.so, PQconnectdb is listed. If I export export LD_DEBUG=files I get 157: build/pg.so: error: symbol lookup error: undefined symbol: PQconnectdb (fatal).

staab commented 4 years ago

Side note: the reason I want to run valgrind is I'm hoping that it'll tell my why I get no output from jpm test other than "All tests passed" if I uncomment https://github.com/staab/janet-pg/blob/master/staab.pg/pg.c#L41.

staab / janet-pqutils

Get the thing compiling #1