search

turbot / steampipe-postgres-fdw

The Steampipe foreign data wrapper (FDW) is a zero-ETL product that provides Postgres foreign tables which translate queries into API calls to cloud services and APIs. It's bundled with Steampipe and also available as a set of standalone extensions for use in your own Postgres database.
https://steampipe.io/
Apache License 2.0
71 stars 16 forks source link

Use go version 1.20 to build and test FDW #296

Open pskrbasu opened 1 year ago

pdecat commented 1 year ago

FWIW, I had issues when building steampipe-postgres-fdw using the current golang:1.20 docker image (shared tag currently pointing to 1.20.2-bullseye for Linux).

Error at runtime is undefined symbol: __res_search:

# steampipe --install-dir /var/task/.steampipe plugin list
...
2023-03-21 14:23:58.671 UTC [TRACE] steampipe: Pinging
Error: plugin listing failed - could not load library "/var/task/.steampipe/db/14.2.0/postgres/lib/postgresql/steampipe_postgres_fdw.so": /var/task/.steampipe/db/14
.2.0/postgres/lib/postgresql/steampipe_postgres_fdw.so: undefined symbol: __res_search (SQLSTATE 58P01)

That symbol is provided by libresolv:

root@d9b971426328:/app# grep __res_search /usr/lib/x86_64-linux-gnu/*.so
grep: /usr/lib/x86_64-linux-gnu/libresolv.so: binary file matches

Inspecting the binary compiled with go 1.20:

root@d9b971426328:/app# nm build-Linux/steampipe_postgres_fdw.so |grep __res_search
                 U __res_search

root@d9b971426328:/app# ldd build-Linux/steampipe_postgres_fdw.so
        linux-vdso.so.1 (0x00007ffe477d2000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ffb79423000)
        /lib64/ld-linux-x86-64.so.2 (0x00007ffb7b736000)

Comparing that to the binary compiled with go 1.19, we can see that this symbol is not used:

root@c70a8b6ba81f:/app# nm build-Linux/steampipe_postgres_fdw.so | grep __res_search

root@c70a8b6ba81f:/app# ldd build-Linux/steampipe_postgres_fdw.so
        linux-vdso.so.1 (0x00007ffdeb88b000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fceef8ef000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fcef1bf4000)

Downgrading to golang:1.19 docker image resolves the issue.

Something changed on that front when using -buildmode=c-archive with go 1.20, but supposedly only for macOS according to https://github.com/golang/go/issues/58159

pdecat commented 1 year ago

~Defining SHLIB_LINK="-lresolv" seem to do the trick~.

Edit: it does not work because all symbols from steampipe_postgres_fdw.a are dropped.

Without it:

root@387a98fca485:/app# go version
go version go1.20.2 linux/amd64

root@387a98fca485:/app# ./build_binary.sh
rm -f steampipe_postgres_fdw.so   libsteampipe_postgres_fdw.a  libsteampipe_postgres_fdw.pc
rm -f datum.o query.o fdw.o logging.o datum.bc query.bc fdw.bc logging.bc
rm -rf steampipe_postgres_fdw.a fdw.h
rm -rf results/ regression.diffs regression.out tmp_check/ tmp_check_iso/ log/ output_iso/
go build -o steampipe_postgres_fdw.a -buildmode=c-archive ../*.go
gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Werror=vla -Wendif-labels -Wmissing-format-attribute -Wimplicit-fallthrough=3 -Wcast-function-type -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -Wno-format-truncation -Wno-stringop-truncation -O2 -I/usr/local/include/server -I/usr/local/include/internal -g -fPIC -I. -I./ -I/usr/local/include/postgresql/server -I/usr/local/include/postgresql/internal  -D_GNU_SOURCE   -c -o datum.o datum.c
gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Werror=vla -Wendif-labels -Wmissing-format-attribute -Wimplicit-fallthrough=3 -Wcast-function-type -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -Wno-format-truncation -Wno-stringop-truncation -O2 -I/usr/local/include/server -I/usr/local/include/internal -g -fPIC -I. -I./ -I/usr/local/include/postgresql/server -I/usr/local/include/postgresql/internal  -D_GNU_SOURCE   -c -o query.o query.c
In file included from /usr/local/include/postgresql/server/postgres.h:47,
                 from fdw_helpers.h:2,
                 from fdw.go:6,
                 from query.c:1:
query.c: In function 'canonicalOpExpr':
query.c:227:77: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast]
  227 |     elog(DEBUG5, "canonicalOpExpr, arg length: %d, base_relids %x", length, (int)base_relids);
      |                                                                             ^
/usr/local/include/postgresql/server/utils/elog.h:141:4: note: in definition of macro 'ereport_domain'
  141 |    __VA_ARGS__, errfinish(__FILE__, __LINE__, PG_FUNCNAME_MACRO); \
      |    ^~~~~~~~~~~
/usr/local/include/postgresql/server/utils/elog.h:233:2: note: in expansion of macro 'ereport'
  233 |  ereport(elevel, errmsg_internal(__VA_ARGS__))
      |  ^~~~~~~
query.c:227:5: note: in expansion of macro 'elog'
  227 |     elog(DEBUG5, "canonicalOpExpr, arg length: %d, base_relids %x", length, (int)base_relids);
      |     ^~~~
gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Werror=vla -Wendif-labels -Wmissing-format-attribute -Wimplicit-fallthrough=3 -Wcast-function-type -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -Wno-format-truncation -Wno-stringop-truncation -O2 -I/usr/local/include/server -I/usr/local/include/internal -g -fPIC -I. -I./ -I/usr/local/include/postgresql/server -I/usr/local/include/postgresql/internal  -D_GNU_SOURCE   -c -o fdw.o fdw.c
fdw.c: In function 'fdwGetForeignRelSize':
fdw.c:105:5: warning: ISO C90 forbids mixed declarations and code [-Wdeclaration-after-statement]
  105 |     FdwPlanState *planstate = palloc0(sizeof(FdwPlanState));
      |     ^~~~~~~~~~~~
fdw.c: In function 'fdwGetForeignPlan':
fdw.c:291:2: warning: ISO C90 forbids mixed declarations and code [-Wdeclaration-after-statement]
  291 |  ForeignScan * s = make_foreignscan(
      |  ^~~~~~~~~~~
gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Werror=vla -Wendif-labels -Wmissing-format-attribute -Wimplicit-fallthrough=3 -Wcast-function-type -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -Wno-format-truncation -Wno-stringop-truncation -O2 -I/usr/local/include/server -I/usr/local/include/internal -g -fPIC -I. -I./ -I/usr/local/include/postgresql/server -I/usr/local/include/postgresql/internal  -D_GNU_SOURCE   -c -o logging.o logging.c
gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Werror=vla -Wendif-labels -Wmissing-format-attribute -Wimplicit-fallthrough=3 -Wcast-function-type -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -Wno-format-truncation -Wno-stringop-truncation -O2 -I/usr/local/include/server -I/usr/local/include/internal -g -fPIC -shared -o steampipe_postgres_fdw.so datum.o query.o fdw.o logging.o -L/usr/local/lib    -Wl,--as-needed -Wl,-rpath,'/usr/local/lib',--enable-new-dtags  steampipe_postgres_fdw.a
mkdir -p ../build-Linux
rm -f ../build-Linux/*
cp steampipe_postgres_fdw.so ../build-Linux
cp steampipe_postgres_fdw.control ../build-Linux
cp steampipe_postgres_fdw--1.0.sql ../build-Linux
rm steampipe_postgres_fdw.so
rm steampipe_postgres_fdw.a
rm steampipe_postgres_fdw.h
rm ./*.o
/app

root@387a98fca485:/app# nm build-Linux/steampipe_postgres_fdw.so | grep __res_search
                 U __res_search

With it:

root@387a98fca485:/app# go version
go version go1.20.2 linux/amd64

root@387a98fca485:/app# cd fdw

root@387a98fca485:/app/fdw# make clean
rm -f steampipe_postgres_fdw.so   libsteampipe_postgres_fdw.a  libsteampipe_postgres_fdw.pc
rm -f datum.o query.o fdw.o logging.o datum.bc query.bc fdw.bc logging.bc
rm -rf steampipe_postgres_fdw.a fdw.h
rm -rf results/ regression.diffs regression.out tmp_check/ tmp_check_iso/ log/ output_iso/

root@387a98fca485:/app/fdw# make go
go build -o steampipe_postgres_fdw.a -buildmode=c-archive ../*.go

root@387a98fca485:/app/fdw# make SHLIB_LINK="-lresolv"
gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Werror=vla -Wendif-labels -Wmissing-format-attribute -Wimplicit-fallthrough=3 -Wcast-function-type -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -Wno-format-truncation -Wno-stringop-truncation -O2 -I/usr/local/include/server -I/usr/local/include/internal -g -fPIC -I. -I./ -I/usr/local/include/postgresql/server -I/usr/local/include/postgresql/internal  -D_GNU_SOURCE   -c -o datum.o datum.c
gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Werror=vla -Wendif-labels -Wmissing-format-attribute -Wimplicit-fallthrough=3 -Wcast-function-type -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -Wno-format-truncation -Wno-stringop-truncation -O2 -I/usr/local/include/server -I/usr/local/include/internal -g -fPIC -I. -I./ -I/usr/local/include/postgresql/server -I/usr/local/include/postgresql/internal  -D_GNU_SOURCE   -c -o query.o query.c
In file included from /usr/local/include/postgresql/server/postgres.h:47,
                 from fdw_helpers.h:2,
                 from fdw.go:6,
                 from query.c:1:
query.c: In function 'canonicalOpExpr':
query.c:227:77: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast]
  227 |     elog(DEBUG5, "canonicalOpExpr, arg length: %d, base_relids %x", length, (int)base_relids);
      |                                                                             ^
/usr/local/include/postgresql/server/utils/elog.h:141:4: note: in definition of macro 'ereport_domain'
  141 |    __VA_ARGS__, errfinish(__FILE__, __LINE__, PG_FUNCNAME_MACRO); \
      |    ^~~~~~~~~~~
/usr/local/include/postgresql/server/utils/elog.h:233:2: note: in expansion of macro 'ereport'
  233 |  ereport(elevel, errmsg_internal(__VA_ARGS__))
      |  ^~~~~~~
query.c:227:5: note: in expansion of macro 'elog'
  227 |     elog(DEBUG5, "canonicalOpExpr, arg length: %d, base_relids %x", length, (int)base_relids);
      |     ^~~~
gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Werror=vla -Wendif-labels -Wmissing-format-attribute -Wimplicit-fallthrough=3 -Wcast-function-type -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -Wno-format-truncation -Wno-stringop-truncation -O2 -I/usr/local/include/server -I/usr/local/include/internal -g -fPIC -I. -I./ -I/usr/local/include/postgresql/server -I/usr/local/include/postgresql/internal  -D_GNU_SOURCE   -c -o fdw.o fdw.c
fdw.c: In function 'fdwGetForeignRelSize':
fdw.c:105:5: warning: ISO C90 forbids mixed declarations and code [-Wdeclaration-after-statement]
  105 |     FdwPlanState *planstate = palloc0(sizeof(FdwPlanState));
      |     ^~~~~~~~~~~~
fdw.c: In function 'fdwGetForeignPlan':
fdw.c:291:2: warning: ISO C90 forbids mixed declarations and code [-Wdeclaration-after-statement]
  291 |  ForeignScan * s = make_foreignscan(
      |  ^~~~~~~~~~~
gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Werror=vla -Wendif-labels -Wmissing-format-attribute -Wimplicit-fallthrough=3 -Wcast-function-type -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -Wno-format-truncation -Wno-stringop-truncation -O2 -I/usr/local/include/server -I/usr/local/include/internal -g -fPIC -I. -I./ -I/usr/local/include/postgresql/server -I/usr/local/include/postgresql/internal  -D_GNU_SOURCE   -c -o logging.o logging.c
gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Werror=vla -Wendif-labels -Wmissing-format-attribute -Wimplicit-fallthrough=3 -Wcast-function-type -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -Wno-format-truncation -Wno-stringop-truncation -O2 -I/usr/local/include/server -I/usr/local/include/internal -g -fPIC -shared -o steampipe_postgres_fdw.so datum.o query.o fdw.o logging.o -L/usr/local/lib    -Wl,--as-needed -Wl,-rpath,'/usr/local/lib',--enable-new-dtags  -lresolv

root@387a98fca485:/app/fdw# nm steampipe_postgres_fdw.so | grep __res_search

root@387a98fca485:/app/fdw# make inst
mkdir -p ../build-Linux
rm -f ../build-Linux/*
cp steampipe_postgres_fdw.so ../build-Linux
cp steampipe_postgres_fdw.control ../build-Linux
cp steampipe_postgres_fdw--1.0.sql ../build-Linux
rm steampipe_postgres_fdw.so
rm steampipe_postgres_fdw.a
rm steampipe_postgres_fdw.h
rm ./*.o
root@387a98fca485:/app/fdw# cd ..

root@387a98fca485:/app# nm build-Linux/steampipe_postgres_fdw.so | grep __res_search
binaek commented 1 year ago

Handled by #339