Open devrimgunduz opened 3 years ago
rdunklau
commented
3 years ago Hello,
There is already a PR for that here: #260 but random errors are reported on different builds. I haven't been able to reproduce it or understand the cause yet, so this is still on hold.
pgsql-io
commented
3 years ago Any word on supporting PostgreSQL 13?
cpaelzer
commented
3 years ago Hi @rdunklau - did these random build issues resolve in the meantime or is there any new update available?
pgsql-io
commented
3 years ago I've had the same problem and have the same question. I'm a little technical, but... I couldn't figure out myself a suggested patch for basic compatibility of multicorn with Python3 and PG13.
On Mon, Nov 2, 2020 at 8:57 AM Christian Ehrhardt notifications@github.com wrote:
Hi @rdunklau https://github.com/rdunklau - did these random build issues resolve in the meantime or is there any new update available?
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/Segfault-Inc/Multicorn/issues/261#issuecomment-720487643, or unsubscribe https://github.com/notifications/unsubscribe-auth/AMHOUSIVYGOIFUYAXGBVYC3SN23E5ANCNFSM4QS4I5KQ .
cpaelzer
commented
3 years ago Well the initial patch seems to be in #260, yet it is incomplete for (to me) unknown reasons.
Saketha016
commented
3 years ago Hello,
I am running bitnami/postgresql chart which uses Debian 10 Buster. (Python 3.7; Debian 10; Postgresql 13)
After a successful apt-get install pgxnclient, I tried pgxn install multicorn and got this error:
root@postgresql-postgresql-ha-postgresql-1:/# pgxn install multicorn
INFO: best version: multicorn 1.4.0
INFO: saving /tmp/tmpg_fx0hy5/multicorn-1.4.0.zip
INFO: unpacking: /tmp/tmpg_fx0hy5/multicorn-1.4.0.zip
INFO: building extension
Python version is 3.7
[ -d sql ] || mkdir sql
[ -d src ] || mkdir src
touch directories.stamp
gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Werror=vla -Wendif-labels -Wmissing-format-attribute -Wimplicit-fallthrough=3 -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -Wno-format-truncation -Wno-stringop-truncation -s -fPIC -I/usr/include/python3.7m -I/usr/include/python3.7m -I. -I./ -I/opt/bitnami/postgresql/include/server -I/opt/bitnami/postgresql/include/internal -I/opt/bitnami/common/include -I/opt/bitnami/repmgr/include -I/opt/bitnami/postgresql/include -D_GNU_SOURCE -I/usr/include/libxml2 -c -o src/errors.o src/errors.c
src/errors.c: In function ‘reportException’:
src/errors.c:71:6: error: too many arguments to function ‘errstart’
if (errstart(severity, __FILE__, __LINE__, PG_FUNCNAME_MACRO, TEXTDOMAIN))
^~~~~~~~
In file included from /opt/bitnami/postgresql/include/server/postgres.h:47,
from src/multicorn.h:2,
from src/errors.c:15:
/opt/bitnami/postgresql/include/server/utils/elog.h:149:13: note: declared here
extern bool errstart(int elevel, const char *domain);
^~~~~~~~
src/errors.c:73:7: error: too many arguments to function ‘errstart’
if (errstart(severity, __FILE__, __LINE__, PG_FUNCNAME_MACRO, TEXTDOMAIN))
^~~~~~~~
In file included from /opt/bitnami/postgresql/include/server/postgres.h:47,
from src/multicorn.h:2,
from src/errors.c:15:
/opt/bitnami/postgresql/include/server/utils/elog.h:149:13: note: declared here
extern bool errstart(int elevel, const char *domain);
^~~~~~~~
src/errors.c:84:2: error: too few arguments to function ‘errfinish’
errfinish(0);
^~~~~~~~~
In file included from /opt/bitnami/postgresql/include/server/postgres.h:47,
from src/multicorn.h:2,
from src/errors.c:15:
/opt/bitnami/postgresql/include/server/utils/elog.h:150:13: note: declared here
extern void errfinish(const char *filename, int lineno, const char *funcname);
^~~~~~~~~
make: *** [<builtin>: src/errors.o] Error 1
ERROR: command returned 2: make PG_CONFIG=/opt/bitnami/postgresql/bin/pg_config all
root@postgresql-postgresql-ha-postgresql-1:/#
Couldn't figure out what's wrong in the file. Appreciate your help !!
wieck
commented
3 years ago The following commit by Tom Lane changed the call signatures of errstart() and errfinish(). This will require PG major version conditional code in Multicorn. I am not sure if these functions were supposed to be used directly by extensions.
commit 17a28b03645e27d73bf69a95d7569b61e58f06eb Author: Tom Lane tgl@sss.pgh.pa.us Date: Tue Mar 24 12:08:48 2020 -0400
Improve the internal implementation of ereport().
Change all the auxiliary error-reporting routines to return void,
now that we no longer need to pretend they are passing something
useful to errfinish(). While this probably doesn't save anything
significant at the machine-code level, it allows detection of some
additional types of mistakes.
Pass the error location details (__FILE__, __LINE__, PG_FUNCNAME_MACRO)
to errfinish not errstart. This shaves a few cycles off the case where
errstart decides we're not going to emit anything.
Re-implement elog() as a trivial wrapper around ereport(), removing
the separate support infrastructure it used to have. Aside from
getting rid of some now-surplus code, this means that elog() now
really does have exactly the same semantics as ereport(), in particular
that it can skip evaluation work if the message is not to be emitted.
Andres Freund and Tom Lane
Discussion: https://postgr.es/m/CA+fd4k6N8EjNvZpM8nme+y+05mz-SM8Z_BgkixzkA34R+ej0Kw@mail.gmail.comI checked the Multicorn code and something like the following code will compile past src/errors.c, but then I get more errors looking similar with regards to PG's list handling.
#if PG_MAJORVERSION_NUM < 13
if (errstart(severity, __FILE__, __LINE__, PG_FUNCNAME_MACRO, TEXTDOMAIN))
{
if (errstart(severity, __FILE__, __LINE__, PG_FUNCNAME_MACRO, TEXTDOMAIN))
#else
if (errstart(severity, TEXTDOMAIN))
{
if (errstart(severity, TEXTDOMAIN))
#endif
errmsg("Error in python: %s", errName);
errdetail("%s", errValue);
errdetail_log("%s", errTraceback);
}
Py_DECREF(pErrType);
Py_DECREF(pErrValue);
Py_DECREF(format_exception);
Py_DECREF(tracebackModule);
Py_DECREF(newline);
Py_DECREF(pTemp);
#if PG_MAJORVERSION_NUM < 13
errfinish(0);
#else
errfinish(__FILE__, __LINE__, PG_FUNCNAME_MACRO);
#endifThat Multicorn code segment also looks a bit odd. Is it intentional to have errstart() and errfinish() to be unbalanced?
rusty-jules
commented
2 years ago @wieck it seems that the debian folks have patched this issue. I got this to build in the postgres:13-buster container. Note that their patch is a version of postgresql-13-python3-multicorn.
FROM postgres:13-buster
RUN apt-get update && apt-get install -y \
python3-multicorn \
libpython3-dev \
python3-dev \
postgresql-server-dev-13 libpq-dev \
python3-setuptools
ENV PYTHON_OVERRIDE python3.7
# postgresql-13-python3-multicorn is only available on bullseye and bookworm,
# not buster so we need to build it ourselves
RUN git clone --depth 1 --branch debian/1.4.0-3 https://salsa.debian.org/postgresql/postgresql-multicorn && \
cd postgresql-multicorn && \
# apply the debian patches
git apply debian/patches/drop-install-target.diff && \
git apply debian/patches/dual-python-build.diff && \
git apply debian/patches/pg13 && \
make && make install && ldconfig && \
echo "create extension multicorn;" > /docker-entrypoint-initdb.d/multicorn.sqlAttached is my own patch file for s3csv_fdw which I got working in this container. Their README does not seem correct, creating a server should be done like so:
CREATE SERVER multicorn_csv FOREIGN DATA WRAPPER multicorn OPTIONS (wrapper 's3fdw.s3fdw.S3Fdw');This patch also completely ignores the user mapping settings, and relies on boto3's credential finder instead.
df7cb
commented
2 years ago @rusty-jules: The patch in the Debian package is simply what I pushed as #260, but as said there, the patch is incomplete. Someone with an understanding of PG's and Multicorn's list handling should sit down a bit with that to fix my naive attempt.
brauliobo
commented
2 years ago Same error here with Archlinux and its postgresql package:
➜ scheduler git:(master) ✗ sudo pgxn install multicorn
INFO: best version: multicorn 1.4.0
INFO: saving /tmp/tmp_ka2l9w2/multicorn-1.4.0.zip
INFO: unpacking: /tmp/tmp_ka2l9w2/multicorn-1.4.0.zip
INFO: building extension
Python version is 2.7
[ -d sql ] || mkdir sql
[ -d src ] || mkdir src
touch directories.stamp
gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Werror=vla -Wendif-labels -Wmissing-format-attribute -Wimplicit-fallthrough=3 -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -Wno-format-truncation -Wno-stringop-truncation -march=x86-64 -mtune=generic -O2 -pipe -fno-plt -fPIC -I. -I./ -I/usr/include/python2.7 -I. -I./ -I/usr/include/postgresql/server -I/usr/include/postgresql/internal -D_FORTIFY_SOURCE=2 -D_GNU_SOURCE -I/usr/include/libxml2 -c -o src/errors.o src/errors.c
src/errors.c: In function ‘reportException’:
src/errors.c:71:13: error: too many arguments to function ‘errstart’
71 | if (errstart(severity, __FILE__, __LINE__, PG_FUNCNAME_MACRO, TEXTDOMAIN))
| ^~~~~~~~
In file included from /usr/include/postgresql/server/postgres.h:47,
from src/multicorn.h:2,
from src/errors.c:15:
/usr/include/postgresql/server/utils/elog.h:149:13: note: declared here
149 | extern bool errstart(int elevel, const char *domain);
| ^~~~~~~~
src/errors.c:73:21: error: too many arguments to function ‘errstart’
73 | if (errstart(severity, __FILE__, __LINE__, PG_FUNCNAME_MACRO, TEXTDOMAIN))
| ^~~~~~~~
In file included from /usr/include/postgresql/server/postgres.h:47,
from src/multicorn.h:2,
from src/errors.c:15:
/usr/include/postgresql/server/utils/elog.h:149:13: note: declared here
149 | extern bool errstart(int elevel, const char *domain);
| ^~~~~~~~
src/errors.c:84:9: error: too few arguments to function ‘errfinish’
84 | errfinish(0);
| ^~~~~~~~~
In file included from /usr/include/postgresql/server/postgres.h:47,
from src/multicorn.h:2,
from src/errors.c:15:
/usr/include/postgresql/server/utils/elog.h:150:13: note: declared here
150 | extern void errfinish(const char *filename, int lineno, const char *funcname);
| ^~~~~~~~~
make: *** [<embutido>: src/errors.o] Erro 1
ERROR: command returned 2: make PG_CONFIG=/usr/bin/pg_config all
erichanson
commented
2 years ago Bump
vaidik
commented
1 year ago @wieck it seems that the debian folks have patched this issue. I got this to build in the postgres:13-buster container. Note that their patch is a version of
postgresql-13-python3-multicorn.FROM postgres:13-buster RUN apt-get update && apt-get install -y \ python3-multicorn \ libpython3-dev \ python3-dev \ postgresql-server-dev-13 libpq-dev \ python3-setuptools ENV PYTHON_OVERRIDE python3.7 # postgresql-13-python3-multicorn is only available on bullseye and bookworm, # not buster so we need to build it ourselves RUN git clone --depth 1 --branch debian/1.4.0-3 https://salsa.debian.org/postgresql/postgresql-multicorn && \ cd postgresql-multicorn && \ # apply the debian patches git apply debian/patches/drop-install-target.diff && \ git apply debian/patches/dual-python-build.diff && \ git apply debian/patches/pg13 && \ make && make install && ldconfig && \ echo "create extension multicorn;" > /docker-entrypoint-initdb.d/multicorn.sqlAttached is my own patch file for
s3csv_fdwwhich I got working in this container. Their README does not seem correct, creating a server should be done like so:CREATE SERVER multicorn_csv FOREIGN DATA WRAPPER multicorn OPTIONS (wrapper 's3fdw.s3fdw.S3Fdw');This patch also completely ignores the user mapping settings, and relies on
boto3's credential finder instead.
This works for me. But it is inconvenient for sure. Would be nice to have the fix in place.
luss
commented
1 year ago I would love it if my friend @wieck would look over the fixes I hacked into pgsql-io/multicorn2 to get it compiling and mostly working against pg13 & pg14. I hacked it into compiling and it seems to mostly work. I'd like to see a pg internals 3expert such as Jan look at my fixes for pg13 & pg14 whilst also making it work for pg15.
bbnickell
commented
1 year ago Would also like to see this fixed and for the patch to be fully rolled out for PG13, attempted with multicorn-1.4.0:
INFO: best version: multicorn 1.4.0
INFO: saving /tmp/tmp5fooo8ur/multicorn-1.4.0.zip
INFO: unpacking: /tmp/tmp5fooo8ur/multicorn-1.4.0.zip
INFO: building extension
Python version is 3.7
[ -d sql ] || mkdir sql
[ -d src ] || mkdir src
touch directories.stamp
gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Werror=vla -Wendif-labels -Wmissing-format-attribute -Wimplicit-fallthrough=3 -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -Wno-format-truncation -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -fPIC -I. -I./ -I/usr/include/python3.7m -I. -I./ -I/usr/include/pgsql/server -I/usr/include/pgsql/internal -D_GNU_SOURCE -I/usr/include/libxml2 -c -o src/errors.o src/errors.c
src/errors.c: In function ‘reportException’:
src/errors.c:71:6: error: too many arguments to function ‘errstart’
if (errstart(severity, __FILE__, __LINE__, PG_FUNCNAME_MACRO, TEXTDOMAIN))
^~~~~~~~
In file included from /usr/include/pgsql/server/postgres.h:47:0,
from src/multicorn.h:2,
from src/errors.c:15:
/usr/include/pgsql/server/utils/elog.h:149:13: note: declared here
extern bool errstart(int elevel, const char *domain);
^~~~~~~~
src/errors.c:73:7: error: too many arguments to function ‘errstart’
if (errstart(severity, __FILE__, __LINE__, PG_FUNCNAME_MACRO, TEXTDOMAIN))
^~~~~~~~
In file included from /usr/include/pgsql/server/postgres.h:47:0,
from src/multicorn.h:2,
from src/errors.c:15:
/usr/include/pgsql/server/utils/elog.h:149:13: note: declared here
extern bool errstart(int elevel, const char *domain);
^~~~~~~~
src/errors.c:84:2: error: too few arguments to function ‘errfinish’
errfinish(0);
^~~~~~~~~
In file included from /usr/include/pgsql/server/postgres.h:47:0,
from src/multicorn.h:2,
from src/errors.c:15:
/usr/include/pgsql/server/utils/elog.h:150:13: note: declared here
extern void errfinish(const char *filename, int lineno, const char *funcname);
^~~~~~~~~
gmake: *** [src/errors.o] Error 1
ERROR: command returned 2: gmake PG_CONFIG=/bin/pg_config all
Hi,
PostgreSQL is in beta3, and we are in the process of testing software. Multicorn fails to build against 13. Can you please take a look?
Thanks! Devrim
~~~~^ src/python.c:527:11: error: too few arguments to function 'lnext' 527~~~~^ src/python.c:528:11: error: too few arguments to function 'lnext' 528~~~~^ src/python.c:534:11: error: too few arguments to function 'lnext' 534~~~~^ src/python.c:535:11: error: too few arguments to function 'lnext' 535~~~~^ src/python.c:541:11: error: too few arguments to function 'lnext' 541In file included from /usr/pgsql-13/include/server/access/tupdesc.h:19, from /usr/pgsql-13/include/server/access/htup_details.h:19, from /usr/pgsql-13/include/server/access/relscan.h:17, from src/multicorn.h:3, from src/python.c:4: /usr/pgsql-13/include/server/nodes/pg_list.h:321:19: note: expected 'const List ' but argument is of type 'ListCell ' 321 | lnext(const List l, const ListCell c) |: src/python.o] Error 1
make[1]: Waiting for unfinished jobs....
src/errors.c: In function 'reportException':
src/errors.c:71:6: error: too many arguments to function 'errstart'
71 | if (errstart(severity, FILE, LINE, PG_FUNCNAME_MACRO, TEXTDOMAIN))
| ^: src/errors.o] Error 1
src/multicorn.c:146:13: error: use of undeclared identifier 'oid_hash'; did you mean 'uuid_hash'?
ctl.hash = oid_hash;
^ [/usr/pgsql-13/lib/pgxs/src/makefiles/../../src/Makefile.global:1051: src/multicorn.bc] Error 1
src/errors.c:71:35: error: too many arguments to function call, expected 2, have 5
if (errstart(severity, FILE, LINE, PG_FUNCNAME_MACRO, TEXTDOMAIN))
~~~~^ src/python.c:542:11: error: too few arguments to function 'lnext' 542 | cell2 = lnext(cell2); | ^~~~~ In file included from /usr/pgsql-13/include/server/access/tupdesc.h:19, from /usr/pgsql-13/include/server/access/htup_details.h:19, from /usr/pgsql-13/include/server/access/relscan.h:17, from src/multicorn.h:3, from src/python.c:4: /usr/pgsql-13/include/server/nodes/pg_list.h:321:1: note: declared here 321 | lnext(const List l, const ListCell c) | ^~~~~ make[1]: [~~~ In file included from /usr/pgsql-13/include/server/postgres.h:47, from src/multicorn.h:2, from src/errors.c:15: /usr/pgsql-13/include/server/utils/elog.h:149:13: note: declared here 149 | extern bool errstart(int elevel, const char domain); | ^~~~ src/errors.c:73:7: error: too many arguments to function 'errstart' 73 | if (errstart(severity, FILE, LINE, PG_FUNCNAME_MACRO, TEXTDOMAIN)) | ^~~~ In file included from /usr/pgsql-13/include/server/postgres.h:47, from src/multicorn.h:2, from src/errors.c:15: /usr/pgsql-13/include/server/utils/elog.h:149:13: note: declared here 149 | extern bool errstart(int elevel, const char domain); | ^~~~ src/errors.c:84:2: error: too few arguments to function 'errfinish' 84 | errfinish(0); | ^~~~~ In file included from /usr/pgsql-13/include/server/postgres.h:47, from src/multicorn.h:2, from src/errors.c:15: /usr/pgsql-13/include/server/utils/elog.h:150:13: note: declared here 150 | extern void errfinish(const char filename, int lineno, const char funcname); | ^~~~~ make[1]: * [~~~ uuid_hash /usr/pgsql-13/include/server/utils/fmgrprotos.h:1702:14: note: 'uuid_hash' declared here extern Datum uuid_hash(PG_FUNCTION_ARGS); ^ src/multicorn.c:146:11: warning: incompatible pointer types assigning to 'HashValueFunc' (aka 'unsigned int ()(const void , unsigned long)') from 'Datum (FunctionCallInfo)' (aka 'unsigned long (struct FunctionCallInfoBaseData )') [-Wincompatible-pointer-types] ctl.hash = oid_hash; ^~~~~ 1 warning and 1 error generated. make[1]: