Segfault-Inc / Multicorn

Data Access Library
https://multicorn.org/
PostgreSQL License
701 stars 145 forks source link

Build fails against PostgreSQL 9.6 #171

Open devrimgunduz opened 7 years ago

devrimgunduz commented 7 years ago

Hi,

Getting this with 1.3.2:

+ make -j3
src/query.c: In function 'extractColumns':
src/query.c:66:19: error: too many arguments to function 'pull_var_clause'
   targetcolumns = pull_var_clause(node,
                   ^~~~~~~~~~~~~~~
In file included from src/query.c:2:0:
/usr/pgsql-9.6/include/server/optimizer/var.h:37:14: note: declared here
 extern List *pull_var_clause(Node *node, int flags);
              ^~~~~~~~~~~~~~~
src/query.c:77:19: error: too many arguments to function 'pull_var_clause'
   targetcolumns = pull_var_clause((Node *) node->clause,
                   ^~~~~~~~~~~~~~~
In file included from src/query.c:2:0:
/usr/pgsql-9.6/include/server/optimizer/var.h:37:14: note: declared here
 extern List *pull_var_clause(Node *node, int flags);
              ^~~~~~~~~~~~~~~
src/query.c: In function 'isAttrInRestrictInfo':
src/query.c:480:18: error: too many arguments to function 'pull_var_clause'
  List    *vars = pull_var_clause((Node *) restrictinfo->clause,
                  ^~~~~~~~~~~~~~~
In file included from src/query.c:2:0:
/usr/pgsql-9.6/include/server/optimizer/var.h:37:14: note: declared here
 extern List *pull_var_clause(Node *node, int flags);
              ^~~~~~~~~~~~~~~
src/query.c: In function 'findPaths':
src/query.c:656:32: error: 'RelOptInfo {aka struct RelOptInfo}' has no member named 'width'
                nbrows * baserel->width,
                                ^~
src/query.c:654:16: warning: passing argument 3 of 'create_foreignscan_path' makes pointer from integer without a cast [-Wint-conversion]
                nbrows,
                ^~~~~~
In file included from src/query.c:4:0:
/usr/pgsql-9.6/include/server/optimizer/pathnode.h:90:21: note: expected 'PathTarget * {aka struct PathTarget *}' but argument is of type 'int'
 extern ForeignPath *create_foreignscan_path(PlannerInfo *root, RelOptInfo *rel,
                     ^~~~~~~~~~~~~~~~~~~~~~~
In file included from /usr/pgsql-9.6/include/server/access/tupdesc.h:19:0,
                 from /usr/pgsql-9.6/include/server/utils/relcache.h:17,
                 from /usr/pgsql-9.6/include/server/access/genam.h:21,
                 from /usr/pgsql-9.6/include/server/access/relscan.h:17,
                 from src/multicorn.h:3,
                 from src/query.c:1:
/usr/pgsql-9.6/include/server/nodes/pg_list.h:69:18: error: incompatible type for argument 6 of 'create_foreignscan_path'
 #define NIL      ((List *) NULL)
                  ^
src/query.c:657:16: note: in expansion of macro 'NIL'
                NIL, /* no pathkeys */
                ^~~
In file included from src/query.c:4:0:
/usr/pgsql-9.6/include/server/optimizer/pathnode.h:90:21: note: expected 'Cost {aka double}' but argument is of type 'List * {aka struct List *}'
 extern ForeignPath *create_foreignscan_path(PlannerInfo *root, RelOptInfo *rel,
                     ^~~~~~~~~~~~~~~~~~~~~~~
src/query.c:652:19: error: too few arguments to function 'create_foreignscan_path'
     foreignPath = create_foreignscan_path(
In file included from src/query.c:4:0:
/usr/pgsql-9.6/include/server/optimizer/pathnode.h:90:21: note: declared here
 extern ForeignPath *create_foreignscan_path(PlannerInfo *root, RelOptInfo *rel,
                     ^~~~~~~~~~~~~~~~~~~~~~~
make[1]: *** [src/query.o] Error 1
make[1]: *** Waiting for unfinished jobs....
In file included from /usr/pgsql-9.6/include/server/access/tupdesc.h:19:0,
                 from /usr/pgsql-9.6/include/server/utils/relcache.h:17,
                 from /usr/pgsql-9.6/include/server/access/genam.h:21,
                 from /usr/pgsql-9.6/include/server/access/relscan.h:17,
                 from src/multicorn.h:3,
                 from src/multicorn.c:9:
src/multicorn.c: In function 'multicornGetForeignRelSize':
src/multicorn.c:271:37: error: 'RelOptInfo {aka struct RelOptInfo}' has no member named 'reltargetlist'; did you mean 'reltarget'?
   foreach(lc, extractColumns(baserel->reltargetlist, baserel->baserestrictinfo))
                                     ^
/usr/pgsql-9.6/include/server/nodes/pg_list.h:156:26: note: in definition of macro 'foreach'
  for ((cell) = list_head(l); (cell) != NULL; (cell) = lnext(cell))
                          ^
src/multicorn.c:295:54: error: 'RelOptInfo {aka struct RelOptInfo}' has no member named 'width'
  getRelSize(planstate, root, &baserel->rows, &baserel->width);
                                                      ^~
src/multicorn.c: In function 'multicornGetForeignPaths':
src/multicorn.c:329:27: error: 'RelOptInfo {aka struct RelOptInfo}' has no member named 'width'
    baserel->rows * baserel->width,
                           ^~
src/multicorn.c:327:4: error: incompatible type for argument 3 of 'create_foreignscan_path'
    baserel->rows,
    ^~~~~~~
In file included from src/multicorn.c:11:0:
/usr/pgsql-9.6/include/server/optimizer/pathnode.h:90:21: note: expected 'PathTarget * {aka struct PathTarget *}' but argument is of type 'double'
 extern ForeignPath *create_foreignscan_path(PlannerInfo *root, RelOptInfo *rel,
                     ^~~~~~~~~~~~~~~~~~~~~~~
In file included from /usr/pgsql-9.6/include/server/access/tupdesc.h:19:0,
                 from /usr/pgsql-9.6/include/server/utils/relcache.h:17,
                 from /usr/pgsql-9.6/include/server/access/genam.h:21,
                 from /usr/pgsql-9.6/include/server/access/relscan.h:17,
                 from src/multicorn.h:3,
                 from src/multicorn.c:9:
/usr/pgsql-9.6/include/server/nodes/pg_list.h:69:18: error: incompatible type for argument 6 of 'create_foreignscan_path'
 #define NIL      ((List *) NULL)
                  ^
src/multicorn.c:330:4: note: in expansion of macro 'NIL'
    NIL,  /* no pathkeys */
    ^~~   
In file included from src/multicorn.c:11:0:
/usr/pgsql-9.6/include/server/optimizer/pathnode.h:90:21: note: expected 'Cost {aka double}' but argument is of type 'List * {aka struct List *}'
 extern ForeignPath *create_foreignscan_path(PlannerInfo *root, RelOptInfo *rel,
                     ^~~~~~~~~~~~~~~~~~~~~~~
src/multicorn.c:326:27: error: too few arguments to function 'create_foreignscan_path'
  pathes = lappend(pathes, create_foreignscan_path(root, baserel,
                           ^~~~~~~~~~~~~~~~~~~~~~~
In file included from src/multicorn.c:11:0:
/usr/pgsql-9.6/include/server/optimizer/pathnode.h:90:21: note: declared here
 extern ForeignPath *create_foreignscan_path(PlannerInfo *root, RelOptInfo *rel,
                     ^~~~~~~~~~~~~~~~~~~~~~~
src/multicorn.c:363:53: error: incompatible type for argument 3 of 'create_foreignscan_path'
    newpath = create_foreignscan_path(root, baserel, path->path.rows,
                                                     ^~~~
In file included from src/multicorn.c:11:0:
/usr/pgsql-9.6/include/server/optimizer/pathnode.h:90:21: note: expected 'PathTarget * {aka struct PathTarget *}' but argument is of type 'double'
 extern ForeignPath *create_foreignscan_path(PlannerInfo *root, RelOptInfo *rel,
                     ^~~~~~~~~~~~~~~~~~~~~~~
src/multicorn.c:365:6: error: incompatible type for argument 6 of 'create_foreignscan_path'
      apply_pathkeys, NULL,
      ^~~~~~~~~~~~~~
In file included from src/multicorn.c:11:0:
/usr/pgsql-9.6/include/server/optimizer/pathnode.h:90:21: note: expected 'Cost {aka double}' but argument is of type 'List * {aka struct List *}'
 extern ForeignPath *create_foreignscan_path(PlannerInfo *root, RelOptInfo *rel,
                     ^~~~~~~~~~~~~~~~~~~~~~~
src/multicorn.c:363:14: error: too few arguments to function 'create_foreignscan_path'
    newpath = create_foreignscan_path(root, baserel, path->path.rows,
              ^~~~~~~~~~~~~~~~~~~~~~~
In file included from src/multicorn.c:11:0:
/usr/pgsql-9.6/include/server/optimizer/pathnode.h:90:21: note: declared here
 extern ForeignPath *create_foreignscan_path(PlannerInfo *root, RelOptInfo *rel,
                     ^~~~~~~~~~~~~~~~~~~~~~~
make[1]: *** [src/multicorn.o] Error 1
error: Bad exit status from /var/tmp/rpm-tmp.ANvcbn (%build)
    Bad exit status from /var/tmp/rpm-tmp.ANvcbn (%build)
rpmsign: no arguments given
../../../common/Makefile.global:58: recipe for target 'build' failed
make: *** [build] Error 1

Regards, Devrim

rdunklau commented 7 years ago

I just releeased 1.3.3, let me know how it works.

df7cb commented 7 years ago

Thanks for the new release!

I'm getting regression failures both with python 2 and 3 (look identical, below is the python3 version):

**** regression.diffs ****
*** /tmp/autopkgtest.o9pDAE/build.svm/postgresql-multicorn-1.3.3/test-3.5/expected/multicorn_planner_test.out   2016-09-27 09:51:11.000000000 +0200
--- /tmp/autopkgtest.o9pDAE/build.svm/postgresql-multicorn-1.3.3/results/multicorn_planner_test.out 2016-09-27 13:10:32.526002201 +0200
***************
*** 17,23 ****
  NOTICE:  [('test1', 'character varying'), ('test2', 'character varying')]
                                QUERY PLAN                              
  ----------------------------------------------------------------------
!  Foreign Scan on testmulticorn  (cost=10.00..400.00 rows=20 width=20)
  (1 row)

  explain select * from testmulticorn m1 inner join testmulticorn m2 on m1.test1 = m2.test1;
--- 17,23 ----
  NOTICE:  [('test1', 'character varying'), ('test2', 'character varying')]
                                QUERY PLAN                              
  ----------------------------------------------------------------------
!  Foreign Scan on testmulticorn  (cost=10.00..400.00 rows=20 width=64)
  (1 row)

  explain select * from testmulticorn m1 inner join testmulticorn m2 on m1.test1 = m2.test1;
***************
*** 55,61 ****
  NOTICE:  [('test1', 'character varying'), ('test2', 'character varying')]
                                      QUERY PLAN                                    
  ----------------------------------------------------------------------------------
!  Foreign Scan on testmulticorn  (cost=10.00..200000000.00 rows=10000000 width=20)
  (1 row)

  explain select * from testmulticorn m1 inner join testmulticorn m2 on m1.test1 = m2.test1;
--- 55,61 ----
  NOTICE:  [('test1', 'character varying'), ('test2', 'character varying')]
                                      QUERY PLAN                                    
  ----------------------------------------------------------------------------------
!  Foreign Scan on testmulticorn  (cost=10.00..200000000.00 rows=10000000 width=64)
  (1 row)

  explain select * from testmulticorn m1 inner join testmulticorn m2 on m1.test1 = m2.test1;

======================================================================

*** /tmp/autopkgtest.o9pDAE/build.svm/postgresql-multicorn-1.3.3/test-3.5/expected/multicorn_test_sort.out  2016-09-27 09:51:11.000000000 +0200
--- /tmp/autopkgtest.o9pDAE/build.svm/postgresql-multicorn-1.3.3/results/multicorn_test_sort.out    2016-09-27 13:10:32.922002571 +0200
***************
*** 17,23 ****
  NOTICE:  [('test1', 'date'), ('test2', 'timestamp without time zone')]
                                QUERY PLAN                              
  ----------------------------------------------------------------------
!  Foreign Scan on testmulticorn  (cost=10.00..400.00 rows=20 width=20)
  (1 row)

  -- Data should be sorted
--- 17,23 ----
  NOTICE:  [('test1', 'date'), ('test2', 'timestamp without time zone')]
                                QUERY PLAN                              
  ----------------------------------------------------------------------
!  Foreign Scan on testmulticorn  (cost=10.00..400.00 rows=20 width=12)
  (1 row)

  -- Data should be sorted

======================================================================

*** /tmp/autopkgtest.o9pDAE/build.svm/postgresql-multicorn-1.3.3/test-3.5/expected/write_filesystem.out 2016-09-27 09:51:11.000000000 +0200
--- /tmp/autopkgtest.o9pDAE/build.svm/postgresql-multicorn-1.3.3/results/write_filesystem.out   2016-09-27 13:10:35.066004580 +0200
***************
*** 72,99 ****
  EXPLAIN select color, size from testmulticorn where color = 'blue' and size = 'big' and name = 'square' and ext = 'txt';
                                                                     QUERY PLAN                                                                    
  -------------------------------------------------------------------------------------------------------------------------------------------------
!  Foreign Scan on testmulticorn  (cost=20.00..120.00 rows=1 width=120)
     Filter: (((color)::text = 'blue'::text) AND ((size)::text = 'big'::text) AND ((name)::text = 'square'::text) AND ((ext)::text = 'txt'::text))
  (2 rows)

  EXPLAIN select color, size from testmulticorn where color = 'blue' and size = 'big';
                                   QUERY PLAN                                  
  -----------------------------------------------------------------------------
!  Foreign Scan on testmulticorn  (cost=20.00..6000.00 rows=100 width=60)
     Filter: (((color)::text = 'blue'::text) AND ((size)::text = 'big'::text))
  (2 rows)

  EXPLAIN select color, size from testmulticorn where color = 'blue';
                                   QUERY PLAN                                 
  ----------------------------------------------------------------------------
!  Foreign Scan on testmulticorn  (cost=20.00..600000.00 rows=10000 width=60)
     Filter: ((color)::text = 'blue'::text)
  (2 rows)

  EXPLAIN select color, size, data from testmulticorn where color = 'blue' and size = 'big' and name = 'square' and ext = 'txt';
                                                                     QUERY PLAN                                                                    
  -------------------------------------------------------------------------------------------------------------------------------------------------
!  Foreign Scan on testmulticorn  (cost=20.00..1000150.00 rows=1 width=1000150)
     Filter: (((color)::text = 'blue'::text) AND ((size)::text = 'big'::text) AND ((name)::text = 'square'::text) AND ((ext)::text = 'txt'::text))
  (2 rows)

--- 72,99 ----
  EXPLAIN select color, size from testmulticorn where color = 'blue' and size = 'big' and name = 'square' and ext = 'txt';
                                                                     QUERY PLAN                                                                    
  -------------------------------------------------------------------------------------------------------------------------------------------------
!  Foreign Scan on testmulticorn  (cost=20.00..120.00 rows=1 width=64)
     Filter: (((color)::text = 'blue'::text) AND ((size)::text = 'big'::text) AND ((name)::text = 'square'::text) AND ((ext)::text = 'txt'::text))
  (2 rows)

  EXPLAIN select color, size from testmulticorn where color = 'blue' and size = 'big';
                                   QUERY PLAN                                  
  -----------------------------------------------------------------------------
!  Foreign Scan on testmulticorn  (cost=20.00..6000.00 rows=100 width=64)
     Filter: (((color)::text = 'blue'::text) AND ((size)::text = 'big'::text))
  (2 rows)

  EXPLAIN select color, size from testmulticorn where color = 'blue';
                                   QUERY PLAN                                 
  ----------------------------------------------------------------------------
!  Foreign Scan on testmulticorn  (cost=20.00..600000.00 rows=10000 width=64)
     Filter: ((color)::text = 'blue'::text)
  (2 rows)

  EXPLAIN select color, size, data from testmulticorn where color = 'blue' and size = 'big' and name = 'square' and ext = 'txt';
                                                                     QUERY PLAN                                                                    
  -------------------------------------------------------------------------------------------------------------------------------------------------
!  Foreign Scan on testmulticorn  (cost=20.00..1000150.00 rows=1 width=96)
     Filter: (((color)::text = 'blue'::text) AND ((size)::text = 'big'::text) AND ((name)::text = 'square'::text) AND ((ext)::text = 'txt'::text))
  (2 rows)

======================================================================

*** /tmp/autopkgtest.o9pDAE/build.svm/postgresql-multicorn-1.3.3/test-3.5/expected/import_test.out  2016-09-27 09:51:11.000000000 +0200
--- /tmp/autopkgtest.o9pDAE/build.svm/postgresql-multicorn-1.3.3/results/import_test.out    2016-09-27 13:10:35.542005026 +0200
***************
*** 33,40 ****
  SELECT relname FROM pg_class c INNER JOIN pg_namespace n on c.relnamespace = n.oid WHERE n.nspname = 'import_dest1';
       relname      
  ------------------
-  imported_table_2
   imported_table_1
  (2 rows)

  DROP EXTENSION multicorn cascade;
--- 33,40 ----
  SELECT relname FROM pg_class c INNER JOIN pg_namespace n on c.relnamespace = n.oid WHERE n.nspname = 'import_dest1';
       relname      
  ------------------
   imported_table_1
+  imported_table_2
  (2 rows)

  DROP EXTENSION multicorn cascade;

======================================================================

(These look harmless so I'm going to add some alternative _1.out expected files as a hotfix here.)

df7cb commented 7 years ago

The actual patch I added to the Debian package is at https://anonscm.debian.org/cgit/pkg-postgresql/postgresql-multicorn.git/tree/debian/patches/9.6 .

rdunklau commented 7 years ago

Yes, the problem lies with how the path width is calculated. It has changed in 9.6, where the size of the individual attributes is used instead of trusting what the fdw explicitly set on the path.

I don't really know how to fix that in a way that would succeed on every version: what do you mean by "adding some alternative_1.out" file ?

df7cb commented 7 years ago

If some pg_regress sql/foo.sql regression test file has two valid results, you can create expected/foo.out and expected/foo_1.out and pg_regress will figure out if the produced results/foo.out file will match either of these. (And if not, it will show you the smaller diff.)

See the link I posted above for the actual files I added. (Unfortunately there's no way to add comments to the *.out files, the only way to tell what they are good for is diffing foo.out and foo_1.out, or to add some README file in that directory.)

rdunklau commented 7 years ago

Ah that makes sense, I didn't know about that feature.

j0k3r commented 6 years ago

I just used the 1.3.4 version with PostgreSQL 9.6 and everything works fine.

df7cb commented 6 years ago

@j0k3r: I'm still seeing the same regression failure as quoted above. Did you run the regression tests?

j0k3r commented 6 years ago

Sorry I tested it in a project where we used postgres 9.5 with Multicorn 1.3.2, upgrading PG to 9.6 results in error posted by OP. Upgrading Multicorn to 1.3.4 fixed that issue. It might be working only for me then. If so, sorry for the noise.