Adding “LIMIT 1” to any SELECT causes segmentation fault

[pcoppinger]

I have the following extremely simple test case. It requires Postgresql 9.5.7, a Mongo database and uses the Mongo FDW Extension. It will fail using a table that has only one record. Here's how:

First, in the mongo shell, run the following command to create some data:

db.table2.insert({sessionKey:ObjectId('555233a2af8f312d060e57be'), catalog:'test'});

Then, run the below in PSQL:

CREATE TABLE table1 AS SELECT column1 AS sid
    FROM (VALUES ('555233a2af8f312d060e57be'::name)) AS data;

CREATE EXTENSION mongo_fdw;
CREATE SERVER mongo_data FOREIGN DATA WRAPPER mongo_fdw OPTIONS (address 'localhost', port '27017');
CREATE USER MAPPING FOR postgres SERVER mongo_data;

CREATE FOREIGN TABLE table2 (
  "sessionKey" NAME,
  "catalog" TEXT
) SERVER mongo_data OPTIONS (DATABASE 'test', COLLECTION 'table2');

When you run the following query, the server will throw a segmentation fault.

SELECT t1.sid, t2.catalog
FROM table1 t1
  LEFT OUTER JOIN LATERAL (
    SELECT catalog FROM table2 WHERE "sessionKey" = t1.sid LIMIT 1
  ) t2 ON TRUE;

However, if you run the query without the "LIMIT 1" it runs without a problem.

Looking at the execution plan, the only difference is the introduction of a filter expression.

# explain SELECT t1.sid, t2.catalog
FROM table1 t1
  LEFT OUTER JOIN LATERAL (
       SELECT catalog FROM table2 WHERE "sessionKey" = t1.sid        
       ) t2 ON TRUE;
                           QUERY PLAN                            
-----------------------------------------------------------------
 Nested Loop Left Join  (cost=5.00..6.08 rows=1 width=96)
   Join Filter: (table2."sessionKey" = t1.sid)
   ->  Seq Scan on table1 t1  (cost=0.00..1.01 rows=1 width=64)
   ->  Foreign Scan on table2  (cost=5.00..5.06 rows=1 width=96)
         Foreign Namespace: test.table2
(5 rows)

# explain SELECT t1.sid, t2.catalog
FROM table1 t1
  LEFT OUTER JOIN LATERAL (
       SELECT catalog FROM table2 WHERE "sessionKey" = t1.sid LIMIT 1
       ) t2 ON TRUE;
                              QUERY PLAN                               
-----------------------------------------------------------------------
 Nested Loop Left Join  (cost=5.00..6.09 rows=1 width=96)
   ->  Seq Scan on table1 t1  (cost=0.00..1.01 rows=1 width=64)
   ->  Limit  (cost=5.00..5.06 rows=1 width=32)
         ->  Foreign Scan on table2  (cost=5.00..5.06 rows=1 width=32)
               Filter: ("sessionKey" = t1.sid)
               Foreign Namespace: test.table2
(6 rows)

It appears that the evaluation of this filter expression is what causes the segmentation fault.

-- UPDATE 1 --

Running GDB, I received the following backtrace:

Program received signal SIGSEGV, Segmentation fault.
strlen () at ../sysdeps/x86_64/strlen.S:106
106     ../sysdeps/x86_64/strlen.S: No such file or directory.
(gdb) bt
#0  strlen () at ../sysdeps/x86_64/strlen.S:106
#1  0x000056452ce080db in MemoryContextStrdup ()
#2  0x000056452cdec0e7 in FunctionCall1Coll ()
#3  0x000056452cded6b7 in OutputFunctionCall ()
#4  0x000056452cded904 in OidOutputFunctionCall ()
#5  0x00007f056ee694f3 in AppenMongoValue 
    (queryDocument=queryDocument@entry=0x56452e217980, 
    keyName=keyName@entry=0x56452e242278 "sessionKey", value=0,
    isnull=<optimized out>, id=<optimized out>) at mongo_query.c:533
#6  0x00007f056ee69e02 in AppendParamValue (scanStateNode=0x56452e23abd8, 
    paramNode=<optimized out>, keyName=0x56452e242278 "sessionKey", 
    queryDocument=0x56452e217980)
    at mongo_query.c:410
#7  QueryDocument (relationId=relationId@entry=2207239, opExpressionList=
    <optimized out>, scanStateNode=scanStateNode@entry=0x56452e23abd8) at 
    mongo_query.c:218
#8  0x00007f056ee67f9f in MongoBeginForeignScan (scanState=0x56452e23abd8, 
    executorFlags=<optimized out>) at mongo_fdw.c:516
#9  0x000056452cc02315 in ExecInitForeignScan ()
#10 0x000056452cbe1a43 in ExecInitNode ()
#11 0x000056452cbf75bb in ExecInitLimit ()
#12 0x000056452cbe192d in ExecInitNode ()
#13 0x000056452cbfc537 in ExecInitNestLoop ()
#14 0x000056452cbe1a29 in ExecInitNode ()
#15 0x000056452cbdffba in standard_ExecutorStart ()
#16 0x000056452ccec1af in PortalStart ()
#17 0x000056452cce938d in PostgresMain ()
#18 0x000056452ca8305c in ?? ()
#19 0x000056452cc8d1b3 in PostmasterMain ()
#20 0x000056452ca84251 in main ()

Looking at the Mongo FDW code, I can see that the segfault is happening in mongo_query.c:

/* Prepare for parameter expression evaluation */
param_expr = ExecInitExpr((Expr *) paramNode, (PlanState *)scanStateNode);

/* Evaluate the parameter expression */
param_value = ExecEvalExpr(param_expr, econtext, &isNull, NULL);

AppenMongoValue(queryDocument, keyName, param_value, isNull,
            paramNode->paramtype);

I can paramNode->paramtype = 19 (NAMEOID), as expected. However, param_value = 0 (which is what is causing AppenMongoValue to crash). I'm not sure if this is a bug in the mongo_fdw extension or in Posgresql 9.5.7, but I need some help to find a way to work around this.

-- UPDATE 2 --

I dived into the PostgreSQL code (execQual.c, specifically) and I can see that there is no execution plan associated with the parameter "sessionKey".

static Datum
ExecEvalParamExec(ExprState *exprstate, ExprContext *econtext,
                  bool *isNull, ExprDoneCond *isDone)
{
    Param      *expression = (Param *) exprstate->expr;
    int         thisParamId = expression->paramid;
        ParamExecData *prm;

    if (isDone)
        *isDone = ExprSingleResult;

    /*
     * PARAM_EXEC params (internal executor parameters) are stored in the
     * ecxt_param_exec_vals array, and can be accessed by array index.
     */
    prm = &(econtext->ecxt_param_exec_vals[thisParamId]);
    if (prm->execPlan != NULL)
    {
    /* Parameter not evaluated yet, so go do it */
    ExecSetParamPlan(prm->execPlan, econtext);
    /* ExecSetParamPlan should have processed this param... */
    Assert(prm->execPlan == NULL);
    }
    *isNull = prm->isnull;
    return prm->value;
}

Because "prm->execPlan" is NULL, the value of the parameter is never set. I'm not sure that I can take this any further without help.

[ringerc]

Originally posted as https://stackoverflow.com/q/45274258/398670 btw

[ahsanhadi]

Thanks for reporting the bug. We will try and reproduce locally and get a fix committed ASAP.

[pcoppinger]

You’re welcome. It’s a strange one, no doubt about it…and if we can’t use LIMIT with LATERAL, then that kind of defeats one of the main reasons for using LATERAL in the first place. A lot of people will benefit from this fix when you have it.

Regards, Paul

On Jul 25, 2017, at 2:33 AM, Ahsan Hadi <notifications@github.com mailto:notifications@github.com> wrote:

Thanks for reporting the bug. We will try and reproduce locally and get a fix committed ASAP.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/EnterpriseDB/mongo_fdw/issues/82#issuecomment-317654972, or mute the thread https://github.com/notifications/unsubscribe-auth/ADzfhtmUsgnCt-BfBY38sFtLMmdjk6T5ks5sRZpFgaJpZM4OhzFw.

[pcoppinger]

Hi Ahsan,

By the way, I have my whole PG environment (PostgreSQL 9.5.7 and Mongo_FDW) running in my debugger right now. If you like, I can do some tests for you.

Regards, Paul

On Jul 25, 2017, at 2:33 AM, Ahsan Hadi notifications@github.com wrote:

Thanks for reporting the bug. We will try and reproduce locally and get a fix committed ASAP.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/EnterpriseDB/mongo_fdw/issues/82#issuecomment-317654972, or mute the thread https://github.com/notifications/unsubscribe-auth/ADzfhtmUsgnCt-BfBY38sFtLMmdjk6T5ks5sRZpFgaJpZM4OhzFw.

[pcoppinger]

So what do you think? Were you able to reproduce it on your side?

[ahsanhadi]

Hi,

I have run the given queries and it seem to be getting the correct result back, i am running this with 9.6...

edb=# SELECT t1.sid, t2.catalog edb-# FROM table1 t1 edb-# LEFT OUTER JOIN LATERAL ( edb(# SELECT catalog FROM table2 WHERE "sessionKey" = t1.sid LIMIT 1 edb(# ) t2 ON TRUE; sid | catalog --------------------------+--------- 555233a2af8f312d060e57be | test (1 row)

edb=# edb=# edb=# SELECT t1.sid, t2.catalog FROM table1 t1 LEFT OUTER JOIN LATERAL ( SELECT catalog FROM table2 WHERE "sessionKey" = t1.sid
) t2 ON TRUE; sid | catalog --------------------------+--------- 555233a2af8f312d060e57be | test (1 row)

edb=# SELECT t1.sid, t2.catalog edb-# FROM table1 t1 edb-# LEFT OUTER JOIN LATERAL ( edb(# SELECT catalog FROM table2 WHERE "sessionKey" = t1.sid LIMIT 1 edb(# ) t2 ON TRUE; sid | catalog --------------------------+--------- 555233a2af8f312d060e57be | test (1 row)

[pcoppinger]

Have you tried it on PG 9.5.7?

[pcoppinger]

Okay, I just upgraded my PG to 9.6.3 (via "brew install postgresql"), checked out REL-5_0_0, rebuilt and installed everything. It still crashes in exactly the same way.

upamsdata=# CREATE TABLE table1 AS SELECT column1 AS sid upamsdata-# FROM (VALUES ('555233a2af8f312d060e57be'::name)) AS data; SELECT 1 upamsdata=# CREATE SERVER mongo_data FOREIGN DATA WRAPPER mongo_fdw OPTIONS (address 'localhost', port '27017'); CREATE SERVER upamsdata=# CREATE USER MAPPING FOR postgres SERVER mongo_data; CREATE USER MAPPING upamsdata=# CREATE FOREIGN TABLE table2 ( upamsdata(# "sessionKey" NAME, upamsdata(# "catalog" TEXT upamsdata(# ) SERVER mongo_data OPTIONS (DATABASE 'test', COLLECTION 'table2'); CREATE FOREIGN TABLE upamsdata=# SELECT t1.sid, t2.catalog upamsdata-# FROM table1 t1 upamsdata-# LEFT OUTER JOIN LATERAL ( upamsdata(# SELECT catalog FROM table2 WHERE "sessionKey" = t1.sid LIMIT 1 upamsdata(# ) t2 ON TRUE; server closed the connection unexpectedly This probably means the server terminated abnormally before or while processing the request. The connection to the server was lost. Attempting reset: Failed.

[ahsanhadi]

Did you also rebuild mongodb_fdw?

[pcoppinger]

Absolutely.

make -f Makefile.meta clean make -f Makefile.meta make -f Makefile.meta install

[pcoppinger]

By the way, this just isn't on my machine. It happens on three different machines. Very repeatable.

[ringerc]

Sounds like you need to collect a backtrace.

[ahsanhadi]

Are you using mongo C driver or legacy?

[pcoppinger]

mongo-c.

$ cat Makefile.meta
# mongo_fdw/Makefile.meta
#
# Portions Copyright © 2004-2014, EnterpriseDB Corporation.
#
# Portions Copyright © 2012–2014 Citus Data, Inc.
#

MODULE_big = mongo_fdw

#
# We assume we are running on a POSIX compliant system (Linux, OSX). If you are
# on another platform, change env_posix.os in MONGO_OBJS with the appropriate
# environment object file.
#
LIBJSON = json-c
LIBJSON_OBJS =  $(LIBJSON)/json_util.o $(LIBJSON)/json_object.o $(LIBJSON)/json_tokener.o \
                            $(LIBJSON)/json_object_iterator.o $(LIBJSON)/printbuf.o $(LIBJSON)/linkhash.o \
                            $(LIBJSON)/arraylist.o $(LIBJSON)/random_seed.o $(LIBJSON)/debug.o

MONGO_INCLUDE = $(shell pkg-config --cflags libmongoc-1.0)
PG_CPPFLAGS = --std=c99 $(MONGO_INCLUDE) -I$(LIBJSON) -DMETA_DRIVER
SHLIB_LINK = $(shell pkg-config --libs libmongoc-1.0)

OBJS = connection.o option.o mongo_wrapper_meta.o mongo_fdw.o mongo_query.o $(LIBJSON_OBJS)

EXTENSION = mongo_fdw
DATA = mongo_fdw--1.0.sql

REGRESS = mongo_fdw
REGRESS_OPTS = --load-extension=$(EXTENSION)

#
# Users need to specify their Postgres installation path through pg_config. For
# example: /usr/local/pgsql/bin/pg_config or /usr/lib/postgresql/9.1/bin/pg_config
#

PG_CONFIG = pg_config
PGXS := $(shell $(PG_CONFIG) --pgxs)
include $(PGXS)

ifndef MAJORVERSION
    MAJORVERSION := $(basename $(VERSION))
endif

ifeq (,$(findstring $(MAJORVERSION), 9.3 9.4 9.5 9.6))
    $(error PostgreSQL 9.3, 9.4, 9.5 or 9.6 is required to compile this extension)
endif

[pcoppinger]

Also, the backtrace was provided in my original comments.

[pcoppinger]

What are your thought? Anything you want me to try?

[ringerc]

On 10 August 2017 at 23:49, Paul Coppinger notifications@github.com wrote:

Also, the backtrace was provided in my original comments

Ah, sorry. My bad, didn't check the history, I'm just a spectator here really.

[ahsanhadi]

What is your mongodb version?

[pcoppinger]

$ mongo --version
MongoDB shell version v3.4.6
git version: c55eb86ef46ee7aede3b1e2a5d184a7df4bfb5b5
OpenSSL version: OpenSSL 1.0.2l  25 May 2017
allocator: system
modules: none
build environment:
    distarch: x86_64
    target_arch: x86_64

[pcoppinger]

I have an update. It appears that this has nothing to do with the JOIN (LATERAL or not) and everything to do with LIMIT 1. In other words, the driver crashes Postgres any time you specify LIMIT 1.

[pcoppinger]

Did anyone ever fix this bug?

[andreasscherbaum]

Is this case still open? As in: is the "LIMIT 1" still crashing for you?

[vaibhavdalvi93]

Hi @pcoppinger , Are you still facing this issue with latest code base? Can you please check from you end? so that we can work on this ASAP.