Segmentation fault when connecting to SQL Server with an access token

[jesseryoungVUMC]

Environment

• Python: 3.8
• pyodbc: 4.0.34
• OS: Ubuntu 20.04
• DB: Azure SQL Server
• driver: msodbcsql17 (17.10)

Issue

When using the latest version of pyodbc (4.0.34) connecting to an Azure SQL Server with an access token a segmentation fault occurs. The same code does not produce a segmentation fault when running on 4.0.32.

Repro

Create an Azure SQL Server with AAD Auth enabled and your login as the server admin:

# Unique server name
SERVER_NAME=""
# Your public IP Address to add the firewall rule
PUBLIC_IP=""

az group create \
    --name issue-repro \
    --location centralus

az sql server create \
    --enable-ad-only-auth \
    --external-admin-principal-type User \
    --external-admin-name "$(az ad signed-in-user show --query "mail" --out tsv)" \
    --external-admin-sid "$(az ad signed-in-user show --query "objectId" --out tsv)" \
    --resource-group issue-repro \
    --name $SERVER_NAME

az sql server firewall-rule create \
    --resource-group issue-repro \
    --server $SERVER_NAME \
    --start-ip-address $PUBLIC_IP \
    --end-ip-address $PUBLIC_IP  \
    --name "Client IP"

The following will produce a segmentation fault on version 4.0.34 of pyodbc, but print "Connected" on version 4.0.32.

import struct

import pyodbc
from azure.identity import DefaultAzureCredential

server_address = "<server name>.database.windows.net"
azure_credential = DefaultAzureCredential()

# From https://github.com/mkleehammer/pyodbc/wiki/Connecting-to-SQL-Server-from-Linux
token_bytes = azure_credential.get_token("https://database.windows.net/").token.encode(
    "utf-16-le"
)
token_struct = struct.pack(f"<I{len(token_bytes)}s", len(token_bytes), token_bytes)
SQL_COPT_SS_ACCESS_TOKEN = (
    1256  # This connection option is defined by microsoft in msodbcsql.h
)

with pyodbc.connect(
    f"Driver={{ODBC Driver 17 for SQL Server}};Server={server_address};",
    attrs_before={SQL_COPT_SS_ACCESS_TOKEN: token_struct},
) as conn:
    print("Connected")

[v-chojas]

Could you post an ODBC trace of the connection, and if possible, attach a debugger to the python process so you can catch the segfault and get a stack trace?

[jesseryoungVUMC]

Yup: ODBC Trace

[ODBC][5169][1658175341.631500][__handles.c][460]
                Exit:[SQL_SUCCESS]
                        Environment = 0x55a9347d1190
[ODBC][5169][1658175341.631544][SQLSetEnvAttr.c][189]
                Entry:
                        Environment = 0x55a9347d1190
                        Attribute = SQL_ATTR_ODBC_VERSION
                        Value = 0x3
                        StrLen = 4
[ODBC][5169][1658175341.631555][SQLSetEnvAttr.c][364]
                Exit:[SQL_SUCCESS]
[ODBC][5169][1658175341.631564][SQLAllocHandle.c][375]
                Entry:
                        Handle Type = 2
                        Input Handle = 0x55a9347d1190
[ODBC][5169][1658175341.631595][SQLAllocHandle.c][493]
                Exit:[SQL_SUCCESS]
                        Output Handle = 0x55a9347d7340
[ODBC][5169][1658175341.631626][SQLSetConnectAttr.c][396]
                Entry:
                        Connection = 0x55a9347d7340
                        Attribute = 1256
                        Value = 0x55a9347d5f20
                        StrLen = -4
[ODBC][5169][1658175341.631642][SQLSetConnectAttr.c][671]
                Exit:[SQL_SUCCESS]
[ODBC][5169][1658175341.631680][SQLDriverConnectW.c][290]
                Entry:
                        Connection = 0x55a9347d7340
                        Window Hdl = (nil)
                        Str In = [Driver={ODBC Driver 17 for SQL Server};Server=****.database.windows.net;][length = 79 (SQL_NTS)]
                        Str Out = (nil)
                        Str Out Max = 0
                        Str Out Ptr = (nil)
                        Completion = 0
                UNICODE Using encoding ASCII 'UTF-8' and UNICODE 'UCS-2LE'

GDB backtrace (apologies if this isn't what you need - I'm not all that familiar with gdb)

#0  0x00007fa65feb9844 in ?? () from /opt/microsoft/msodbcsql17/lib64/libmsodbcsql-17.10.so.1.1
#1  0x00007fa65feb9583 in ?? () from /opt/microsoft/msodbcsql17/lib64/libmsodbcsql-17.10.so.1.1
#2  0x00007fa65feadae4 in ?? () from /opt/microsoft/msodbcsql17/lib64/libmsodbcsql-17.10.so.1.1
#3  0x00007fa65feae371 in ?? () from /opt/microsoft/msodbcsql17/lib64/libmsodbcsql-17.10.so.1.1
#4  0x00007fa65feb1619 in ?? () from /opt/microsoft/msodbcsql17/lib64/libmsodbcsql-17.10.so.1.1
#5  0x00007fa65feb1e34 in ?? () from /opt/microsoft/msodbcsql17/lib64/libmsodbcsql-17.10.so.1.1
#6  0x00007fa65fe20c89 in ?? () from /opt/microsoft/msodbcsql17/lib64/libmsodbcsql-17.10.so.1.1
#7  0x00007fa65fe5530e in ?? () from /opt/microsoft/msodbcsql17/lib64/libmsodbcsql-17.10.so.1.1
#8  0x00007fa65fe200aa in SQLDriverConnectW () from /opt/microsoft/msodbcsql17/lib64/libmsodbcsql-17.10.so.1.1
#9  0x00007fa66224c9a2 in SQLDriverConnectW () from /root/.cache/pypoetry/virtualenvs/python-workspace-O19yQgtn-py3.8/lib/python3.8/site-packages/pyodbc.libs/libodbc-2003e41d.so.2.0.0
#10 0x00007fa66249c2a9 in Connect (encoding=..., timeout=0, fAnsi=false, hdbc=0x55efef736100, pConnectString=0x7fa66034e5b0) at src/connection.cpp:114
#11 Connection_New (pConnectString=pConnectString@entry=0x7fa66034e5b0, fAutoCommit=false, fAnsi=<optimized out>, timeout=timeout@entry=0, fReadOnly=<optimized out>, attrs_before=<optimized out>, encoding=...) at src/connection.cpp:286
#12 0x00007fa6624a7e08 in mod_connect (self=<optimized out>, args=<optimized out>, kwargs=<optimized out>) at src/pyodbcmodule.cpp:554
#13 0x000055efedd39de5 in cfunction_call_varargs (func=0x7fa6626e5f40, args=<optimized out>, kwargs=<optimized out>) at Objects/call.c:743
#14 0x000055efedd39edd in _PyObject_MakeTpCall (callable=0x7fa6626e5f40, args=<optimized out>, nargs=<optimized out>, keywords=0x7fa66288c4c0) at Objects/call.c:159
#15 0x000055efedd23193 in _PyObject_Vectorcall (kwnames=0x7fa66288c4c0, nargsf=<optimized out>, args=<optimized out>, callable=0x7fa6626e5f40) at ./Include/cpython/abstract.h:125
#16 _PyObject_Vectorcall (kwnames=<optimized out>, nargsf=<optimized out>, args=<optimized out>, callable=<optimized out>) at ./Include/cpython/abstract.h:115
#17 call_function (tstate=tstate@entry=0x55efef0469b0, pp_stack=pp_stack@entry=0x7ffcce68a480, oparg=<optimized out>, kwnames=kwnames@entry=0x7fa66288c4c0) at Python/ceval.c:4963
#18 0x000055efedd26042 in _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3515
#19 0x000055efeddebf32 in PyEval_EvalFrameEx (throwflag=0, f=0x7fa662803440) at Python/ceval.c:741
#20 _PyEval_EvalCodeWithName (_co=_co@entry=0x7fa66275d9d0, globals=globals@entry=0x7fa662865380, locals=locals@entry=0x7fa662865380, args=args@entry=0x0, argcount=argcount@entry=0, kwnames=kwnames@entry=0x0, kwargs=0x0, kwcount=0, kwstep=2, defs=0x0, 

    defcount=0, kwdefs=0x0, closure=0x0, name=0x0, qualname=0x0) at Python/ceval.c:4298
#21 0x000055efeddec297 in PyEval_EvalCodeEx (closure=0x0, kwdefs=0x0, defcount=0, defs=0x0, kwcount=0, kws=0x0, argcount=0, args=0x0, locals=0x7fa662865380, globals=0x7fa662865380, _co=0x7fa66275d9d0) at Python/ceval.c:4327
#22 PyEval_EvalCode (co=co@entry=0x7fa66275d9d0, globals=globals@entry=0x7fa662865380, locals=locals@entry=0x7fa662865380) at Python/ceval.c:718
#23 0x000055efede2b044 in run_eval_code_obj (locals=0x7fa662865380, globals=0x7fa662865380, co=0x7fa66275d9d0) at Python/pythonrun.c:1165
#24 run_mod (mod=<optimized out>, filename=filename@entry=0x7fa662827f80, globals=globals@entry=0x7fa662865380, locals=locals@entry=0x7fa662865380, flags=flags@entry=0x7ffcce68a6f8, arena=arena@entry=0x7fa66285dd90) at Python/pythonrun.c:1187
#25 0x000055efede2d132 in pyrun_file (flags=0x7ffcce68a6f8, closeit=1, locals=0x7fa662865380, globals=0x7fa662865380, start=257, filename=0x7fa662827f80, fp=0x55efef042340) at Python/pythonrun.c:1084
--Type <RET> for more, q to quit, c to continue without paging--
#26 pyrun_simple_file (fp=fp@entry=0x55efef042340, filename=filename@entry=0x7fa662827f80, closeit=closeit@entry=1, flags=flags@entry=0x7ffcce68a6f8) at Python/pythonrun.c:439
#27 0x000055efede2d701 in PyRun_SimpleFileExFlags (flags=0x7ffcce68a6f8, closeit=1, filename=<optimized out>, fp=0x55efef042340) at Python/pythonrun.c:472
#28 PyRun_AnyFileExFlags (fp=fp@entry=0x55efef042340, filename=<optimized out>, closeit=closeit@entry=1, flags=flags@entry=0x7ffcce68a6f8) at Python/pythonrun.c:90
#29 0x000055efedd2d3bf in pymain_run_file (cf=0x7ffcce68a6f8, config=0x55efef045e20) at Modules/main.c:385
#30 pymain_run_python (exitcode=exitcode@entry=0x7ffcce68a830) at Modules/main.c:610
#31 0x000055efedd2d9ef in Py_RunMain () at Modules/main.c:689
#32 pymain_main (args=0x7ffcce68a7f0) at Modules/main.c:719
#33 Py_BytesMain (argc=<optimized out>, argv=<optimized out>) at Modules/main.c:743
#34 0x00007fa662957083 in __libc_start_main (main=0x55efedd21d80 <main>, argc=4, argv=0x7ffcce68a958, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffcce68a948) at ../csu/libc-start.c:308
#35 0x000055efedd2c5ce in _start () at ./Include/object.h:478

[v-chojas]

What version of unixODBC are you using? (You can check using odbcinst -j command.)

Leave ODBC trace enabled and direct it to trace to stdout (TraceFile=/dev/stdout). This will show the trace output in the console as the program runs. Run python your repro script under gdb, and when it catches the segfault and breaks, run x/i $rip to print the instruction where it segfaulted, and then p/x $rax. Compare this with the value passed in to SQLSetConnectAttr to set the access token (which should've appeared in the trace output.) Please show this information.

[jesseryoungVUMC]

# odbcinst -j
unixODBC 2.3.7
DRIVERS............: /etc/odbcinst.ini
SYSTEM DATA SOURCES: /etc/odbc.ini
FILE DATA SOURCES..: /etc/ODBCDataSources
USER DATA SOURCES..: /root/.odbc.ini
SQLULEN Size.......: 8
SQLLEN Size........: 8
SQLSETPOSIROW Size.: 8

GDB Output:

warning: Error disabling address space randomization: Operation not permitted
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[Detaching after fork from child process 4249]
[Detaching after fork from child process 4261]
[ODBC][4242][1658178603.355130][__handles.c][460]
                Exit:[SQL_SUCCESS]
                        Environment = 0x560b8f4c4020
[ODBC][4242][1658178603.355171][SQLSetEnvAttr.c][189]
                Entry:
                        Environment = 0x560b8f4c4020
                        Attribute = SQL_ATTR_ODBC_VERSION
                        Value = 0x3
                        StrLen = 4
[ODBC][4242][1658178603.355204][SQLSetEnvAttr.c][364]
                Exit:[SQL_SUCCESS]
[ODBC][4242][1658178603.355214][SQLAllocHandle.c][375]
                Entry:
                        Handle Type = 2
                        Input Handle = 0x560b8f4c4020
[ODBC][4242][1658178603.355225][SQLAllocHandle.c][493]
                Exit:[SQL_SUCCESS]
                        Output Handle = 0x560b8f4cb170
[ODBC][4242][1658178603.355235][SQLSetConnectAttr.c][396]
                Entry:
                        Connection = 0x560b8f4cb170
                        Attribute = 1256
                        Value = 0x560b8f4c9d50
                        StrLen = -4
[ODBC][4242][1658178603.355248][SQLSetConnectAttr.c][671]
                Exit:[SQL_SUCCESS]
[ODBC][4242][1658178603.355282][SQLDriverConnectW.c][290]
                Entry:
                        Connection = 0x560b8f4cb170
                        Window Hdl = (nil)
                        Str In = [Driver={ODBC Driver 17 for SQL Server};Server=*******.database.windows.net;][length = 79 (SQL_NTS)]
                        Str Out = (nil)
                        Str Out Max = 0
                        Str Out Ptr = (nil)
                        Completion = 0
                UNICODE Using encoding ASCII 'UTF-8' and UNICODE 'UCS-2LE'

Program received signal SIGSEGV, Segmentation fault.
0x00007f5790475f54 in ?? () from /opt/microsoft/msodbcsql17/lib64/libmsodbcsql-17.10.so.1.1
(gdb) x/i $rip
=> 0x7f5790475f54:      mov    (%rax),%edx
(gdb) p/x $rax
$1 = 0x8f4c9d50

[v-chojas]

... Value = 0x560b8f4c9d50 ... $1 = 0x8f4c9d50 http://www.unixodbc.org Fixed in 2.3.5: "Custom pre-connect pointer attributes are truncated to 32 bits "

Are you actually using unixODBC 2.3.7 Run

ldd  `which odbcinst`

and compare with an ldd on the pyODBC module itself (which you can find from within Python by showing the value of the pyodbc.__file__.) Where is it loading libodbc.so from? That is the DM lib.

[jesseryoungVUMC]

Interesting.... so, comparing the outputs of ldd on the pyODBC module between pyODBC versions 4.0.32 and 4.0.34 it looks like the newer version pyODBC is including libodbc. That tracks with what I see in the wheels over on PyPi. 4.0.32:

linux-vdso.so.1 (0x00007ffe369f1000)
libodbc.so.2 => /usr/lib/x86_64-linux-gnu/libodbc.so.2 (0x00007fd80d8ac000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fd80d6df000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fd80d59b000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fd80d581000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fd80d3bc000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fd80d3b6000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fd80d392000)
/lib64/ld-linux-x86-64.so.2 (0x00007fd80db55000)

4.0.34

linux-vdso.so.1 (0x00007ffd65909000)
libodbc-2003e41d.so.2.0.0 => /root/.cache/pypoetry/virtualenvs/python-workspace-O19yQgtn-py3.8/lib/python3.8/site-packages/pyodbc.libs/libodbc-2003e41d.so.2.0.0 (0x00007f9596ce8000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f9596b12000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f95969ce000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f95969b4000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f9596992000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f95967cd000)
libltdl-738907ff.so.7 => /root/.cache/pypoetry/virtualenvs/python-workspace-O19yQgtn-py3.8/lib/python3.8/site-packages/pyodbc.libs/libltdl-738907ff.so.7 (0x00007f95965c0000)
/lib64/ld-linux-x86-64.so.2 (0x00007f9597188000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f95965ba000)

[jesseryoungVUMC]

It looks like the lib included in the wheel is the culprit. Replacing libodbc-2003e41d.so.2.0.0 with a symlink to /usr/lib/x86_64-linux-gnu/libodbc.so.2 fixes my repro.

[sdebruyn]

I can consistently reproduce this with 4.0.34 and it goes away after downgrade to 4.0.32

/usr/local/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown                                                                                                                       
  warnings.warn('resource_tracker: There appear to be %d '

[Atheuz]

I also get this. 4.0.32 works fine, but upgrading to 4.0.34 causes segmentation faults when connecting to SQL Server with an AAD Access Token. It does work in some circumstances for me though, but not when it matters. Here's an example using sqlalchemy and threading, where it works without threading but if I introduce threading and a do_connect event listener it breaks:

from pyauth import BearerTokenGenerator
from sqlalchemy import create_engine
from sqlalchemy import event
import struct
from threading import Thread

class ODBCTester:
    def __init__(self):
        self.client_id = "12345-54321-12345-54321"
        self.cert_public = "public.crt"
        self.cert_private = "private.key"
        self.scope = ["https://database.windows.net/.default"]
        self.server = "12345.database.windows.net,1433"
        self.database = "Db_name"
        self.engine = None
        self.__setup()

    def __setup(self):
        token = ODBCTester.get_access_token(
            self.client_id,
            self.cert_public,
            self.cert_private,
            self.scope,
        )
        connection_info = ODBCTester.get_connection_args(
            token["access_token"], self.server, self.database
        )
        self.engine = create_engine(
            connection_info["connection_string"],
        )
        ODBCTester.add_event_listener(self.engine, self.client_id, self.cert_public, self.cert_private, self.scope,
                                      self.server, self.database)

    @staticmethod
    def get_connection_string(server, database, driver="{ODBC Driver 18 for SQL Server}"):
        params = f"Driver={driver};SERVER={server};DATABASE={database}"
        return f"mssql+pyodbc:///?odbc_connect={params}"

    @staticmethod
    def get_token_struct(token_str):
        token_bytes = bytes(token_str, "UTF-16-LE")
        return struct.pack(f"<I{len(token_bytes)}s", len(token_bytes), token_bytes)

    @staticmethod
    def add_event_listener(engine, client_id, cert_public, cert_private, scope, server, database):
        @event.listens_for(engine, "do_connect")
        def connect(dialect, conn_rec, cargs, cparams):
            cargs[0] = cargs[0].replace(";Trusted_Connection=Yes", "")
            token = ODBCTester.get_access_token(
                client_id,
                cert_public,
                cert_private,
                scope,
            )
            connection_info = ODBCTester.get_connection_args(
                token["access_token"], server, database
            )
            cparams["attrs_before"] = connection_info["connect_args"]

    @staticmethod
    def get_connection_args(access_token, server, db):
        sql_copt_ss_access_token = 1256

        connect_str = ODBCTester.get_connection_string(server=server, database=db)
        token_struct = ODBCTester.get_token_struct(access_token)

        return {
            "connection_string": connect_str,
            "connect_args": {sql_copt_ss_access_token: token_struct},
        }

    @staticmethod
    def get_access_token(client_id, public_cert, private_key, scope):
        # used to generate an access token of format {"access_token": "abcdef", "expires_in": 3600}
        token_generator = BearerTokenGenerator(client_id=client_id)
        access_token = token_generator.get_token_with_certificate(
            private_key_file=private_key, public_cert_file=public_cert, scope=scope
        )
        return access_token

    def run(self):
        result = self.engine.execute("SELECT 1")
        for row in result:
            print(row)

if __name__ == '__main__':
    runner = ODBCTester()
    # DOES WORK
    #runner.run()

    # DOES NOT WORK
    t = Thread(target=runner.run)
    t.start()
    t.join()

Similarly, if I go through the dbg steps outlined above:

dbg python
(gdb) run concurrency-app.py
Starting program: /home/user/.local/share/virtualenvs/odbc-test-MjM8mjaz/bin/python concurrency-app.py
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffff4601700 (LWP 311405)]
[ODBC][311402][1658239085.294393][__handles.c][460]
                Exit:[SQL_SUCCESS]
                        Environment = 0x7fffec0d30f0
[ODBC][311402][1658239085.294424][SQLSetEnvAttr.c][189]
                Entry:
                        Environment = 0x7fffec0d30f0
                        Attribute = SQL_ATTR_ODBC_VERSION
                        Value = 0x3
                        StrLen = 4
[ODBC][311402][1658239085.294430][SQLSetEnvAttr.c][364]
                Exit:[SQL_SUCCESS]
[ODBC][311402][1658239085.294435][SQLAllocHandle.c][375]
                Entry:
                        Handle Type = 2
                        Input Handle = 0x7fffec0d30f0
[ODBC][311402][1658239085.294439][SQLAllocHandle.c][493]
                Exit:[SQL_SUCCESS]
                        Output Handle = 0x7fffec0d0370
[ODBC][311402][1658239085.294444][SQLSetConnectAttr.c][396]
                Entry:
                        Connection = 0x7fffec0d0370
                        Attribute = 1256
                        Value = 0x7fffec0cf980
                        StrLen = -4
[ODBC][311402][1658239085.294449][SQLSetConnectAttr.c][671]
                Exit:[SQL_SUCCESS]
[ODBC][311402][1658239085.294482][SQLDriverConnectW.c][290]
                Entry:
                        Connection = 0x7fffec0d0370
                        Window Hdl = (nil)
                        Str In = ...
                        Str Out = (nil)
                        Str Out Max = 0
                        Str Out Ptr = (nil)
                        Completion = 0
                UNICODE Using encoding ASCII 'UTF-8' and UNICODE 'UCS-2LE'

Thread 2 "python" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff4601700 (LWP 311405)]
0x00007ffff3ae33e4 in ?? () from /opt/microsoft/msodbcsql18/lib64/libmsodbcsql-18.0.so.1.1
(gdb) x/i $rip
=> 0x7ffff3ae33e4:      mov    (%rax),%edx
(gdb) p/x $rax
$1 = 0xec0cf980
(gdb)

You see that 0xec0cf980 is missing some bits before it equals 0x7fffec0cf980.

Whereas, if I run without threading (runner.run()), I get this:

(gdb) run concurrency-app.py
Starting program: /home/user/.local/share/virtualenvs/odbc-test-MjM8mjaz/bin/python concurrency-app.py
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[ODBC][311653][1658239622.221048][__handles.c][460]
                Exit:[SQL_SUCCESS]
                        Environment = 0x1275f10
[ODBC][311653][1658239622.221105][SQLSetEnvAttr.c][189]
                Entry:
                        Environment = 0x1275f10
                        Attribute = SQL_ATTR_ODBC_VERSION
                        Value = 0x3
                        StrLen = 4
[ODBC][311653][1658239622.221111][SQLSetEnvAttr.c][364]
                Exit:[SQL_SUCCESS]
[ODBC][311653][1658239622.221116][SQLAllocHandle.c][375]
                Entry:
                        Handle Type = 2
                        Input Handle = 0x1275f10
[ODBC][311653][1658239622.221121][SQLAllocHandle.c][493]
                Exit:[SQL_SUCCESS]
                        Output Handle = 0x1267ba0
[ODBC][311653][1658239622.221127][SQLSetConnectAttr.c][396]
                Entry:
                        Connection = 0x1267ba0
                        Attribute = 1256
                        Value = 0x126b380
                        StrLen = -4
[ODBC][311653][1658239622.221148][SQLSetConnectAttr.c][671]
                Exit:[SQL_SUCCESS]
[ODBC][311653][1658239622.221202][SQLDriverConnectW.c][290]
                Entry:
                        Connection = 0x1267ba0
                        Window Hdl = (nil)
                        Str In = ...
                        Str Out = (nil)
                        Str Out Max = 0
                        Str Out Ptr = (nil)
                        Completion = 0
                UNICODE Using encoding ASCII 'UTF-8' and UNICODE 'UCS-2LE'

[ODBC][311653][1658239622.431374][__handles.c][460]
                Exit:[SQL_SUCCESS]
                        Environment = 0x1304f90
[ODBC][311653][1658239622.431413][SQLGetEnvAttr.c][157]
                Entry:
                        Environment = 0x1304f90
                        Attribute = 65002
                        Value = 0x7fffffff5d90
                        Buffer Len = 128
                        StrLen = 0x7fffffff5d2c
[ODBC][311653][1658239622.431419][SQLGetEnvAttr.c][264]
                Exit:[SQL_SUCCESS]
[ODBC][311653][1658239622.431424][SQLFreeHandle.c][219]
                Entry:
                        Handle Type = 1
                        Input Handle = 0x1304f90
(1,)
[Inferior 1 (process 311653) exited normally]

The pyodbc.__file__ for pyodbc==4.0.34 is /home/user/.local/share/virtualenvs/odbc-test-MjM8mjaz/lib/python3.9/site-packages/pyodbc.cpython-39-x86_64-linux-gnu.so, and here's the ldd of that:

$ ldd /home/user/.local/share/virtualenvs/odbc-test-MjM8mjaz/lib/python3.9/site-packages/pyodbc.cpython-39-x86_64-linux-gnu.so
        linux-vdso.so.1 (0x00007fff7e1d4000)
        libodbc-2003e41d.so.2.0.0 => /home/user/.local/share/virtualenvs/odbc-test-MjM8mjaz/lib/python3.9/site-packages/pyodbc.libs/libodbc-2003e41d.so.2.0.0 (0x00007f26775b0000)
        libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f26773bd000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f267726e000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f2677253000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f2677230000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f267703c000)
        libltdl-738907ff.so.7 => /home/user/.local/share/virtualenvs/odbc-test-MjM8mjaz/lib/python3.9/site-packages/pyodbc.libs/libltdl-738907ff.so.7 (0x00007f2676e31000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f2677a50000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f2676e2b000)

The pyodbc.__file__ for pyodbc==4.0.32 is /home/user/.local/share/virtualenvs/odbc-test-MjM8mjaz/lib/python3.9/site-packages/pyodbc.cpython-39-x86_64-linux-gnu.so, and here's the ldd of that:

ldd /home/user/.local/share/virtualenvs/odbc-test-MjM8mjaz/lib/python3.9/site-packages/pyodbc.cpython-39-x86_64-linux-gnu.so
        linux-vdso.so.1 (0x00007ffcb3dcf000)
        libodbc.so.2 => /lib/x86_64-linux-gnu/libodbc.so.2 (0x00007f4f789cb000)
        libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f4f787e9000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f4f7869a000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f4f7867f000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f4f7848d000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f4f78487000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f4f78462000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f4f78c7c000)

Seems like there's a new libodbc-2003e41d.so.2.0.0 and libltdl-738907ff.so.7 in 4.0.34 that are included in the package itself and not using the system ones.

[v-chojas]

Yes, that is definitely the problem. pyODBC should not be including unixODBC libs. I suspect the one that got included was a very old one, pre-2.3.5 as it still contains the attribute truncation bug.

It does work in some circumstances for me though

That's because if the access token is allocated below 4GB, it will work. You may still encounter some of the other bugs, however, depending on what version it was.

[gordthompson]

Issue submitted as #1082 . In the meantime, either pin pyodbc at 4.0.32 or use

pip install pyodbc --no-binary pyodbc

to install the latest version from source. Build tools, unixodbc-dev (or similar) will be required as before.

[jesseryoungVUMC]

It looks like there may be several CVE's out for older versions of unixodbc - it may be prudent to yank 4.0.34 from PyPi and anywhere else it's published.

https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-7409 https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-7485

Seeing as 4.0.32 -> 4.0.34 was only a patch version change, I'm willing to be a lot of people (such as myself) may have picked this change up without knowing due to their systems automatically accepting updates to libraries at the patch or minor version.

[Atheuz]

I think I found a hint to the problem in the build wheel pipeline: it installs unixodbc-dev using apt-get install unixodbc-dev, and the version that it finds is: Get:7 http://deb.debian.org/debian stretch/main amd64 unixodbc-dev amd64 2.3.4-1 [249 kB]

https://github.com/mkleehammer/pyodbc/runs/7347636270?check_suite_focus=true#step:3:216

From what @v-chojas said, it sounds like the problem is that it installed a version older than 2.3.5, which seems to be the case ^. Though not sure if it's supposed to include binaries in the wheel.

[v-chojas]

unixODBC >= 2.3.1 all have the same ABI so it is perfectly fine to build pyODBC on a system with an older version as the resulting binary will remain compatible with the newer versions; but redistributing it is what must not be done.

[gordthompson]

Covered by #1082

[sdebruyn]

If you ever would want an easy way to repro this, the automated functional tests in https://github.com/dbt-msft/dbt-sqlserver can repro this 100% of the time.

[moseleyi]

I downgraded to 4.0.32 and still getting Segmentation fault (using dbt)

[Atheuz]

I downgraded to 4.0.32 and still getting Segmentation fault (using dbt)

@moseleyi did you try 4.0.35? 4.0.34 is known to be bad, but I haven't seen seg fault on 4.0.32 or 4.0.35. If you're having issues on 4.0.32, maybe 4.0.35 works.

Also did you try upgrading unixodbc-dev? What version of it do you use? odbcinst -j.

[moseleyi]

I had 4.0.35 before that (must have been installed with the latest dbt core update?). I reinstalled to 4.0.35

$ pip install pyodbc --force
Collecting pyodbc
  Using cached pyodbc-4.0.35-cp38-cp38-win_amd64.whl (66 kB)
Installing collected packages: pyodbc
  Attempting uninstall: pyodbc
    Found existing installation: pyodbc 4.0.32
    Uninstalling pyodbc-4.0.32:
      Successfully uninstalled pyodbc-4.0.32
Successfully installed pyodbc-4.0.35

and still getting the error:

I don't have odbcinst:

[v-chojas]

What does ldd on the pyodbc lib show? See above discussion for the details.

[gordthompson]

@moseleyi - By chance are you running WSL on Windows?

[moseleyi]

no WSL, just w in the python (dbt) on Windows

[v-chojas]

python --version

What are you actually using...?

[gordthompson]

python --version

What are you actually using...?

IIRC, python --version returns a short response like Python 3.11.0.

python -VV

(that's double uppercase letter V) would be more informative. From within Python code, that would be

print(sys.version)

[moseleyi]

The thing is also that I can't find the reason why it's showing up. I can have 60 models running fine but there is one, that uses dbt macros that always fails with Segmentation fault error

[gordthompson]

Hmm, okay, it looks like you are using MINGW64 or similar. What ODBC driver are you using?

[v-chojas]

That's Windows so I don't think unixODBC is relevant.

Also, attach a debugger to the process and see where the segfault is, get a stack trace for more information.

[alascorz]

I might have a related issue. I've run what is mentioned in this thread, and I get:

odbcinst -j

unixODBC 2.3.11 DRIVERS............: /etc/odbcinst.ini SYSTEM DATA SOURCES: /etc/odbc.ini FILE DATA SOURCES..: /etc/ODBCDataSources USER DATA SOURCES..: /root/.odbc.ini SQLULEN Size.......: 8 SQLLEN Size........: 8 SQLSETPOSIROW Size.: 8

ldd which odbcinst

linux-vdso.so.1 (0x00007fff5f2d4000)
libodbcinst.so.2 => /usr/lib/x86_64-linux-gnu/libodbcinst.so.2 (0x00007fa496620000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fa4965ff000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fa49643f000)
libltdl.so.7 => /usr/lib/x86_64-linux-gnu/libltdl.so.7 (0x00007fa496434000)
/lib64/ld-linux-x86-64.so.2 (0x00007fa49683d000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fa49642f000)

ldd pyodbc.cpython-37m-x86_64-linux-gnu.so

linux-vdso.so.1 (0x00007fff2e598000)
libodbc.so.2 => /usr/lib/x86_64-linux-gnu/libodbc.so.2 (0x00007f5fbd841000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f5fbd6bd000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f5fbd53a000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f5fbd520000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f5fbd4ff000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f5fbd33f000)
libltdl.so.7 => /usr/lib/x86_64-linux-gnu/libltdl.so.7 (0x00007f5fbd332000)
/lib64/ld-linux-x86-64.so.2 (0x00007f5fbdae4000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f5fbd32d000)

I've tried with versions from 4.0.25 to 4.0.35 and they all fail with a segmentation fault when I try to create a connection. Any help would be greatly appreciated

[v-chojas]

Post an ODBC trace and/or use the discussion above to determine where the segfault is happening, to decide whether it is the same issue.

[gordthompson]

@alascorz - Please follow-up on this in #1188 . This issue is closed.