search

apache / arrow-adbc

Database connectivity API standard and libraries for Apache Arrow
https://arrow.apache.org/adbc/
Apache License 2.0
386 stars 98 forks source link

adbc_driver_postgresql.dbapi causes silent program exit on connection attempt to db #1531

Open programmingwithalex opened 10 months ago

programmingwithalex commented 10 months ago

Config:

Program exits silently (no exceptions raised) on adbc_driver_postgresql.dbapi.connect(uri).

import adbc_driver_postgresql.dbapi

uri = (
    'postgresql://' +
    f'{config["USERNAME"]}:' +
    f'{config["PASSWORD"]}@' +
    f'{config["HOST"]}:' +
    '5432/' +
    f'{config["DB_NAME"]}'
)

with adbc_driver_postgresql.dbapi.connect(uri) as conn:  # program exits silently here
   pass

Debugging

import sqlalchemy

uri = (
    'postgresql://' +
    f'{config["USERNAME"]}:' +
    f'{config["PASSWORD"]}@' +
    f'{config["HOST"]}:' +
    '5432/' +
    f'{config["DB_NAME"]}'
)

engine = sqlalchemy.create_engine(uri)

with engine.connect() as conn:
    pass
lidavidm commented 10 months ago

Thanks for the report. Windows support is a bit difficult for me but I will spin up a VM to investigate when I get a chance.

lidavidm commented 10 months ago

Hmm, I am unable to reproduce this on a Windows 11 developer VM

PS C:\Users\User> python
Python 3.11.6 (tags/v3.11.6:8b6ee5b, Oct  2 2023, 14:57:12) [MSC v.1935 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import adbc_driver_postgresql.dbapi
>>> adbc_driver_postgresql.dbapi.connect("postgresql://postgres:password@10.0.2.2:5432/postgres")
<adbc_driver_manager.dbapi.Connection object at 0x0000025F132CB850>
>>> exit()
PS C:\Users\User> pip freeze
adbc-driver-manager==0.9.0
adbc-driver-postgresql==0.9.0
colorama==0.4.6
duckdb==0.9.2
iniconfig==2.0.0
numpy==1.26.4
packaging==23.2
pandas==2.2.0
pluggy==1.4.0
pyarrow==15.0.0
pytest==8.0.0
python-dateutil==2.8.2
pytz==2024.1
six==1.16.0
typing_extensions==4.9.0
tzdata==2023.4

I tried the context manager as well in case that was the issue, but no dice.

PS C:\Users\User> python
Python 3.11.6 (tags/v3.11.6:8b6ee5b, Oct  2 2023, 14:57:12) [MSC v.1935 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import adbc_driver_postgresql.dbapi
>>> import adbc_driver_postgresql.dbapi
KeyboardInterrupt
>>> conn = adbc_driver_postgresql.dbapi.connect("postgresql://postgres:password@10.0.2.2:5432/postgres")
>>> with conn:
...     pass
...
>>> conn
<adbc_driver_manager.dbapi.Connection object at 0x0000027E1B888650>
>>> with adbc_driver_postgresql.dbapi.connect("postgresql://postgres:password@10.0.2.2:5432/postgres") as conn:
...     with conn.cursor() as cur:
...         cur.execute("SELECT 1")
...         print(cur.fetchone())
...
(1,)
>>> exit()
programmingwithalex commented 10 months ago

I should have been more specific to simulate the error. The PostgreSQL database is hosted on AWS RDS and I'm running Windows 10 now.

It's so strange that you don't replicate the error though either. I can't think of any AWS RDS db configurations that would allow connections with sqlalchemy but not adbc_driver_postgresql.dbapi. Could you please try to replicate the issue once more on Windows 10?

lidavidm commented 10 months ago

Do you know of a way to get a Windows 10 developer VM?

programmingwithalex commented 9 months ago

Unfortunately I don't and based on my research Windows no longer supports them. I'm hoping someone else has the same issue and discovers this thread to contribute.

The only other information I can give is when I attempt the connection I get the hourglass next to the mouse indicating some sort of processing is going on for a second before the python program dies.

lidavidm commented 9 months ago

Ok. It turns out, I dug up an old laptop (that I was planning on donating/recycling) but it happens to still be on Windows 10. When I can find more time again I'll give it another shot.

programmingwithalex commented 9 months ago

You're the man David!

lidavidm commented 9 months ago

No dice, unfortunately:

PS C:\Windows\system32> python
Python 3.11.8 (tags/v3.11.8:db85d51, Feb  6 2024, 22:03:32) [MSC v.1937 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import adbc_driver_postgresql.dbapi
>>> uri = "postgresql://postgres:password@10.0.0.201:5432/postgres"
>>> adbc_driver_postgresql.dbapi.connect(uri)
<adbc_driver_manager.dbapi.Connection object at 0x0000029912EDED50>
>>> conn = _
>>> conn.close()
>>> with adbc_driver_postgresql.dbapi.connect(uri) as conn:
...  with conn.cursor() as cur:
...   cur.execute("SELECT 1")
...   print(cur.fetchall())
...
[(1,)]
>>> exit()
PS C:\Windows\system32> pip freeze
adbc-driver-manager==0.9.0
adbc-driver-postgresql==0.9.0
numpy==1.26.4
pandas==2.2.0
pyarrow==15.0.0
python-dateutil==2.8.2
pytz==2024.1
six==1.16.0
tzdata==2024.1
PS C:\Windows\system32> [Environment]::OSVersion

Platform ServicePack Version      VersionString
-------- ----------- -------      -------------
 Win32NT             10.0.18362.0 Microsoft Windows NT 10.0.18362.0
lidavidm commented 9 months ago

Are you possibly able to load WinDbg (it's in the Windows Store) and get a backtrace? 

windbgx -g python myscript.py

should pop up the GUI and run the program

programmingwithalex commented 9 months ago

Thanks for the suggestion. I ran the tool you suggested and indeed got a backtrace. It seems the error is related to the package itself and an integrity error on a single file.

(6ea4.33f0): Access violation - code c0000005 (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
*** WARNING: Unable to verify checksum for C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\adbc_driver_postgresql\libadbc_driver_postgresql.so
libadbc_driver_postgresql!PostgresqlDriverInit+0x13f172:
00007ffa`05e4b9c2 f00fc119        lock xadd dword ptr [rcx],ebx ds:feeefeee`feeeffa6=????????
lidavidm commented 9 months ago

Bleh. Any chance you have more of the traceback than that?

That's quite odd since that function implicated just fills out a struct, basically. I'm not great at reading disassembly, but I think I found the instruction implicated; the disassembly is extremely oddly long (hundreds of thousands of instructions), too.

lidavidm commented 9 months ago

Ok, the disassembly is a few hundred lines of what I would expect, a ret, and then 600,000 instructions of garbage, and it appears the crash you have is inside that garbage. Based on that, it seems like the problem is a few levels up; something is jumping to the wrong address for you. So if there's a full backtrace from windbg, that would be interesting.

programmingwithalex commented 9 months ago 

************* Preparing the environment for Debugger Extensions Gallery repositories **************
   ExtensionRepository : Implicit
   UseExperimentalFeatureForNugetShare : true
   AllowNugetExeUpdate : true
   AllowNugetMSCredentialProviderInstall : true
   AllowParallelInitializationOfLocalRepositories : true

   -- Configuring repositories
      ----> Repository : LocalInstalled, Enabled: true
      ----> Repository : UserExtensions, Enabled: true

>>>>>>>>>>>>> Preparing the environment for Debugger Extensions Gallery repositories completed, duration 0.000 seconds

************* Waiting for Debugger Extensions Gallery to Initialize **************

>>>>>>>>>>>>> Waiting for Debugger Extensions Gallery to Initialize completed, duration 0.297 seconds
   ----> Repository : UserExtensions, Enabled: true, Packages count: 0
   ----> Repository : LocalInstalled, Enabled: true, Packages count: 36

Microsoft (R) Windows Debugger Version 10.0.25921.1001 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.

CommandLine: C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\python.exe .\pyarrow_faster_reads.py

************* Path validation summary **************
Response                         Time (ms)     Location
Deferred                                       srv*
Symbol search path is: srv*
Executable search path is: 
ModLoad: 00007ff7`fb940000 00007ff7`fb95a000   python.exe
ModLoad: 00007ffa`937f0000 00007ffa`939e8000   ntdll.dll
ModLoad: 00007ffa`92fb0000 00007ffa`9306f000   C:\WINDOWS\System32\KERNEL32.DLL
ModLoad: 00007ffa`914d0000 00007ffa`917a2000   C:\WINDOWS\System32\KERNELBASE.dll
ModLoad: 00007ffa`913d0000 00007ffa`914d0000   C:\WINDOWS\System32\ucrtbase.dll
ModLoad: 00007ffa`3e480000 00007ffa`3ea53000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\python311.dll
ModLoad: 00007ffa`88e20000 00007ffa`88e3d000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\VCRUNTIME140.dll
ModLoad: 00007ffa`92770000 00007ffa`927db000   C:\WINDOWS\System32\WS2_32.dll
ModLoad: 00007ffa`80c10000 00007ffa`80c1a000   C:\WINDOWS\SYSTEM32\VERSION.dll
ModLoad: 00007ffa`92500000 00007ffa`92625000   C:\WINDOWS\System32\RPCRT4.dll
ModLoad: 00007ffa`92400000 00007ffa`9249e000   C:\WINDOWS\System32\msvcrt.dll
ModLoad: 00007ffa`910a0000 00007ffa`910c7000   C:\WINDOWS\System32\bcrypt.dll
ModLoad: 00007ffa`93520000 00007ffa`935ce000   C:\WINDOWS\System32\ADVAPI32.dll
ModLoad: 00007ffa`93440000 00007ffa`934dc000   C:\WINDOWS\System32\sechost.dll
ModLoad: 00007ffa`91340000 00007ffa`913c2000   C:\WINDOWS\System32\bcryptprimitives.dll
ModLoad: 0000022d`b67b0000 0000022d`b67bf000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\python3.DLL
ModLoad: 0000022d`b67b0000 0000022d`b67bf000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\python3.DLL
ModLoad: 00007ffa`24350000 00007ffa`243a2000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\adbc_driver_manager\_lib.cp311-win_amd64.pyd
ModLoad: 00007ff9`fe870000 00007ff9`fe90b000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\adbc_driver_manager.libs\msvcp140-a6ccfe46d550248f88945c0533c21bb8.dll
ModLoad: 00007ff9`fd980000 00007ff9`fdc28000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pyarrow\lib.cp311-win_amd64.pyd
ModLoad: 00007ff9`fe7d0000 00007ff9`fe86b000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pyarrow\MSVCP140.dll
ModLoad: 00007ff9`fcff0000 00007ff9`fd13f000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pyarrow\arrow_python.dll
ModLoad: 00007ff9`e0bc0000 00007ff9`e2952000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pyarrow\arrow.dll
ModLoad: 0000022d`b7070000 0000022d`b8e02000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pyarrow\arrow.dll
ModLoad: 00007ffa`910d0000 00007ffa`91226000   C:\WINDOWS\System32\CRYPT32.dll
ModLoad: 00007ffa`93120000 00007ffa`932c1000   C:\WINDOWS\System32\USER32.dll
ModLoad: 00007ffa`91800000 00007ffa`91822000   C:\WINDOWS\System32\win32u.dll
ModLoad: 00007ffa`6e650000 00007ffa`6eb29000   C:\WINDOWS\SYSTEM32\WININET.dll
ModLoad: 00007ffa`935d0000 00007ffa`935fb000   C:\WINDOWS\System32\GDI32.dll
ModLoad: 00007ffa`90ee0000 00007ffa`90fef000   C:\WINDOWS\System32\gdi32full.dll
ModLoad: 00007ffa`91230000 00007ffa`912cd000   C:\WINDOWS\System32\msvcp_win.dll
ModLoad: 00007ffa`8fe20000 00007ffa`8ff2c000   C:\WINDOWS\SYSTEM32\WINHTTP.dll
ModLoad: 00007ffa`91ba0000 00007ffa`922e4000   C:\WINDOWS\System32\SHELL32.dll
ModLoad: 00007ffa`76100000 00007ffa`7610c000   C:\WINDOWS\SYSTEM32\Secur32.dll
ModLoad: 00007ffa`92640000 00007ffa`9276a000   C:\WINDOWS\System32\ole32.dll
ModLoad: 00007ffa`91830000 00007ffa`91b85000   C:\WINDOWS\System32\combase.dll
ModLoad: 00007ffa`909b0000 00007ffa`909d7000   C:\WINDOWS\SYSTEM32\ncrypt.dll
ModLoad: 00007ffa`90de0000 00007ffa`90e0e000   C:\WINDOWS\SYSTEM32\USERENV.dll
ModLoad: 00007ffa`906f0000 00007ffa`906fc000   C:\WINDOWS\SYSTEM32\CRYPTBASE.DLL
ModLoad: 00007ffa`90da0000 00007ffa`90dd2000   C:\WINDOWS\SYSTEM32\SSPICLI.DLL
ModLoad: 00007ffa`90970000 00007ffa`909ab000   C:\WINDOWS\SYSTEM32\NTASN1.dll
ModLoad: 00007ffa`934e0000 00007ffa`93512000   C:\WINDOWS\System32\IMM32.DLL
ModLoad: 00007ffa`27f40000 00007ffa`27f7f000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\DLLs\_decimal.pyd
ModLoad: 00007ff9`fbb20000 00007ff9`fbdf8000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\numpy\core\_multiarray_umath.cp311-win_amd64.pyd
ModLoad: 00007ff9`d8940000 00007ff9`da9f6000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\numpy.libs\libopenblas64__v0.3.23-293-gc2f4bdbb-gcc_10_3_0-2bde3a66a51006b2b53eb373ff767a3f.dll
ModLoad: 00007ffa`4d940000 00007ffa`4d954000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\numpy\core\_multiarray_tests.cp311-win_amd64.pyd
ModLoad: 00007ffa`76030000 00007ffa`7604f000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\DLLs\_ctypes.pyd
ModLoad: 00007ffa`936e0000 00007ffa`937ad000   C:\WINDOWS\System32\OLEAUT32.dll
ModLoad: 00007ffa`7e0c0000 00007ffa`7e0cb000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\DLLs\libffi-8.dll
ModLoad: 00007ffa`4d6a0000 00007ffa`4d6bd000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\numpy\linalg\_umath_linalg.cp311-win_amd64.pyd
ModLoad: 00007ffa`48f10000 00007ffa`48f2e000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\numpy\fft\_pocketfft_internal.cp311-win_amd64.pyd
ModLoad: 00007ff9`fe4b0000 00007ff9`fe547000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\numpy\random\mtrand.cp311-win_amd64.pyd
ModLoad: 00007ffa`3fc80000 00007ffa`3fcac000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\numpy\random\bit_generator.cp311-win_amd64.pyd
ModLoad: 00007ffa`3c150000 00007ffa`3c17f000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\numpy\random\_common.cp311-win_amd64.pyd
ModLoad: 00007ffa`48a00000 00007ffa`48a11000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\DLLs\_hashlib.pyd
ModLoad: 00007ffa`3df80000 00007ffa`3e471000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\DLLs\libcrypto-3.dll
ModLoad: 00007ffa`270e0000 00007ffa`27123000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\numpy\random\_bounded_integers.cp311-win_amd64.pyd
ModLoad: 00007ffa`45720000 00007ffa`45736000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\numpy\random\_mt19937.cp311-win_amd64.pyd
ModLoad: 00007ffa`40ce0000 00007ffa`40cf5000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\numpy\random\_philox.cp311-win_amd64.pyd
ModLoad: 00007ffa`3dde0000 00007ffa`3ddf9000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\numpy\random\_pcg64.cp311-win_amd64.pyd
ModLoad: 00007ffa`60310000 00007ffa`60320000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\numpy\random\_sfc64.cp311-win_amd64.pyd
ModLoad: 00007ff9`fd8d0000 00007ff9`fd97f000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\numpy\random\_generator.cp311-win_amd64.pyd
ModLoad: 00007ffa`7a590000 00007ffa`7a5a5000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\DLLs\_socket.pyd
ModLoad: 00007ffa`90270000 00007ffa`902ab000   C:\WINDOWS\SYSTEM32\IPHLPAPI.DLL
ModLoad: 00007ffa`88820000 00007ffa`88829000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\DLLs\select.pyd
ModLoad: 00007ffa`8be00000 00007ffa`8be09000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\DLLs\_queue.pyd
ModLoad: 00007ffa`3b520000 00007ffa`3b544000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pyarrow\_hdfsio.cp311-win_amd64.pyd
ModLoad: 00007ff9`fcf50000 00007ff9`fcfec000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pyarrow\_dataset.cp311-win_amd64.pyd
ModLoad: 00007ff9`fb9d0000 00007ff9`fbb19000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pyarrow\arrow_dataset.dll
ModLoad: 00007ff9`fb890000 00007ff9`fb9ca000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pyarrow\arrow_acero.dll
ModLoad: 00007ff9`e0600000 00007ff9`e0bb4000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pyarrow\parquet.dll
ModLoad: 00007ffa`2c850000 00007ffa`2c87b000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pyarrow\_acero.cp311-win_amd64.pyd
ModLoad: 00007ff9`fcc40000 00007ff9`fccf1000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pyarrow\_compute.cp311-win_amd64.pyd
ModLoad: 00007ffa`24200000 00007ffa`24255000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pyarrow\_fs.cp311-win_amd64.pyd
ModLoad: 00007ffa`1fe40000 00007ffa`1fe7c000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pyarrow\_csv.cp311-win_amd64.pyd
ModLoad: 00007ffa`3c290000 00007ffa`3c2a4000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pyarrow\_json.cp311-win_amd64.pyd
ModLoad: 00007ffa`763c0000 00007ffa`763d5000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\DLLs\_bz2.pyd
ModLoad: 00007ffa`66e90000 00007ffa`66eb9000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\DLLs\_lzma.pyd
ModLoad: 00007ffa`5fda0000 00007ffa`5fdab000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\pandas_parser.cp311-win_amd64.pyd
ModLoad: 00007ffa`5ba50000 00007ffa`5ba5b000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\pandas_datetime.cp311-win_amd64.pyd
ModLoad: 00007ff9`f7db0000 00007ff9`f7ec7000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\interval.cp311-win_amd64.pyd
ModLoad: 00007ff9`f7be0000 00007ff9`f7da8000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\hashtable.cp311-win_amd64.pyd
ModLoad: 00007ffa`23420000 00007ffa`23450000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\missing.cp311-win_amd64.pyd
ModLoad: 00007ffa`1a5d0000 00007ffa`1a5fd000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\tslibs\dtypes.cp311-win_amd64.pyd
ModLoad: 00007ffa`2caf0000 00007ffa`2cb05000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\tslibs\ccalendar.cp311-win_amd64.pyd
ModLoad: 00007ffa`1fe20000 00007ffa`1fe40000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\tslibs\np_datetime.cp311-win_amd64.pyd
ModLoad: 00007ffa`19bf0000 00007ffa`19c2b000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\tslibs\conversion.cp311-win_amd64.pyd
ModLoad: 00007ffa`1ab20000 00007ffa`1ab31000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\tslibs\base.cp311-win_amd64.pyd
ModLoad: 00007ff9`f7b00000 00007ff9`f7bdd000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\tslibs\offsets.cp311-win_amd64.pyd
ModLoad: 00007ff9`f7a70000 00007ff9`f7af1000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\tslibs\timestamps.cp311-win_amd64.pyd
ModLoad: 00007ffa`18c40000 00007ffa`18c75000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\tslibs\nattype.cp311-win_amd64.pyd
ModLoad: 00007ff9`f79f0000 00007ff9`f7a68000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\tslibs\timedeltas.cp311-win_amd64.pyd
ModLoad: 00007ffa`18c00000 00007ffa`18c3a000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\tslibs\timezones.cp311-win_amd64.pyd
ModLoad: 00007ffa`59e20000 00007ffa`59e2d000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\DLLs\_zoneinfo.pyd
ModLoad: 0000022d`c7f40000 0000022d`c7f43000   C:\WINDOWS\SYSTEM32\tzres.dll
ModLoad: 0000022d`c7f40000 0000022d`c7f43000   C:\WINDOWS\SYSTEM32\tzres.dll
ModLoad: 00007ffa`11300000 00007ffa`11346000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\tslibs\fields.cp311-win_amd64.pyd
ModLoad: 00007ffa`186a0000 00007ffa`186e0000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\tslibs\tzconversion.cp311-win_amd64.pyd
ModLoad: 00007ffa`1aa40000 00007ffa`1aa54000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\properties.cp311-win_amd64.pyd
ModLoad: 00007ff9`fd870000 00007ff9`fd8c3000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\tslibs\parsing.cp311-win_amd64.pyd
ModLoad: 00007ff9`fcf00000 00007ff9`fcf4e000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\tslibs\strptime.cp311-win_amd64.pyd
ModLoad: 00007ff9`fb820000 00007ff9`fb887000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\tslibs\period.cp311-win_amd64.pyd
ModLoad: 00007ff9`fc880000 00007ff9`fc8b2000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\tslibs\vectorized.cp311-win_amd64.pyd
ModLoad: 00007ffa`590d0000 00007ffa`590e0000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\ops_dispatch.cp311-win_amd64.pyd
ModLoad: 00007ff9`f6de0000 00007ff9`f6fa2000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\algos.cp311-win_amd64.pyd
ModLoad: 00007ff9`f73e0000 00007ff9`f7485000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\lib.cp311-win_amd64.pyd
ModLoad: 00007ff9`f5050000 00007ff9`f5168000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\DLLs\unicodedata.pyd
ModLoad: 00007ff9`f73a0000 00007ff9`f73d5000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\ops.cp311-win_amd64.pyd
ModLoad: 00007ff9`fe7a0000 00007ff9`fe7cb000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\hashing.cp311-win_amd64.pyd
ModLoad: 00007ffa`1a5b0000 00007ffa`1a5cc000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\arrays.cp311-win_amd64.pyd
ModLoad: 00007ff9`f7350000 00007ff9`f7392000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\tslib.cp311-win_amd64.pyd
ModLoad: 00007ff9`f4f70000 00007ff9`f5041000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\sparse.cp311-win_amd64.pyd
ModLoad: 00007ff9`f7200000 00007ff9`f7250000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\internals.cp311-win_amd64.pyd
ModLoad: 00007ffa`19bd0000 00007ffa`19be1000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\indexing.cp311-win_amd64.pyd
ModLoad: 00007ff9`f4eb0000 00007ff9`f4f64000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\index.cp311-win_amd64.pyd
ModLoad: 00007ff9`f7070000 00007ff9`f70a3000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\writers.cp311-win_amd64.pyd
ModLoad: 00007ff9`f4d90000 00007ff9`f4ea6000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\join.cp311-win_amd64.pyd
ModLoad: 00007ff9`f6d80000 00007ff9`f6dd7000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\window\aggregations.cp311-win_amd64.pyd
ModLoad: 00007ffa`59030000 00007ffa`5903c000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\VCRUNTIME140_1.dll
ModLoad: 00007ff9`f4cf0000 00007ff9`f4d8b000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas.libs\msvcp140-fa0758dedafbbe194d3ee96e3dc2b9a3.dll
ModLoad: 00007ff9`fcc10000 00007ff9`fcc3b000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\window\indexers.cp311-win_amd64.pyd
ModLoad: 00007ff9`f6710000 00007ff9`f674f000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\reshape.cp311-win_amd64.pyd
ModLoad: 00007ff9`e03e0000 00007ff9`e05f4000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\groupby.cp311-win_amd64.pyd
ModLoad: 00007ffa`8c140000 00007ffa`8c147000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\DLLs\_uuid.pyd
ModLoad: 00007ffa`066b0000 00007ffa`066c1000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\json.cp311-win_amd64.pyd
ModLoad: 00007ff9`f6480000 00007ff9`f64e6000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\parsers.cp311-win_amd64.pyd
ModLoad: 00007ffa`00680000 00007ffa`00699000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\testing.cp311-win_amd64.pyd
ModLoad: 00007ff9`f51d0000 00007ff9`f520c000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pyarrow\_dataset_parquet.cp311-win_amd64.pyd
ModLoad: 00007ff9`f4c90000 00007ff9`f4ce9000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pyarrow\_parquet.cp311-win_amd64.pyd
ModLoad: 00007ff9`ffb20000 00007ff9`ffb34000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pyarrow\_dataset_parquet_encryption.cp311-win_amd64.pyd
ModLoad: 00007ff9`fcbd0000 00007ff9`fcbfa000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pyarrow\_parquet_encryption.cp311-win_amd64.pyd
ModLoad: 00007ffa`58780000 00007ffa`5878d000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pyarrow\arrow_python_parquet_encryption.dll
ModLoad: 00007ff9`fde80000 00007ff9`fde9b000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\adbc_driver_manager\_reader.cp311-win_amd64.pyd
ModLoad: 00007ffa`67420000 00007ffa`6744d000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\DLLs\_ssl.pyd
ModLoad: 00007ffa`4db00000 00007ffa`4dbc3000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\DLLs\libssl-3.dll
ModLoad: 00007ffa`90560000 00007ffa`905ca000   C:\WINDOWS\system32\mswsock.dll
ModLoad: 00007ff9`fde30000 00007ff9`fde50000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\DLLs\_elementtree.pyd
ModLoad: 00007ff9`f4c50000 00007ff9`f4c82000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\DLLs\pyexpat.pyd
ModLoad: 00007ff9`dfff0000 00007ff9`e03d8000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\lxml\etree.cp311-win_amd64.pyd
ModLoad: 00007ff9`fc060000 00007ff9`fc089000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\lxml\_elementpath.cp311-win_amd64.pyd
ModLoad: 00007ff9`dfd70000 00007ff9`dffee000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\PIL\_imaging.cp311-win_amd64.pyd
ModLoad: 00007ffa`902b0000 00007ffa`9037a000   C:\WINDOWS\SYSTEM32\DNSAPI.dll
ModLoad: 00007ffa`91b90000 00007ffa`91b98000   C:\WINDOWS\System32\NSI.dll
ModLoad: 00007ffa`87d00000 00007ffa`87d0a000   C:\Windows\System32\rasadhlp.dll
ModLoad: 00007ffa`7f420000 00007ffa`7f4a0000   C:\WINDOWS\System32\fwpuclnt.dll
ModLoad: 00007ff9`df8a0000 00007ff9`dfd62000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\adbc_driver_postgresql\libadbc_driver_postgresql.so
ModLoad: 00007ffa`93680000 00007ffa`936d6000   C:\WINDOWS\System32\WLDAP32.dll
ModLoad: 00007ffa`6f090000 00007ffa`6f099000   C:\WINDOWS\SYSTEM32\WSOCK32.dll
ModLoad: 00007ffa`8ee30000 00007ffa`8f5c2000   C:\WINDOWS\SYSTEM32\windows.storage.dll
ModLoad: 00007ffa`90780000 00007ffa`907b0000   C:\WINDOWS\System32\Wldp.dll
ModLoad: 00007ffa`92350000 00007ffa`923fd000   C:\WINDOWS\System32\SHCORE.dll
ModLoad: 00007ffa`93620000 00007ffa`93675000   C:\WINDOWS\System32\shlwapi.dll
ModLoad: 00007ffa`90830000 00007ffa`90848000   C:\WINDOWS\SYSTEM32\CRYPTSP.dll
ModLoad: 00007ffa`8fd90000 00007ffa`8fdc4000   C:\WINDOWS\system32\rsaenh.dll
(7108.5f2c): Access violation - code c0000005 (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
*** WARNING: Unable to verify checksum for C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\adbc_driver_postgresql\libadbc_driver_postgresql.so
libadbc_driver_postgresql!PostgresqlDriverInit+0x13f172:
00007ff9`dfb2b9c2 f00fc119        lock xadd dword ptr [rcx],ebx ds:feeefeee`feeeffa6=????????

I hope this helps. I only didn't include it initially because it didn't appear to have any clues as to what might be causing the issue.

CurtHagenlocher commented 9 months ago

Okay, this is a super-naive question but is a dynamic library generated by a toolchain producing an ".so" file even compatible with what the driver manager is expecting to load?

EDIT to minimize spamming: I thought checksum failure might be indicative of something, but if this weren't a valid DLL then we presumably wouldn't be referencing a symbol inside of it.

lidavidm commented 9 months ago

It's a DLL, just that I didn't bother to special case the extension in the build pipeline

lidavidm commented 9 months ago

Hmm, that gives everything except for the rest of the traceback which is what I'm curious about: how/why are we jumping into what appears to not even be valid code?

lidavidm commented 9 months ago

It does appear I should try with pyenv instead of Chocolatey python, though

lidavidm commented 9 months ago

Or apparently pyenv-win? Is that what you're using?

lidavidm commented 9 months ago

No dice, tried Python 3.11.7 via pyenv-win and it did not crash either.

reinieroost commented 9 months ago

I have the same issue on windows 11.

When running dbapi.connect(uri) it crashes with: Process finished with exit code -1073740940 (0xC0000374)

On ubuntu I haven't had this issue when connecting to the same database.

lidavidm commented 9 months ago

Thanks, good to know. I'll try to find time to replicate this again but I've not been having any luck, or else I'll try to figure out the backtrace handler (#1583) and see if we can provide a nightly wheel to test with

lidavidm commented 9 months ago

Oh, some more questions:

reinieroost commented 9 months ago

I'm not running it on a notebook but in pycharm.

I ran faulthander before the crash:

Windows fatal exception: code 0xc0000374

Thread 0x00008adc (most recent call first):
  File "C:\Program Files\JetBrains\PyCharm Community Edition 2023.3.2\plugins\python-ce\helpers\pydev\pydevd.py", line 217 in acquire
  File "C:\Program Files\JetBrains\PyCharm Community Edition 2023.3.2\plugins\python-ce\helpers\pydev\pydevd.py", line 225 in __enter__
  File "C:\Program Files\JetBrains\PyCharm Community Edition 2023.3.2\plugins\python-ce\helpers\pydev\pydevd.py", line 803 in process_internal_commands
  File "C:\Program Files\JetBrains\PyCharm Community Edition 2023.3.2\plugins\python-ce\helpers\pydev\pydevd.py", line 151 in _on_run
  File "C:\Program Files\JetBrains\PyCharm Community Edition 2023.3.2\plugins\python-ce\helpers\pydev\_pydevd_bundle\pydevd_comm.py", line 219 in run
  File "C:\Users\~\anaconda3\envs\~\lib\threading.py", line 1016 in _bootstrap_inner
  File "C:\Users\~\anaconda3\envs\~\lib\threading.py", line 973 in _bootstrap

Thread 0x00007710 (most recent call first):
  File "C:\Program Files\JetBrains\PyCharm Community Edition 2023.3.2\plugins\python-ce\helpers\pydev\_pydevd_bundle\pydevd_comm.py", line 293 in _on_run
  File "C:\Program Files\JetBrains\PyCharm Community Edition 2023.3.2\plugins\python-ce\helpers\pydev\_pydevd_bundle\pydevd_comm.py", line 219 in run
  File "C:\Users\~\anaconda3\envs\~\lib\threading.py", line 1016 in _bootstrap_inner
  File "C:\Users\~\anaconda3\envs\~\lib\threading.py", line 973 in _bootstrap

Thread 0x0000715c (most recent call first):
  File "C:\Users\~\anaconda3\envs\~\lib\threading.py", line 324 in wait
  File "C:\Users\~\anaconda3\envs\~\lib\queue.py", line 180 in get
  File "C:\Program Files\JetBrains\PyCharm Community Edition 2023.3.2\plugins\python-ce\helpers\pydev\_pydevd_bundle\pydevd_comm.py", line 368 in _on_run
  File "C:\Program Files\JetBrains\PyCharm Community Edition 2023.3.2\plugins\python-ce\helpers\pydev\_pydevd_bundle\pydevd_comm.py", line 219 in run
  File "C:\Users\~\anaconda3\envs\~\lib\threading.py", line 1016 in _bootstrap_inner
  File "C:\Users\~\anaconda3\envs\~\lib\threading.py", line 973 in _bootstrap

Current thread 0x00008194 (most recent call first):
  File "C:\Users\~\anaconda3\envs\~\lib\site-packages\adbc_driver_postgresql\__init__.py", line 40 in connect
  File "C:\Users\~\anaconda3\envs\~\lib\site-packages\adbc_driver_postgresql\dbapi.py", line 121 in connect
Process finished with exit code -1073740940 (0xC0000374)
lidavidm commented 9 months ago

Thanks. That gives the Python stack, but unfortunately not the C-level stack that could/might explain how we're getting so sidetracked...

reinieroost commented 9 months ago

Does this give you more insights?

Faulting application name: python.exe, version: 3.10.13150.1013, time stamp: 0x64ff158d
Faulting module name: ntdll.dll, version: 10.0.22621.3085, time stamp: 0xbced4b82
Exception code: 0xc0000374
Fault offset: 0x000000000010c7a9
Faulting process ID: 0x0x4910
Faulting application start time: 0x0x1DA70C7FF12C621
Faulting application path: C:\Users\~\anaconda3\envs\~\python.exe
Faulting module path: C:\Windows\SYSTEM32\ntdll.dll
Report ID: 4f42f518-c955-411f-a2be-e10a50518adc
Faulting package full name: 
Faulting package-relative application ID: 
Fault bucket , type 0
Event Name: APPCRASH
Response: Not available
Cab Id: 0

Problem signature:
P1: python.exe
P2: 3.10.13150.1013
P3: 64ff158d
P4: StackHash_c67d
P5: 10.0.22621.3085
P6: bced4b82
P7: c0000374
P8: PCH_40_FROM_ntdll+0x000000000009FEC4
P9: 
P10:
LaurensBosscher commented 8 months ago

I can confirm this is also an issue on:

Also an AWS RDS database. Connecting using username+password.

lidavidm commented 8 months ago

Hmm.

Is everyone here trying RDS? Does it crash with a local or other non-AWS Postgres instance?

reinieroost commented 8 months ago

I'm not using AWS

mkleinbort-ic commented 8 months ago

I have the same issue, on an Azure hosted db.

I'm running locally and it crashes the process/notebook when ran either in jupyter, IPython, or just Python.

I was suspecting it'd be the antivirus blocking it - but no idea.

On Windows 11, Python 3.10.9

lidavidm commented 8 months ago

I have built some wheels here with debug info (scroll all the way down): https://github.com/apache/arrow-adbc/actions/runs/8331028521?pr=1599

Would it be possible for anyone to give that a shot, preferably under windbgx, and see if they can get a full stack trace?

lidavidm commented 8 months ago

I'm still unable to reproduce this. If anyone is able to produce a full stack trace (not just the last frame), and/or a reproducer, that would be much appreciated.

As far as I can see, the original crash happens before any connection attempt is made (it happens when trying to load the client library), so it doesn't seem to be related to RDS/Azure/etc.

lidavidm commented 8 months ago

Or even knowing how things were installed would help (e.g. I just saw Anaconda above, but we don't ship Conda packages on Windows!)

lidavidm commented 8 months ago

Hmm, no dice either with Anaconda Python + ADBC from wheels

lidavidm commented 8 months ago

Or actually, please use the updated wheels here (scroll down to "python3.12-windows" etc): https://github.com/apache/arrow-adbc/actions/runs/8347053159?pr=1599

Also, please extract "adbc_driver_postgresql.pdb" from the ZIP file and place it next to your script/in the PWD so that the backtrace handler can find it. If all goes well, a crash should print something like this, which will let us debug the issue:

#5    Object "", at 00007FF860185F99, in PyBytes_Repeat
#4    Object "", at 00007FF899441D61, in AdbcDriver::operator=
#3    Object "", at 00007FF89946D5C6, in AdbcConnectionSetOption
#2    Source "C:\Users\User\arrow-adbc\c\driver\sqlite\sqlite.c", line 254, in SqliteConnectionSetOption [00007FF84A115712]
#1    Source "C:\Users\User\arrow-adbc\c\driver\sqlite\sqlite.c", line 145, in ExecuteQuery [00007FF84A115215]
#0    Object "", at 00007FF860EAFF3E, in memset