double free or corruption (out)

balazon commented 10 months ago

OS: ubuntu 22.04
Psycopg version: 2.9.9
Python version: 3.10
PostgreSQL version: don't know, doesn't matter probably
pip version: 23.3.1

Hi, I created a brand new environment using conda (to be more precise it was mamba with miniforge) with python 3.10, installed psycopg==2.9.9, and whenever I tried connecting to a server, I had a crash with this output:

double free or corruption (out) Aborted (core dumped)

This was especially puzzling, because I could successfully connect using pgadmin to the same server. I tried downgrading to older versions of psycopg2, but that didn't help. An older issue gave me a hint here: https://github.com/psycopg/psycopg2/issues/899 which mentioned openssl, so I checked mine in mamba (mamba list | grep openssl): openssl 3.2.0 hd590300_1 conda-forge

I searched for openssl in conda (mamba search -f openssl): openssl 3.1.3 hd590300_0 conda-forge
openssl 3.1.4 hd590300_0 conda-forge
openssl 3.2.0 hd590300_0 conda-forge
openssl 3.2.0 hd590300_1 conda-forge

So I installed openssl 3.1.4 instead: mamba install openssl=3.1.4

And now it seems to work. I don't know if this is an issue of psycopg2, openssl, or conda, or if it's a matter of psycopg2 supporting a newer version of openssl, so I'm sorry if this issue does not belong here.

dvarrazzo commented 10 months ago

we have no relation with conda/mamba/miniforge. If the issue is not reproducible with pip, it doesn't belong to us.

balazon commented 10 months ago

Just because I reproduced it with mamba does not mean it's not reproducible with pip though. I have no sudo on my work computer, so I can't just mess around with installing openssl versions there. Mamba is handy in that case, because it's easy to install stuff like openssl. Anyway, thanks for reading the issue

balazon commented 10 months ago

FYI: Someone else have encountered this also https://github.com/Homebrew/homebrew-core/issues/155651#issuecomment-1828065770 The issue is not on your side, but I have no idea what homebrew or libpq is :D

dvarrazzo commented 10 months ago

I have looked at the conversation in the postgres developers ML: it seems that some work happened towards the end of November and resulted in some code change, which, in the v16 maintenance branch, is in this commit, which has been cherry-picked to all the supported branches.

There hasn't been a new release after that commit. This means that the previous batch of PostgreSQL releases (the libpq is the postgres client library and it gets released together with a postgres release) are not compatible with OpenSSL 3.2.0. These releases are 11.22, 12.17, 13.13, 14.10, 15.5, 16.1, released on the 2023-11-06.

About psycopg:

using the psycopg2 package, compiled from source, will use the system libpq, which in turn will use the system openssl. Until the release of 16.2, or sibling releases, this libpq version is not compatible with openssl 3.2 and a system which packages and installs these two libraries together must be considered - I assume - broken.
using the psycopg2-binary package will install a private version of libpq and openssl. The openssl installed is of the 1.1.1 family, so I don't think it is affected by this issue.
I am not responsible for what conda releases under the psycopg2 name, they do it without our explicit consent. I assume don't install openssl 3.2.0 until you have libpq 16.2 on your system, but you are fundamentally used unauthorized and untested software.

balazon commented 10 months ago

I haven't actually installed psycopg2 from the conda repositories, but with pip. I tried just now installing with conda, and I think it installs the binary version.

ls $HOME/miniforge3/envs/ptest/lib/python3.10/site-packages/psycopg2
errorcodes.py  errors.py  extensions.py  extras.py  __init__.py  _ipaddress.py  _json.py  pool.py  _psycopg.cpython-310-x86_64-linux-gnu.so  __pycache__  _range.py  sql.py  tz.py

The ldd output for the .so file points to these: $HOME/miniforge3/envs/ptest/lib/libssl.so.3 $HOME/miniforge3/envs/ptest/lib/libpq.so.5.16 [...] # and others This version does not crash with the double free or corruption message either, like you said. Same goes for psycopg (version 3): the binary (conda installed) version works, pip one crashes with openssl 3.2.0

Until now I didn't even know there is a psycopg2-binary (or psycopg[binary]) package. I guess I'm gonna use that instead, it works out of the box without openssl issues

titarch commented 10 months ago

I faced this issue with pip, no conda environments. Simply with psycopg2 installed and my system openssl library being on version 3.2.0. Shouldn't this issue be re-opened @dvarrazzo ?

dvarrazzo commented 10 months ago

@titarch the comment above is valid for you too. OpenSSL 3.2.0 is not compatible with libpq 16.1 and sibling releases, it's not a problem with psycopg: if you install a libpq-openssl compatible pair the psycopg installed on your system will work again.

You can work around the problem:

Installing OpenSSL < 3.2.
Installing libpq >= 16.2 when it will be released.
Installing psycopg2-binary which ships with its own libraries.

psycopg / psycopg2

double free or corruption (out) #1658