Issue with shareconnectionwrapper

[fromm1990]

H I'm having an issue while using psycopg2 connections with the shareconnectionwrapper function as shown here (I'm not using Jython though).

More specifically, I get the error: "cannot pickle 'psycopg2.extensions.connection' object".

My environment is as follows:

• Python=3.9.12
• psycopg2=2.9.3
• pygrametl=2.7

Code to reproduce:

import psycopg2
from pygrametl import parallel, ConnectionWrapper

dsn = "host= dbname= user= password="
connection = psycopg2.connect(dsn) 

cw = ConnectionWrapper(connection)
scw = parallel.shareconnectionwrapper(targetconnection=cw) #<- This line should produce the error

Thanks in advance.

[skejserjensen]

What operating system do you use?

From this Stack Overflow question it seems psycopg2.extensions.connection can be pickled on FreeBSD, Linux, and macOS but not on Windows: https://stackoverflow.com/questions/52920954/platform-dependent-error-with-psycopg2-only-win10-says-cant-pickle-psycopg2

Also, the code to reproduce executed without any problems on:

• Linux 5.17.9
• Python 3.10.4
• psycopg2 2.9.3
• pygrametl 2.7

[fromm1990]

I use windows 10, so that may very well be the reason. I will try it out using WSL2 and report back my result.

[fromm1990]

I can now confirm that everything works as should within a WSL2 environment. Therefore, running on Windows seem to be the root cause of this issue and is very likely due to what is explained in the link shared by @skejserjensen.

More information on thread and process safety in psycopg2 can be found here

For what it is worth, I also tested with psycopg version 3.0.14 which again did not work on Windows (different exception though TypeError: no default reduce due to non-trivial cinit) but worked on WSL2. This again seem to be caused by the difference between how Linux and Windows handles multiprocessing, more about it here.

I'm not sure if there exists a work around for this platform specific issue and will therefore let the issue remain open.

Thanks

[mFeigeInvia]

Same on OSX with ARM Proccessors, tested on: python 3.9.15 OSX 12.6 pygrametl 2.7 psycopg2 2.9.3

Traceback (most recent call last): File "/opt/homebrew/Cellar/python@3.9/3.9.15/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 47, in _launch reduction.dump(process_obj, fp) File "/opt/homebrew/Cellar/python@3.9/3.9.15/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/reduction.py", line 60, in dump ForkingPickler(file, protocol).dump(obj) TypeError: cannot pickle 'psycopg2.extensions.connection' object

[mFeigeInvia]

ok, https://bugs.python.org/issue33725#msg365249 is the main issue on OSX

    if platform.system() == 'Darwin':
        multiprocessing.set_start_method('fork')

not good, but works