search

psycopg / psycopg2

PostgreSQL database adapter for the Python programming language
https://www.psycopg.org/
Other
3.35k stars 506 forks source link

django threading issue #1284

Closed sinteur closed 3 years ago

sinteur commented 3 years ago

django/db/backends/base/base.py has a validate_thread_sharing that a django developer who's using psycopg2 cannot easily change the behavior for.

Bit hard to reproduce, but here's how I found out: install https://github.com/NLnetLabs/Internet.nl and modify the pip requirements to require a recent django. Since that project uses both a main python process and a few celery based threads, it's easy to run into an exception. Since that project uses postgresql which has no problems at all with the way the project uses threads, it would be nice to have a way to let psycopg2 take care of this. At the same time, it would also be nice if django was a bit more aware of what each database engine accepts, so I'll be mentioning this in their community as well. Feedback much appreciated

dvarrazzo commented 3 years ago

This is very unclear to me, I'm sorry. I don't understand what is the error and what either psycopg or django can do.

I can try and investigate the issue if you provide a dockerfile to reproduce it.

sinteur commented 3 years ago

Thanks for the fast reply - I'll see if I can easily get you a docker file... in the mean time, here's a bit more info and a direction for the solution:

this function: says in the comments:

Validate that the connection isn't accessed by another thread than the one which originally created it, unless the connection was explicitly authorized to be shared between threads (via the inc_thread_sharing() method). Raise an exception if the validation fails

the internet.nl code is running in multiple threads. We pass a DB object to a task and that other task runs in a different thread, changing that to passing an ID instead would be one possible solution, but quite a bit of work

So another solution would be to have something/someone call "DatabaseWrapper.inc_thread_sharing()" - I'm not sure what the best way would be to expose that to a psycopg2 user, and I'm open to suggestions.

sinteur commented 3 years ago

(and perhaps also informative, this is what the error would look like:

https://john.sinteur.com/BB/DatabaseError.html )

dvarrazzo commented 3 years ago

That is a Django error, for an entirely Django-related concept. There is nothing psycopg2 can do about it. You should hear from the Django people about it.