move-coop / parsons

A python library of connectors for the progressive community.
Other
255 stars 125 forks source link

DRAFT - Migrate from psycopg2 to psycopg3 #987

Open austinweisgrau opened 5 months ago

austinweisgrau commented 5 months ago

psycopg3 is mostly built as a drop-in upgrade from psycopg2. Differences are detailed here: https://www.psycopg.org/psycopg3/docs/basic/from_pg2.html

A few highlights:

One major DX improvement is that Ctrl-C can now cancel a running query. With psycopg2, the python process can be killed but a running query usually could not be killed (without knowing the PID and running pg_cancel_backend(pid) from a separate connection), to the immediate regret of all we who sometimes accidentally execute an unfortunately large query.

See https://www.psycopg.org/psycopg3/docs/advanced/async.html#interrupting-async-operations

A few parameterization patterns that worked with psycopg2 no longer work with psycopg3. They are detailed on the migration guide linked above, but these in particular should be highlighted:

** You can no longer use IN %s with a tuple

This would work with psycopg2 but will not work with psycopg3: conn.execute("SELECT * FROM foo WHERE id IN %s", parameters=[(10,20,30)])

Instead, this format is advised: conn.execute("SELECT * FROM foo WHERE id = ANY(%s)", parameters=[[10,20,30]])

* You can no longer use IS %s This would work with psycopg2 but will not work with psycopg3: `conn.execute("SELECT FROM foo WHERE field IS %s", [None])`

Instead, IS NOT DISTINCT FROM %s should be used conn.execute("SELECT * FROM foo WHERE field IS NOT DISTINCT FROM %s", [None])

copy methods are refactored and consolidated in psycopg3. See the migration guide for details.

shaunagm commented 5 months ago

Looks like there are some file conflicts? Also, I thought we'd fixed it so the tests would get run against PRs to major-release?

If we can get this to run the tests, and the tests pass, we can merge. Though I will do so warily regardless since psycopg has historically caused a huge amount of installation errors. Actually it'd be great if we could test this against an M1 mac, @austinweisgrau do you happen to have one you could test manually?

austinweisgrau commented 5 months ago

I believe the major release branch needs to be brought up to date and that should simplify the diffs considerably

austinweisgrau commented 5 months ago

Ahh it needs more configuration anyways to work propertly, making this a DRAFT for now