search

dbcli / pgspecial

Python implementation of postgres meta commands (backslash commands)
BSD 3-Clause "New" or "Revised" License
74 stars 54 forks source link

Add a few comments about porting to Psycopg 3 #122

Closed dvarrazzo closed 2 years ago

dvarrazzo commented 2 years ago

Description

As per https://github.com/dbcli/pgcli/pull/1318, dropping a few comments about how certain idioms changed from Psycopg 2 to 3. I'll be available if you need more help porting.

See https://github.com/dbcli/pgcli/issues/1085#issuecomment-1047289518 for some background about the problems of this project with psycopg 2.

Let me know if I can help further :wave:

Checklist

j-bennet commented 2 years ago

👍 🚀 💝

dvarrazzo commented 2 years ago

How concrete are plans to move this project to psycopg 3? :slightly_smiling_face:

amjith commented 2 years ago

@dvarrazzo Thank you for the contribution.

I see that psycopg3 had the first release sometime in Oct 2021. Congrats on the release.

We would absolutely love to move the project to psycopg3. I'd like to think our test suite has enough coverage to discover any corner cases. I'll work with @j-bennet (project lead) to get a branch going with psycopg3 and start testing locally.

You should see some activity in the next few weeks. Cheers!

dvarrazzo commented 2 years ago

Hello @amjith

I hope that the changes in psycopg 3 will be helpful with projects such as dbcli. The hope is that it will be easier to install and that there won't be an impossible situation like in psycopg 2, with the confusion arising from two distribution packages installing the same module.

If you need help with porting just shout! Cheers!

dvarrazzo commented 2 years ago

@amjith one thing I have seen is that pgcli will depend on is this issue to be fixed, to make Ctrl-C work as before; at least for the sync path, as your project doesn't use async.

This test works ok on Linux/macOS but I cannot get it to work on Windows: I'm trying to send a Ctrl-C to the process but it doesn't receive it.

If any of you dbcli devs use windows and are able to fix it it would be great! Thank you!

j-bennet commented 2 years ago

Hi @dvarrazzo

I have a question about porting this bit of code:

https://github.com/dbcli/pgspecial/blob/478c74dc35cb6e007ba431a9588c9bbd363f5942/pgspecial/iocommands.py#L125-L131

This was added with support for \\copy command in pgspecial, I think to support Ctrl + C behavior, but I'm not sure (@catherinedevlin or @drocco007, can you advise?):

So the question is, does psycopg has the equivalent of set_wait_callback/get_wait_callback, and do you think we need it here?

dvarrazzo commented 2 years ago

Hello @j-bennet

get/set callbacks are used in psycopg2 in order to support greenlet libraries (eventlet, gevent): in psycopg2 the blocking part is in C, so it can't be monkeypatched; the callback allows to "externalise" I/O wait back to Python.

In psycopg 3, the wait code is in Python, so greenlet libraries can simply monkeypatch the Python code as they do with any pure Python library; only non-blocking code is is C.

I could see that, in its normal behaviour, pgcli installs a wait callback so that ctrl-c works (taken from this recipe). In psycopg 3 it shouldn't be needed: ctrl-c is received ok... almost ok: the connection is left in unrecoverable state after a ctrl-c, but I consider this a psycopg 3 bug, to be fixed in https://github.com/psycopg/psycopg/issues/231 So there is no provision to reintroduce a wait callback.

....ahh, I think I know what is the above! I think that the psycopg2 copy methods don't work in green mode! So the _paused_thread() is used to re-establish temporarily the non-green mode to allow copy to go. In that instance, ctrl-c should work ok anyway, because psycopg2 internally calls file.read()/file.write() and there is a possibility to process the signal too.

If that's the case then there should be no need for it: the new cursor.copy() interface requires to have the read/write loop in Python so ctrl-c works as expected.

As far as I can see, none of the code interacting with the wait callback needs to be duplicated for Psycopg 3. Ctrl-C should work normally, but at the moment you should use the fix-231 branch to allow the connection to recover after Ctrl-C.

j-bennet commented 2 years ago

@dvarrazzo Dummy question: what's green mode? :)

This helps a lot, and it's good to know that I can completely throw away the _paused_thread wrapper. Thank you!

dvarrazzo commented 2 years ago

Ha ha, not dummy, no :)

Some details in Support for coroutine libraries in the docs. "Green" is referred to the green threads, the concurrency primitives gevent and eventlet are based on. "Green mode" is when set_wait_callback() is called with non-None argument: in that mode, psycopg2 uses non-blocking calls and defers to the callback when an I/O operation would block, giving Python the chance to run for a moment. It is used in the libraries above to schedule a task switch; pgcli only uses it to handle a ctrl-c.