search

chtd / psycopg2cffi

Port to cffi with some speed improvements
Other
177 stars 43 forks source link

Single row mode support #17

Closed lopuhin closed 10 years ago

lopuhin commented 10 years ago

libpq has single row mode support since 9.2 - this can save memory when loading large datasets - http://www.postgresql.org/docs/9.2/static/libpq-single-row-mode.html

Maybe I will try to first see it it really reduces memory usage. Is is currently implemented in psycopg2? @dvarrazzo

dvarrazzo commented 10 years ago

No, it's not. I don't think it would be very easy to add. Of course you may create a branch in the psycopg2-cffi implementation to play with it: if it was worth we can reimplement it in C.

I think the case of large datasets is covered well enough by server-side cursors, where we have already wrestled all the bugs due to bookkeeping paginated records etc. I wouldn't know in "single row mode" how to implement scroll() if not loading the entire dataset in the cursor anyway, whereas this is pretty much solved with server side cursors. And so on for all the use cases except the most simple.

lopuhin commented 10 years ago

Aha, I tried to use named cursors under psycopg2cffi and they ate 3x more memory, but just checked with psycopg2 - and they work just fine and do what I wanted. Thank you for the hint! So I will try to see why server side cursors are broken in psycopg2cffi

lopuhin commented 10 years ago

18 is about this bug