coleifer / peewee

a small, expressive orm -- supports postgresql, mysql, sqlite and cockroachdb
http://docs.peewee-orm.com/
MIT License
11.17k stars 1.37k forks source link

playhouse.postgres_ext.ServerSide doesn't create server-side cursor #2899

Closed hordiienko-v closed 5 months ago

hordiienko-v commented 5 months ago

Hi. In peewee v3.16.0 and higher, ServerSide fetches all data at once.

Code snippet for testing:

from peewee import TextField
from playhouse.postgres_ext import PostgresqlExtDatabase, ServerSide

db = PostgresqlExtDatabase(...)

class Test(db.Model):
    class Meta:
        table_name = "test"

    value = TextField()

if __name__ == "__main__":
    for _ in ServerSide(Test.select(), array_size=2000):
        break

PostgresQL logs when run code with peewee v3.15.4:

2024-05-22 11:27:00.850 EEST [40349] test_user@test_db LOG:  statement: BEGIN
2024-05-22 11:27:00.851 EEST [40349] test_user@test_db LOG:  statement: DECLARE "0fb26f06-1815-11ef-9b84-c35986740660" CURSOR WITHOUT HOLD FOR SELECT "t1"."id", "t1"."value" FROM "test" AS "t1"
2024-05-22 11:27:00.852 EEST [40349] test_user@test_db LOG:  statement: FETCH FORWARD 2000 FROM "0fb26f06-1815-11ef-9b84-c35986740660"
2024-05-22 11:27:00.853 EEST [40349] test_user@test_db LOG:  statement: ROLLBACK

With peewee v3.16.0:

2024-05-22 11:27:51.585 EEST [40452] test_user@test_db LOG:  statement: BEGIN
2024-05-22 11:27:51.586 EEST [40452] test_user@test_db LOG:  statement: SELECT "t1"."id", "t1"."value" FROM "test" AS "t1"
2024-05-22 11:27:54.407 EEST [40452] test_user@test_db LOG:  statement: ROLLBACK

I assume that problem may be because PostgresqlExtDatabase.execute doesn't pass named_cursor to Database.execute_sql and to PostgresqlExtDatabase.execute respectively. But after I manually changed the code in order to pass named_cursor, I get peewee.ProgrammingError: can't use a named cursor outside of transactions.

Is it a bug or did I miss something obvious?

coleifer commented 5 months ago

No this was a bug, thank you for the report. Should be fixed now in master.

hordiienko-v commented 5 months ago

Yup, now it works as expected. Thank you!