trinodb / trino-python-client

Python client for Trino
Apache License 2.0
335 stars 168 forks source link

Empty results on large queries when using fetchall() #44

Open gh-dzuluaga opened 4 years ago

gh-dzuluaga commented 4 years ago

Hello there.

I've found that these two examples will not always return the same thing on large queries:

  1. fetchall

    cur.execute(large_query)
    rows = cur.fetchall()
    print(rows)
  2. iterator

    for r in cur.execute(large_query):
    print(r)

The fetchall approach ends up returning an empty list every time I use it when running large queries, but the iterator approach prints the result successfully

Does anyone know why this may be happening?

I don't know how I should supply a working example to debug this problem, as my queries all contain domain logic of my company. Nevertheless, I can say that I'm connecting around 8 Postgres databases with 100k rows per table and equal schemas, and I union the tables from all databases together with UNION views, like this:

CREATE OR REPLACE VIEW hive.default."table_unioned_view" AS 
SELECT 'database1' AS database, "other_column" FROM database1.public.table
UNION ALL
SELECT 'database2' AS database, "other_column" FROM database2.public.table
UNION ALL
SELECT 'database3' AS database, "other_column" FROM database3.public.table;

And the big query that I'm running uses these type of views

findepi commented 4 years ago

Would you like to try to come up with a test case reproducing the problem? See https://github.com/prestosql/presto-python-client/blob/a7438556213bcf954f8a7a5063c7823bbb177edb/integration_tests/test_dbapi.py#L59-L68 for some example test using dockerized Presto