search

Segfault-Inc / Multicorn

Data Access Library
https://multicorn.org/
PostgreSQL License
700 stars 145 forks source link

Huge memory usage from a SELECT .. LIMIT 1 #74

Open strk opened 10 years ago

strk commented 10 years ago

Defining a FDW table against a big remote table (~8 million rows) makes selecting a single row from the foreign table take up all available system memory

rdunklau commented 10 years ago

What FDW are you using ? If the execute method uses a generator, you shouldn't see such a high memory usage.

strk commented 10 years ago

sqlalchemy

rdunklau commented 10 years ago

What driver are you using ? If you are using anything else than psycopg2, could you please confirm that the fix does / doesn't work ?

strk commented 10 years ago

Both the remote and the local servers are PostgreSQL, that means I'm using psycopg2, I guess ? I don't have offending setup anymore for testing

rdunklau commented 10 years ago

So if you're using psycopg2, the bug should be solved for you.

I'll keep this issue opened till I find a reliable way, if any, to stream results from other drivers.