copying large tables subsequently causes query to fail

udnaan commented 6 months ago

What happens?

Copying/caching large tables subsequently causes query to fail and random erros to pop up such as:

duckdb.duckdb.IOException: IO Error: Unable to read binary COPY data from Postgres: no COPY in progress
duckdb.duckdb.IOException: IO Error: Unable to read binary COPY data from Postgres: no COPY in progress message contents do not agree with length in message type "d"
duckdb.duckdb.IOException: IO Error: Postgres scanner - out of buffer in ReadString
duckdb.duckdb.IOException: IO Error: Unable to read binary COPY data from Postgres: lost synchronization with server: got message type "

For example, a table with 100k rows (<2k per row) followed by a table with another 100k or more. Seems to be related to threading. Setting threads to 1 fixes it.

Not particulaly bothered by it since we have a workaround. Just wanted the duckdb team to know since someone else might run into it.

To Reproduce

        self.duckdb.execute(f"""
            ATTACH '{dbstr}' AS pgdb (TYPE postgres);
            CREATE TABLE table1 AS FROM pgdb.table1;
            CREATE TABLE table2 AS FROM pgdb.table2;
        """)

Work around:

               SET threads TO 1;

OS:

debian

PostgreSQL Version:

all

DuckDB Version:

0.9.2

DuckDB Client:

Python

Full Name:

Adnan Yunus

Affiliation:

OpusAI

Have you tried this on the latest `main` branch?

[X] I agree

Have you tried the steps to reproduce? Do they include all relevant data and configuration? Does the issue you report still appear there?

[X] I agree

Mytherin commented 6 months ago

Thanks for the report - this sounds like it might be related to #149. Could you try this on the latest nightly to check if this still reproduces:

force install postgres_scanner from 'http://nightly-extensions.duckdb.org';

davetapley commented 6 months ago

@Mytherin nightly appears to fix intermittent no COPY in progress for me! 🙏🏻

udnaan commented 6 months ago

I'm using python so I tried pip install duckdb==0.9.3.dev2258 and that seems to work. 0.9.2 is the version where I'm still getting this error

Mytherin commented 5 months ago

Do you still get the error after installing a new version of the extension (force install postgres_scanner)?

Mytherin commented 3 months ago

Likely fixed already - feel free to re-open if you encounter this error in a newer release

duckdb / postgres_scanner