sfackler / rust-postgres

Native PostgreSQL driver for the Rust programming language
Apache License 2.0
3.44k stars 436 forks source link

Using multiple `query_raw`, rust-postgres will get stuck #958

Closed cloverzero closed 1 year ago

cloverzero commented 1 year ago

Code:

let mut node_conn = self.config.connect(NoTls)?;
let mut tag_conn = self.config.connect(NoTls)?;

println!("1");
let mut node_iter = node_conn.query_raw(
    "SELECT e.id, e.latitude, e.longitude, e.changeset_id, e.timestamp, e.\"version\", \
        u.id as user_id, u.display_name \
    FROM current_nodes e \
    INNER JOIN changesets c ON e.changeset_id = c.id \
    INNER JOIN users u ON c.user_id = u.id \
    WHERE visible = true ORDER BY id",
    Vec::<String>::with_capacity(0),
)?;

println!("2");
let mut tag_iter = tag_conn.query_raw(
    "SELECT node_id, k, v FROM current_node_tags ORDER BY node_id",
    Vec::<String>::with_capacity(0),
)?;

println!("3");

Output:

1
2

You can't see 3 in the output, and you'll receive an error message from the database saying "write: connection reset by peer".

sfackler commented 1 year ago

I cannot reproduce this issue:

#[test]
fn two_connections() {
    let mut node_conn = Client::connect("host=localhost port=5433 user=postgres", NoTls).unwrap();
    let mut tag_conn = Client::connect("host=localhost port=5433 user=postgres", NoTls).unwrap();

    node_conn.batch_execute("
        CREATE TEMPORARY TABLE current_nodes (id BIGINT, latitude DOUBLE PRECISION, longitude DOUBLE PRECISION);

        INSERT INTO current_nodes (id, latitude, longitude)
        SELECT n, n, n
        FROM generate_series(1, 10000) AS n;
    ").unwrap();

    tag_conn
        .batch_execute(
            "
        CREATE TEMPORARY TABLE current_node_tags (node_id BIGINT, k TEXT, v TEXT);

        INSERT INTO current_node_tags (node_id, k, v)
        SELECT n, n, n
        FROM generate_series(1, 10000) AS n;
    ",
        )
        .unwrap();

    println!("1");
    let mut node_iter = node_conn
        .query_raw(
            "SELECT e.id, e.latitude, e.longitude FROM current_nodes e",
            Vec::<String>::new(),
        )
        .unwrap();

    println!("2");

    let mut tag_iter = tag_conn
        .query_raw(
            "SELECT node_id, k, v FROM current_node_tags ORDER BY node_id",
            Vec::<String>::new(),
        )
        .unwrap();

    println!("3");
}
cargo test two_connections -- --nocapture                                                                                                                    6.4s  Mon Oct 17 07:42:02 2022
   Compiling postgres v0.19.4 (/Users/sfackler/code/rust-postgres/postgres)
warning: unused variable: `node_iter`
   --> postgres/src/test.rs:538:13
    |
538 |     let mut node_iter = node_conn
    |             ^^^^^^^^^ help: if this is intentional, prefix it with an underscore: `_node_iter`
    |
    = note: `#[warn(unused_variables)]` on by default

warning: unused variable: `tag_iter`
   --> postgres/src/test.rs:547:13
    |
547 |     let mut tag_iter = tag_conn
    |             ^^^^^^^^ help: if this is intentional, prefix it with an underscore: `_tag_iter`

warning: variable does not need to be mutable
   --> postgres/src/test.rs:538:9
    |
538 |     let mut node_iter = node_conn
    |         ----^^^^^^^^^
    |         |
    |         help: remove this `mut`
    |
    = note: `#[warn(unused_mut)]` on by default

warning: variable does not need to be mutable
   --> postgres/src/test.rs:547:9
    |
547 |     let mut tag_iter = tag_conn
    |         ----^^^^^^^^
    |         |
    |         help: remove this `mut`

warning: `postgres` (lib test) generated 4 warnings
    Finished test [unoptimized + debuginfo] target(s) in 0.62s
     Running unittests src/lib.rs (/Users/sfackler/code/rust-postgres/target/debug/deps/postgres-2eba88937e252ce8)

running 1 test
1
2
3
test test::two_connections ... ok
cloverzero commented 1 year ago

I think it's due to the amount of data. Both of my tables have over 400,000 rows. If I add limit 100 to the SQL, it works.

sfackler commented 1 year ago

I'd recommend creating portals and paging through them rather than trying to stream through the entire response in two separate connections.

cloverzero commented 1 year ago

I just wanner know why the second connection reset. Do these two connections interfere when using streams to get data?

sfackler commented 1 year ago

It would help to provide a fully self contained test case that reproduces the issue.