cornucopia-rs / cornucopia

Generate type-checked Rust from your PostgreSQL.
Other
835 stars 38 forks source link

Add doc section on pipelining #156

Closed LouisGariepy closed 1 year ago

LouisGariepy commented 2 years ago

Concept described here https://docs.rs/tokio-postgres/latest/tokio_postgres/#pipelining.

Transcript of the Discord thread that sparked this (thanks to user korin):

So maybe more context. I have 2 tables. And I want to copy one to another. But I do not know how to do it in SQL I am doing it through Rust code.

So first what I am create is a stream of that table (it's 800GB so I can't use all()).

   let mut stmts = select_all_transfer_cache_old();
   let mut stream_of_data = stmts.bind(&client).iter().await?.boxed();

and now I tried with stream_of_data.try_for_each_conncurent(1, |input_data| async move { insert_function().bind(&client, ...) } ).await?; but I want to move input_data only and not client.

Ok I have it:

stream_of_data
    .try_for_each_concurrent(64, |v| {
         let client = &client;
         async move { ... }
}).await?;

just make sure that v is moved and client is moved by reference and this works

LouisGariepy commented 1 year ago

Moving to the book repo https://github.com/cornucopia-rs/website/issues/7