search

r-dbi / dbi3

DBI revisited
https://r-dbi.github.io/dbi3
37 stars 2 forks source link

Progress bar for dbWriteTable? #16

Open cboettig opened 3 years ago

cboettig commented 3 years ago

Is it possible to show a progress bar for long-running dbWriteTable commands?

krlmlr commented 2 years ago

Maybe with a callback function?

If we do it, it should be part of dbAppendTable() . The way the default method is currently implemented (via sqlAppendTableTemplate()) is abysmally slow for remote connections. The only portable solution seems to be to use dbQuoteLiteral() to create a long SQL query, which is what sqlAppendTable() does (or should do, r-dbi/DBI#362).

How long may these queries become? How to make the process RAM-friendly?

All of this is a lot of work and doesn't really fit the scope of DBI; doing this in the individual backends feels even worse. Maybe there's room for a package that specializes in writing/copying large datasets across DBI backends?