brianc / node-pg-query-stream

Query results from node-postgres as a readable (object) stream
MIT License
312 stars 37 forks source link

Is there a way to get the total number of rows as soon as the query executor knows the total result size? #54

Open gajus opened 5 years ago

gajus commented 5 years ago

I am streaming results and I would to cut off the stream after X rows, but I need to know how many rows there are in total.

I have tried to simply count chunks after I stop reading them, but the overhead is too big. It takes +30 seconds to count 30k+ rows.

matthieusieben commented 4 years ago

Add a count(*) as total in you select.

You can then make use of async iterables:

let current = 0
let max = 10
let total = 0
for await (const row of stream) {
  total = row.total
  current ++
  if (curent >= max) break // no need to go further
}

console.log("consumed " + current + " out of " + total + " rows")

Be aware of issues #52, #56, #66 if you choose to do that.