search

snowflakedb / snowflake-connector-nodejs

NodeJS driver
Apache License 2.0
120 stars 124 forks source link

SNOW-874803: stream rows api blocks the event loop #593

Open dynamikus opened 1 year ago

dynamikus commented 1 year ago

Please answer these questions before submitting your issue. In order to accurately debug the issue this information is required. Thanks!

  1. What version of NodeJS driver are you using? 1.6.23

  2. What operating system and processor architecture are you using? Windows

  3. What version of NodeJS are you using? (node --version and npm --version) 20.2.0

  4. What are the component versions in the environment (npm list)?

5.Server version:* E.g. 1.90.1 7.24.2

  1. What did you do?

I am using the stream rows api, to stream the results back. I have a modest data set of 100K. While data is getting downloaded no other process in the event loop is getting processed. I would assume the culprit is[ process.nextTick()] (https://github.com/snowflakedb/snowflake-connector-nodejs/blob/master/lib/connection/result/row_stream.js#L174) or something related to that.

This is because process.nextTick() is not technically part of the event loop. Instead, the nextTickQueue will be processed after the current operation is completed, regardless of the current phase of the event loop. Basically, any other task that is part of the event loop will get deferred till Snowflake is done getting the data.

  1. What did you expect to see?

I expect streaming data should be a non-blocking event, so it can give a chance to other processes in the event loop to run.

sfc-gh-dszmolka commented 1 year ago

hello and thank you for submitting this issue. it is quite astonishing to me, because looking at the git blame, this code part has been there forever (but at least since it was originally uploaded to Github, 5 years ago).

of course it doesn't mean it has to be like this going forward, so we're going to take a look how the row stream could be reimplemented to play better along with the eventloop.

dynamikus commented 1 year ago

Yeah same here, I have been using this sdk for 4+ years now, and never come across a scenario to process something else while waiting for the data to load till last week.

Anyway thank you for looking into this, let me know if you want a code example to replicate it.