search

snowflakedb / snowflake-connector-nodejs

NodeJS driver
Apache License 2.0
125 stars 132 forks source link

SNOW-889843: Feature Request: support direct read from SF in Apache Arrow format #613

Open davisford opened 1 year ago

davisford commented 1 year ago

What is the current behavior?

A while back the Snowflake team announced support in the Python and JDBC drivers to read direct from the database in Apache Arrow format: https://www.snowflake.com/blog/fetching-query-results-from-snowflake-just-got-a-lot-faster-with-apache-arrow/

It doesn't appear that the NodeJS driver supports this now.

What is the desired behavior?

I would love to see this ability supported in the NodeJS driver, if possible. The client we are sending the data to supports arrow and so this would be a huge performance boon by avoiding the serdes steps during data transfer.

How would this improve snowflake-connector-nodejs?

Apache Arrow seems to be the up and coming de-facto standard for data transfer format, so I'm sure many other users of the driver would find it useful to be able to pull data from Snowflake in the arrow format.

References, Other Background

What is your Snowflake account identifier, if any?

NABANCARD

sfc-gh-dszmolka commented 1 year ago

hello and thank you for raising this request with us - it is a very valid one. There are already plans in place to support Arrow format natively in snowflake-sdk, but at this moment I cannot give any estimation for when it could be actually available. I'll keep this Issue updated as new information becomes available - which might take a while so thank you in advance bearing with us !

Since you already seem to be a Snowflake customer, if you did not do so yet, I would suggest reaching out to your Sales representative and explaining how this feature is important to you - these inputs can help the team prioritize the requests.

davisford commented 1 year ago

Thanks @sfc-gh-dszmolka -- I did reach out to our sales rep and let him know. We'll keep an eye on this ticket.

bradleygore commented 11 months ago

Not wanting to spam this with +1 comments - is there a project board or roadmap where we can view items like this and where they are in the In Progress flow?

sfc-gh-dszmolka commented 11 months ago

unfortunately there is no publicly visible timeline / roadmap exposed, but I now asked for an update from the product team to share if they have any insight or even a rough estimation on this. Will update this thread with any new information as it becomes available to me. I think I added the In Progress label a bit prematurely; removed it for now.

If (any of) you are a Snowflake customer already; you can also ask your account team / Sales rep. for progress update on the matter; the internal tracking id is in the subject of the request - SNOW-889843. You can also express the importance of the feature implementation to them as well