Result Set Streaming

[jeffm13]

In #6, @cjbj asked for preferences on a result set streaming interface. I thought I'd start a new issue to begin the conversation.

One of my goals is to be able to use node-oracledb to drive ETL-style activities. I used node-oracle in the past to pull about 25 million rows from an Oracle database to an in-memory Redis no-sql database. I expect I'll need to be able to take on that type of activity even more frequently in the future for various reasons. To do it efficiently, streaming is a reasonable pattern.

Node-oracle implemented a reader concept that went in a good direction. But I'd like to see node-oracledb fully implement a Node.js readable stream, similar to how it was implemented with node-mysql or node-pt-query-stream with support for data, end, and error events. There might be a couple types of stream: maybe a 'raw' object stream that would return data in an array format, a 'formatted' object stream that would return a row in object form with column labels. I'd also need access to metadata for the query that originated the stream, to help me transform the results if I need to.

I'd like to be able to pipe a node-oracledb stream through transform streams and into writable streams, so conformance with the Node.js stream interface is important.

I think it might also be a good idea to think about streams for LOBs (maybe similar to how streams work in JDBC), but that's a separate issue.

[gx0r]

I wrote a lib https://github.com/llambda/oracle-query-stream is this useful?

[cjbj]

@jeff thanks for starting the thread. @llambda thanks for sharing. Let's see what other people think.,

[cjbj]

I'm going to close this and we can continue discussion in https://github.com/oracle/node-oracledb/issues/13