search

CraZySacX / node-jdbc

JDBC Wrapper for node.js
140 stars 106 forks source link

.toObjArray performance question #168

Closed pnahtanoj closed 6 years ago

pnahtanoj commented 6 years ago

We're doing some exploratory work with an Athena connector, and node-jdbc works great -- except with v large data sets. I have a query returning ~50k rows, and the toObjArray function is adding about 60secs to the process (running on my macbook pro). I verified this by commenting out the resultset.toObjArray call and returning the ResultSet directly.

Is this to be expected? We're trying to asses whether this strategy is going to work for us.

CraZySacX commented 6 years ago

Yeah toObjectArray has definitely not been optimized for performance over large datasets. You would probably be better off operating on the result set directly.

pnahtanoj commented 6 years ago

This is pretty foreign territory for me - any insights on how to operate directly on the result set?

CraZySacX commented 6 years ago

Take a look at the docs "https://docs.oracle.com/javase/7/docs/api/java/sql/ResultSet.html"

toObjArray uses the defaults for ResultSet, and they are fairly constraining. I would take a look creating a more custom result set and use the Java API to move as needed along it.