Generic query result bulk loading operations

[GoogleCodeExporter]

Right now there's SQLiteStatement.loadLongs() which is a fast way to transfer 
one column of integer values from SQLite into Java. A generic method to 
transfer any type of data and in multiple columns is needed.

Original issue reported on code.google.com by ser...@gmail.com on 29 Jun 2011 at 12:36

[GoogleCodeExporter]

It turns out that modern JDK have quite efficient implementation of JNI calls. 
Our benchmarks showed only 20% gain from loadLongs() on Windows, Linux, OS X 
with JDK 1.6 - 1.8. Only Android platform has shown 7x gain.

Considering large effort to design a generalization of loadLongs for arbitrary 
columns, we have decided to postpone the implementation of this feature.

Since there are people who have starred this request, I'd like to ask for your 
feedback - is this really needed and what are your circumstances?

Kind regards,
Igor

Original comment by ser...@almworks.com on 22 Sep 2014 at 10:15

• Added labels: Priority-Low
• Removed labels: Priority-Medium

[GoogleCodeExporter]

I think it is perfectly reasonable to postpone this optimization.
Best.
Le 22 sept. 2014 12:15, <sqlite4java@googlecode.com> a écrit :

Original comment by dbouys...@gmail.com on 24 Sep 2014 at 8:31

[GoogleCodeExporter]

I am also interested in this feature. The question might sound stupid after the 
responses, but would be also many effort to make a version for other data 
types? For example, SQLiteStatement.loadDouble() or 
SQLiteStatement.loadString(). I think that covering most of the data types 
would be interesting. I am specifically interested in the double one, since I 
am creating a database with data collected from wearable devices.

Regards,
Alejandro.

Original comment by alejandr...@gmail.com on 11 Nov 2014 at 5:08

[GoogleCodeExporter]

Alejandro, do you need to just load multiple double values or do you need a 
speed-optimized method? In the former case, you can just iterate through 
statement results and retrieve the values with columnDouble().

If you need an optimized method, let me ask you, what platform does your code 
run on? What JVM version do you have there?

Igor

Original comment by ser...@gmail.com on 11 Nov 2014 at 7:29

[GoogleCodeExporter]

Currently I am interating through statement results and retrieving the values 
with columnDouble(). I did a quick benchmark selecting 5 millions of rows and 
got a great improvement with the SQLiteStatement.loadInt() (in some cases even 
50% faster!). Probably wasn´t the most reliable benchmark ever, but it was 
enough for me.The database will have eventually 50 millions of rows or even 
more so that I am interesting in a speed-optimized method (any idea is 
welcome!). I am running the application on Windows but in the future will run 
on OSX and maybe on Linux. I am using the JVM 7.

Regards,
Alejandro.

Original comment by alejandr...@gmail.com on 11 Nov 2014 at 11:29

[GoogleCodeExporter]

I forgor to say that you did a great job with the wrapper. As a developer, I 
know how good is when people recognise your work.

Alejandro

Original comment by alejandr...@gmail.com on 11 Nov 2014 at 11:35

[GoogleCodeExporter]

Alejandro, thanks for your kind words :) 

At the time when loadLongs() method was introduced, the performance improvement 
was around 10x faster, so it was a big deal. We did benchmarks with the modern 
JVMs - the results are mentioned above, only 20% improvement, maybe even 50% 
improvement as you mentioned. In our opinion, this is not that bad. 

We may still work on providing a generic method for bulk loading results, but 
given the complexity involved (and we're not going to just add loadDoubles() 
method, but rather load multiple columns of different types into a buffer), we 
won't be able to work on it in the near future.

So the issue remains open, but low priority. Sorry about that! If you very much 
need loadDoubles() method, you can try adding it yourself, basically making a 
copy of the call chain under loadLongs() method. Please let us know if you 
succeed.

Kind regards,
Igor

Original comment by ser...@gmail.com on 12 Nov 2014 at 7:29

[GoogleCodeExporter]

I totally understand it. No worries! I don't think I will try it now, but if I 
will let you know if I try it in the future. Thank you for your response.

Regards,
Alejandro.

Original comment by alejandr...@gmail.com on 13 Nov 2014 at 8:52