datasalt / splout-db

A web-latency SQL spout for Hadoop.
50 stars 14 forks source link

Thread safe SQLite connections #6

Closed ivanprado closed 11 years ago

ivanprado commented 11 years ago

SQLite4Java has a Connection object that is not completely thread safe (http://almworks.com/sqlite4java/javadoc/com/almworks/sqlite4java/SQLiteConnection.html) . To overcome this limitation there is one connection per each thread on the DNode. But each connection opens a database file, increasing the number of files opened a lot.

In order to reduce the number of opened files, would be nice to be able to share one Connection between various threads. Seems that some of the methods on the Connection object are thread safe, so the idea to analyze them carefully (see http://almworks.com/sqlite4java/javadoc/com/almworks/sqlite4java/SQLiteConnection.html) and implement the use of one connection by many threads.

pereferrera commented 11 years ago

AFAIK the relevant methods are not thread-safe so I don't see what we can do here. The only relevant operation which is thread-safe is interrupting a query.

One way of decreasing the number of needed connections would be to make the DNode have less threads, for example by handling requests using NIO and few reactor threads.

pereferrera commented 11 years ago

Which makes this thread somewhat linked to issue #5

pereferrera commented 11 years ago

I'm closing this issue by now after c501d36 .