search

epam / Indigo

Universal cheminformatics toolkit, utilities and database search tools
http://lifescience.opensource.epam.com
Apache License 2.0
315 stars 105 forks source link

Concurrent Bingo inserts result in corruption #108

Open dan2097 opened 6 years ago

dan2097 commented 6 years ago

When performing inserts from multiple threads concurrently into a column with a Bingo chemical index the resulting index becomes corrupt. I attach some code that should quickly elicit the issue. I tested with PSQL9.6 + latest code compiled from source, but the same issue was also reproducible on PSQL9.4 and the last official Bingo release. Test.java.txt The postgres driver I used was:

    <dependency>
      <groupId>org.postgresql</groupId>
      <artifactId>postgresql</artifactId>
      <version>9.4.1212.jre7</version>
    </dependency>

The code initially produces Exceptions with messages like "An I/O error occurred while sending to the backend." followed by exceptions complaining about corrupted blocks. If the same code is run with one thread (rather than ten threads) it completes successfully.

This may be the same issue as was reported here: https://groups.google.com/forum/#!topic/indigo-general/wIfmsla8gXA

voddan commented 6 years ago

The results of running the attached example Test.java.txt (by @dan2097) :

The static block will create the table/index in a database called bingodb. Then 10 threads add linear alkanes to the database. It initially gives Exceptions like:

    at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:314)
    at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:430)
    at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:356)
    at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:168)
    at org.postgresql.jdbc.PgPreparedStatement.execute(PgPreparedStatement.java:157)
    at Test$IndexerThread.run(Test.java:48)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketException: Connection reset
    at java.net.SocketInputStream.read(SocketInputStream.java:210)
    at java.net.SocketInputStream.read(SocketInputStream.java:141)
    at org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:140)
    at org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:109)
    at org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:67)
    at org.postgresql.core.PGStream.receiveChar(PGStream.java:280)
    at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1916)
    at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:288)
    ... 8 more

Subsequently trying to do inserts gives Exceptions like:

    at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2455)
    at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2155)
    at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:288)
    at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:430)
    at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:356)
    at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:168)
    at org.postgresql.jdbc.PgPreparedStatement.execute(PgPreparedStatement.java:157)