cloudant / java-cloudant

A Java client for Cloudant
Apache License 2.0
79 stars 68 forks source link

Thread safe or Not ? [ com.cloudant.client.api.Database ] #281

Closed sagardiwan closed 8 years ago

sagardiwan commented 8 years ago

The javadoc does not explicitly mention if the Database objects are thread safe, so would request you to confirm.

So far we have seen that most of the methods on "Database" object are thread safe, except for findByIndex(). As part of a stress test against our application, we have seen that findByIndex() returns multiple results even though cloudant database has only one single document matching the cloudant selector query. This occurs very rarely , under 0.5% over a one hour test which includes more than 50 reads per second.

The version used was 2.4.2 when the test was run.

ricellis commented 8 years ago

Yes, we consider the Database object thread safe. [1]

Regarding the multiple results from findByIndex() I would be very surprised if that was related to threading on the client side. Are you seeing identical document ID and revision for all the results? What type of index and selector are you using?

sagardiwan commented 8 years ago

In this specific use case, currently the application does not expect multiple entries to be returned and simply throws an exception if it finds more than one results. We will try to make a code change and re-run the test to exactly log the the document and rev ids. The occurrence is very rare, can be reproduced only in a stress test.

We are using a "Text" type of index based on Apache Lucene [ Not Json index ] . The total number of records is apprx. 40 million and the selector json is very simple -

{"selector":{"empUserId":"abcdefghijklmno"}}

sagardiwan commented 8 years ago

An update - We added a quick fix to log _id and _rev and found that intermittently when we had multiple results, they were exactly the same record ... Same _id as well as _rev.

Few excerpts from the log after removing some details -

W readProfileByIndex Multiple profiles (2) found W 0:id:47f92fd717f911db8dd4d14dcc71f2db W 0:rev:1-21e44eccfb45a5915fe3ec2e068175c1 W 1:id:47f92fd717f911db8dd4d14dcc71f2db W 1:rev:1-21e44eccfb45a5915fe3ec2e068175c1

The stress test we are running can almost certainly recreate this issue.

We added a work around to just retry the search , i.e. exact same call again [ findByIndex ] and it works fine with retry , we get exactly one result back.

So we have a work around in place now.

ricellis commented 8 years ago

That seems very unusual - I suggest you send that information to support@cloudant.com (or open a support request from your Cloudant dashboard). Make sure to include your Cloudant account name as that will help them start investigating immediately.