search

objectbox / objectbox-java

Android Database - first and fast, lightweight on-device vector database
https://objectbox.io
Apache License 2.0
4.38k stars 302 forks source link

Transactions throw DbMaxReadersExceededException even when there are no active readers #1000

Closed MrSlateZB closed 2 years ago

MrSlateZB commented 3 years ago

Describe the bug When opening transactions in a rapid fashion on multiple threads DbMaxReadersExceededException is often thrown trying to start a transaction. I've created a test in a PR that will recreate this issue despite only allowing one transaction to occur at a time. The issue seems related to running tests on a thread-pool.

I suspect there is some code happening on the native side that is allowing the other transaction tests that do not use thread-pools to succeed despite rapid creation of transactions.

Basic info (please complete the following information):

I have created https://github.com/objectbox/objectbox-java/pull/999 that adds a test that failed on multiple of my development machines. In this test, we are creating transactions on an unbounded thread pool. However, we are synchronizing each transaction call so, in reality, only 1 transaction should be occurring a time. Despite this, we still receive DbMaxReadersExceededException issues.

MrSlateZB commented 3 years ago

As an FYI, I have also discovered that if you bound the ThreadPool to have threads <= the max number of DB readers, the issue no longer seems to occur. I've added a test for this to the PR.

greenrobot-team commented 3 years ago

Thank you for this detailed report and even the test PR! I have reported this internally.

greenrobot commented 3 years ago

Quick update: got some confirmation by using .debugFlags(DebugFlags.LOG_TRANSACTIONS_READ):

...
006-13:06:45.4678 [INFO ] TX #170 (read)
006-13:06:45.4678 [INFO ] TX #170 recycle
006-13:06:45.4679 [INFO ] TX #170 to be destroyed on owner thread...
006-13:06:45.4679 [INFO ] TX #170 destroyed
126-13:06:45.4679 [INFO ] TX #171 (read)
126-13:06:45.4679 [INFO ] TX #171 recycle
126-13:06:45.4679 [INFO ] TX #171 to be destroyed on owner thread...
126-13:06:45.4679 [INFO ] TX #171 destroyed
058-13:06:45.4680 [INFO ] TX #172 (read)
058-13:06:45.4680 [INFO ] TX #172 recycle
058-13:06:45.4680 [INFO ] TX #172 to be destroyed on owner thread...
058-13:06:45.4680 [INFO ] TX #172 destroyed
127-13:06:45.4681 [INFO ] TX #173 (read)
127-13:06:45.4681 [ERROR] Storage error "MDB_READERS_FULL: Environment maxreaders limit reached" (code -30790)

127 is the thread number and it's extremely close to the number of default readers. So indeed looks like the reader slot of a thread is not vacated in this case.

PS.: Added a core (native) test case, which seems to run fine with 3000 threads... 

000-18:45:48.1287 [INFO ] TX #3003 (read)
000-18:45:48.1287 [INFO ] TX #3003 recycle
000-18:45:48.1287 [INFO ] Cursor for TX #3003 closed
000-18:45:48.1287 [INFO ] TX #3003 to be destroyed on owner thread...
000-18:45:48.1287 [INFO ] TX #3003 aborted
000-18:45:48.1287 [INFO ] TX #3003 destroyed
greenrobot commented 3 years ago

Unbounded threads (pools) will require a setting special in the next version. If you are interested in details before the release, here are the two most relevant commits: https://github.com/objectbox/objectbox-java/commit/8fd8a8c29293407bc4fd993ad3eaea2261ebc76c and https://github.com/objectbox/objectbox-java/commit/b76c39bdae801a479ac3ecb8ae1303b2cb43a238

Not sure if giving a general recommendation on that potentially heavy multithreading makes sense yet. Sometimes less threads is better, e.g. number of cores - depends on the use case ofc...

PS.: Our test suites completed just fine with the new noReaderThreadLocals flags set always to true manually.

PPS.: on Windows, thread locals seem rather flaky (32 bit Windows seems to be affected more than 64 bit). If you run into this issue on this platform, the new flag should help.

MrSlateZB commented 3 years ago

I'd gladly test this out on my machine as well. When I attempt to run tests though, I get this error in gradle:

Could not find io.objectbox:objectbox-windows:2.9.2-dev-SNAPSHOT.

What is the correct way to resolve this?

greenrobot commented 3 years ago

Sorry, dev-SNAPSHOTs are not publicly available. Maybe we could grab it for this purpose from CI and put in into some release here on github, @greenrobot-team ?

greenrobot-team commented 3 years ago

@MrSlateZB The Java library requires native counterparts. As snapshots are not available publicly you can only run tests in this library when using a release tag (or by overriding the dependency to a published version).

Anyhow, the suggested change requires updated native libraries. They will be part of the next (preview) release. Then you should be able to test it on your machine.

greenrobot-team commented 3 years ago

@MrSlateZB The BoxStoreBuilder.noReaderThreadLocals() flag is now available publicly with the (unannounced) 2.9.2-RC4 release!

greenrobot-team commented 2 years ago

This is now available with the 3.0.1 release.