during startup there is a lock race

One thread is starting up (on request), this triggers merging of leveldb structures. As this is a query the lock is shared. But the query has a side-effect, so another query issued in parallel runs into:

net.strus.service.impl.ws.WebService$WebServiceException: error creating storage client: error creating storage client: failed to create database client: error creating database client: failed to open key value store database: IO error: lock /data/struswebservice/storage/aza/LOCK: Resource temporarily unavailable
    at net.strus.service.impl.ws.WebService.query(WebService.java:426)
    at com.eurospider.og.webui.servlet.StrusService.search(StrusService.java:129)
    at com.eurospider.og.webui.servlet.SearchServlet.search(SearchServlet.java:197)
    at com.eurospider.og.webui.servlet.SearchServlet.doSearch(SearchServlet.java:175)
    at com.eurospider.og.webui.servlet.SearchServlet.doGet(SearchServlet.java:108)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:624)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:218)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:505)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
    at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:956)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:442)
    at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:190)
    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:640)
    at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:745)

This is not dramatic, but shows that:

opening indexes on demand should know that leveldb needs a merge and must promote the read lock to a write lock
those merges should be triggerable on demand or have configuration options to trigger them after N seconds/minutes or after N documents.

patrickfrey / strusWebService

during startup there is a lock race #61