DatastorePersistenceHandler has unsafe threading behaviour on DevServer resulting in infinite loops in HashMap.get

[GoogleCodeExporter]

What steps will reproduce the problem?
1. Since this is a concurrency problem multiple runs may be required, recommend 
that testing occurs on a multi-core machine as not observed on single core VM.
2. Have some multiple clients concurrently access a servlet which uses a single 
  private static final PersistenceManagerFactory pmfInstance = JDOHelper
      .getPersistenceManagerFactory("transactions-optional");
  and uses a pattern like:
  @Override
  public boolean deleteUser(User existingUser) {
    PersistenceManager pm = pmfInstance.getPersistenceManager();
    try {
      AEUser existing = pm.getObjectById(AEUser.class, AEUser.keyForUser(existingUser.getPublicKey()));
      if (existing != null) {
        pm.deletePersistent(existing);
        return deleteUserData(existing);
      } else {
        return true;
      }
    } catch (JDOObjectNotFoundException e) {
      return false;
    } finally {
      pm.close();
    }
  }
  @Override
  public RevValue getRevision(User user, byte[] index, byte[] revision) throws IOException {
    PersistenceManager pm = pmfInstance.getPersistenceManager();
    try {
      Key lookupKey = getLookupKey(user, index);
      Key revisionKey = AppEngineRecord.makeKey(lookupKey, new BytesRevision(revision));
      // If this doesn't exist there is no key so null gets returned by JDOObjectNotFoundException
      AppEngineRecord record = pm.getObjectById(AppEngineRecord.class, revisionKey);
      return new RevValue(revision, record.getValue());
    } catch (JDOObjectNotFoundException e) {
      return null;
    } finally {
      pm.close();
    }
  }
  @Override
  public boolean putRecord(User user, byte[] index, byte[] bRevision, byte[] data) {
    PersistenceManager pm = pmfInstance.getPersistenceManager();
    Revision revision = new BytesRevision(bRevision);
    try {
      Key lookupKey = getLookupKey(user,index);
      Lookup lookup;
      try {
        lookup = pm.getObjectById(Lookup.class, lookupKey);
      } catch (JDOObjectNotFoundException e) {
        lookup = new Lookup(lookupKey, index);
        pm.makePersistent(lookup);
      }
      AppEngineRecord record = new AppEngineRecord(lookup.getKey(), revision, data);
      pm.makePersistent(record);
      return true;
    } finally {
      pm.close();
    }
  }
  @Override
  public boolean checkAndAddNonce(Nonce nonce, byte[] publicKey) {
    if (!nonce.isRecent()) {
      return false;
    }
    PersistenceManager pm = pmfInstance.getPersistenceManager();
    try {
      AENonce aeNonce = new AENonce(nonce,publicKey);
      try {
        pm.getObjectById(AENonce.class,aeNonce.getKey());
        return false;// getObjectById should throw an exception
      } catch (JDOObjectNotFoundException e) {
        // We haven't seen this nonce yet so add it and return true
        pm.makePersistent(aeNonce);
        return true;
      }
    } finally {
      pm.close();
    }
  }
  for interacting with it. (I can put the full source code somewhere necessary if this helps)
3. Notice that the server eventually ends up with one or two threads stuck in 
an infinite loop inside HashMap.get.

What is the expected output? What do you see instead?
Expected: normal termination. Instead: does not terminate terminate.

What version of the product are you using? On what operating system?
2.0.0-final on Linux Ubuntu 11.04
$ java -version
java version "1.6.0_26"
Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode)
$ mvn --version
Apache Maven 3.0.3 (r1075438; 2011-02-28 17:31:09+0000)
Maven home: /home/drt24/bin/apache-maven-3.0.3
Java version: 1.6.0_26, vendor: Sun Microsystems Inc.
Java home: /usr/lib/jvm/java-6-sun-1.6.0.26/jre
Default locale: en_GB, platform encoding: UTF-8
OS name: "linux", version: "2.6.38-14-generic", arch: "amd64", family: "unix"

Please provide any additional information below.
com.google.appengine.datanucleus.DatastorePersistenceHandler has two maps:
private final Map<ExecutionContext, BatchPutManager> 
batchPutManagerByExecutionContext = new HashMap();

  private final Map<ExecutionContext, BatchDeleteManager> batchDeleteManagerByExecutionContext = new HashMap();

Used for example in:
  protected BatchPutManager getBatchPutManager(ExecutionContext ec) {
    BatchPutManager putMgr = batchPutManagerByExecutionContext.get(ec);
    if (putMgr == null) {
      putMgr = new BatchPutManager();
      batchPutManagerByExecutionContext.put(ec, putMgr);
    }
    return putMgr;
  }
However when multiple clients are concurrently accessing the server concurrent 
accesses can be made to the batchPutManagerByExecutionContext resulting in it 
sometimes entering an inconsistent state were there is a loop in its data 
structure resulting in an infinite loop in the get method called in the code 
fragment above and the following stack:

Thread [2146992056@qtp-1975801649-36] (Suspended)   
    HashMap<K,V>.get(Object) line: 320  
    DatastorePersistenceHandler.getBatchPutManager(ExecutionContext) line: 153  
    DatastorePersistenceHandler.insertObject(ObjectProvider) line: 208  
    JDOStateManager.internalMakePersistent() line: 2371 
    JDOStateManager.makePersistent() line: 2347 
    MultithreadedObjectManager(ObjectManagerImpl).persistObjectInternal(Object, FieldValues, ObjectProvider, int, int) line: 1798   
    MultithreadedObjectManager(ObjectManagerImpl).persistObjectWork(Object, boolean) line: 1647 
    MultithreadedObjectManager(ObjectManagerImpl).persistObject(Object, boolean) line: 1512 
    MultithreadedObjectManager.persistObject(Object, boolean) line: 298 
    JDOPersistenceManager.jdoMakePersistent(Object) line: 740   
    JDOPersistenceManager.makePersistent(Object) line: 765  
    AppEngineDatabase.checkAndAddNonce(Nonce, byte[]) line: 335 
    DatabaseNigoriProtocol.authenticateUser(NigoriMessages$AuthenticateRequest, byte[]...) line: 83 
    DatabaseNigoriProtocol.get(NigoriMessages$GetRequest) line: 138 
    NigoriServlet$JsonGetRequestHandler.handle(HttpServletRequest, HttpServletResponse) line: 183   
    NigoriServlet.doPost(HttpServletRequest, HttpServletResponse) line: 371 
    NigoriServlet(HttpServlet).service(HttpServletRequest, HttpServletResponse) line: 637   
    NigoriServlet(HttpServlet).service(ServletRequest, ServletResponse) line: 717   
    ServletHolder.handle(ServletRequest, ServletResponse) line: 511 
    ServletHandler$CachedChain.doFilter(ServletRequest, ServletResponse) line: 1166 
    HeaderVerificationFilter.doFilter(ServletRequest, ServletResponse, FilterChain) line: 35    
    ServletHandler$CachedChain.doFilter(ServletRequest, ServletResponse) line: 1157 
    ServeBlobFilter.doFilter(ServletRequest, ServletResponse, FilterChain) line: 60 
    ServletHandler$CachedChain.doFilter(ServletRequest, ServletResponse) line: 1157 
    TransactionCleanupFilter.doFilter(ServletRequest, ServletResponse, FilterChain) line: 43    
    ServletHandler$CachedChain.doFilter(ServletRequest, ServletResponse) line: 1157 
    StaticFileFilter.doFilter(ServletRequest, ServletResponse, FilterChain) line: 122   
    ServletHandler$CachedChain.doFilter(ServletRequest, ServletResponse) line: 1157 
    BackendServersFilter.doFilter(ServletRequest, ServletResponse, FilterChain) line: 97    
    ServletHandler$CachedChain.doFilter(ServletRequest, ServletResponse) line: 1157 
    ServletHandler.handle(String, HttpServletRequest, HttpServletResponse, int) line: 388   
    SecurityHandler.handle(String, HttpServletRequest, HttpServletResponse, int) line: 216  
    SessionHandler.handle(String, HttpServletRequest, HttpServletResponse, int) line: 182   
    DevAppEngineWebAppContext(ContextHandler).handle(String, HttpServletRequest, HttpServletResponse, int) line: 765    
    DevAppEngineWebAppContext(WebAppContext).handle(String, HttpServletRequest, HttpServletResponse, int) line: 418 
    DevAppEngineWebAppContext.handle(String, HttpServletRequest, HttpServletResponse, int) line: 78 
    JettyContainerService$ApiProxyHandler(HandlerWrapper).handle(String, HttpServletRequest, HttpServletResponse, int) line: 152    
    JettyContainerService$ApiProxyHandler.handle(String, HttpServletRequest, HttpServletResponse, int) line: 369    
    Server(HandlerWrapper).handle(String, HttpServletRequest, HttpServletResponse, int) line: 152   
    Server.handle(HttpConnection) line: 326 
    HttpConnection.handleRequest() line: 542    
    HttpConnection$RequestHandler.content(Buffer) line: 938 
    HttpParser.parseNext() line: 755    
    HttpParser.parseAvailable() line: 218   
    HttpConnection.handle() line: 404   
    SelectChannelConnector$ConnectorEndPoint(SelectChannelEndPoint).run() line: 409 
    QueuedThreadPool$PoolThread.run() line: 582 

It looks like the unsafe use of HashMap was added in r616.

Original issue reported on code.google.com by scottish...@googlemail.com on 23 Apr 2012 at 12:46

[GoogleCodeExporter]

SVN trunk has those as ConcurrentHashMaps. No multi-core available here, so 
left as an exercise ...

Original comment by googleco...@yahoo.co.uk on 26 Apr 2012 at 5:01

• Changed state: Fixed
• Added labels: TargetRelease-2.1.0

[GoogleCodeExporter]

Is there an ETA for 2.1.0, I would like to know how vigorously I need to look 
for workarounds.

Original comment by scottish...@googlemail.com on 14 May 2012 at 4:49

[GoogleCodeExporter]

Google are the only ones who release this, so up to them to provide timescales. 
v2.0.1 ought to be released pretty soon IIRC, but that's as far as I've heard. 
You could always build manually ("mvn clean install") and try

Original comment by googleco...@yahoo.co.uk on 23 May 2012 at 7:16