Server Exception on transformation precondition

[pablojan]

During a long automatized test of collaborative writing (3 writers, 1 read-only) one of the writer clients got disconnected and the server shown following exception:

java.lang.IllegalStateException
    at com.google.common.base.Preconditions.checkState(Preconditions.java:429)
    at org.waveprotocol.box.server.waveserver.WaveletContainerImpl.transformSubmittedDelta(WaveletContainerImpl.java:399)
    at org.waveprotocol.box.server.waveserver.WaveletContainerImpl.maybeTransformSubmittedDelta(WaveletContainerImpl.java:358)
    at org.waveprotocol.box.server.waveserver.LocalWaveletContainerImpl.transformAndApplyLocalDelta(LocalWaveletContainerImpl.java:141)
    at org.waveprotocol.box.server.waveserver.LocalWaveletContainerImpl.submitRequest(LocalWaveletContainerImpl.java:99)
    at org.waveprotocol.box.server.waveserver.WaveServerImpl.submitDelta(WaveServerImpl.java:566)
    at org.waveprotocol.box.server.waveserver.WaveServerImpl.submitRequest(WaveServerImpl.java:367)
    at org.waveprotocol.box.server.frontend.ClientFrontendImpl.submitRequest(ClientFrontendImpl.java:228)
    at org.waveprotocol.box.server.frontend.WaveClientRpcImpl.submit(WaveClientRpcImpl.java:169)
    at org.waveprotocol.box.common.comms.WaveClientRpc$ProtocolWaveClientRpc$1.submit(WaveClientRpc.java:11208)
    at org.waveprotocol.box.common.comms.WaveClientRpc$ProtocolWaveClientRpc.callMethod(WaveClientRpc.java:11347)
    at org.waveprotocol.box.server.rpc.ServerRpcControllerImpl.run(ServerRpcControllerImpl.java:202)
    at org.waveprotocol.box.server.executor.RequestScopeExecutor$1.run(RequestScopeExecutor.java:73)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
``

[pablojan]

The exception recorded in the previous message was caused by an unexpected behavior of DeltaStoreBasedWaveletState.cachedDeltas: this map must contain all deltas already applied to the the wavelet snapshot DeltaStoreBasedWaveletState.snapshot, but the exception shows that sometimes, the latest delta is not present in cachedDeltas, whereas snapshot already has been updated with it. Both data structures are updated in the method appendDelta() (see below) but in original code, invocation to cachedDeltas.put()is done outside the synchronized block. With debugging code, it's demonstrated that cachedDeltas' map got desynchronized sometimes.

@Override
  public void appendDelta(WaveletDeltaRecord deltaRecord)
      throws OperationException {
    HashedVersion currentVersion = getCurrentVersion();
    Preconditions.checkArgument(currentVersion.equals(deltaRecord.getAppliedAtVersion()),
        "Applied version %s doesn't match current version %s", deltaRecord.getAppliedAtVersion(),
        currentVersion);

      if (deltaRecord.getAppliedAtVersion().getVersion() == 0) {
        Preconditions.checkState(lastPersistedVersion.get() == null);
        snapshot = WaveletDataUtil.buildWaveletFromFirstDelta(getWaveletName(), deltaRecord.getTransformedDelta());
        contributions = new WaveletContributions(deltasAccess.getWaveletName());
        contributions.apply(deltaRecord.getTransformedDelta());
      } else {
        // Avoid to update snapshot when it has being persisted
        synchronized (persistLock) {
          WaveletDataUtil.applyWaveletDelta(deltaRecord.getTransformedDelta(), snapshot);
          contributions.apply(deltaRecord.getTransformedDelta());
        }
      }

    // Now that we built the snapshot without any exceptions, we record the delta.
    cachedDeltas.put(deltaRecord.getAppliedAtVersion(), deltaRecord); <------- UNSAFE UPDATE

    // Increment counter controlling snapshot persistence
    deltasCountBeforeSnapshotStore++;
  }