Session/state replication

[ebruchez]

Current status

As of 2011-09-26, the XForms engine session cannot be replicated because:

• some objects are non-serializable
• some data is not stored into the session anyway directly (indirection via keys) the dynamic state is not serialized after each mutation anyway

  Implementing efficient session replication

• if possible, we don't want to serialize all the dynamic state right away after each interaction
  • NOTE: Tomcat cluster seems to force this after each request
• could use Externalizable to serialize efficiently when requested
• session would store, for a given form UUID, separate keys:
  • immutable dynamic state data
  • request number
  • 1 key per instance
  • control data when needed (to split too?)
  • ⇒ this assumes that setAttribute or removeAttribute used by session replication code to detect, so might not be even useful at all! See Tomcat doc.

    Other questions/issues

• Q: any way to tell app server what has changed/hasn't changed in session?
• Q: if separate session keys are used, how to ensure that the values are consistent?

  External resources

• Tomcat:
  • "For each request the entire session is replicated, this allows code that modifies attributes in the session without calling setAttribute or removeAttribute to be replicated." ⇒ unclear how up to date this is, based on this SO question.
  • "If your client accesses the same session simultanously using multiple requests, then the last request will override the other sessions in the cluster."
• JBossSerialization
• WebSphere: "Distributed HTTPSession support does not guarantee transactional integrity of an attribute in a failover scenario or when session affinity is broken."

+1 from customer

[avernet]

+1 from community

[ebruchez]

We discussed today with @avernet that we could also use Ehcache replication.

[ebruchez]

Steps for a prototype:

• baseline analysis
  • what kind of state do we have?
  • how much state do we have (objects, KB)?
• what should replicate at each update
  • whole state?
  • or smaller part of the state?
• expectations
  • feasibility
  • typical time to replicate state

Q: Any way to do anything using browser local state?

[ebruchez]

Tomat has a DeltaManager session manager which "only replicat[es] the deltas in data". However documentation doesn't stay what that actually means (how does the algorithm work?). It might be by session attribute.

[ebruchez]

What we store in the session:

• cache session listener (oxf.xforms.state.manager.session-listeners-key.$uuid)
• SessionDocument(lock: Lock, uuid: String) via oxf.xforms.state.manager.uuid-key.$uuid
• Connection state
• fr-language
• UploadProgress
• oxf.xforms.state.async-submissions.$uuid
• deleteFileOnSessionTermination
• DynamicResource via orbeon.resources.dynamic.$digest
• anything else?

What we store indirectly via UUID into caches:

• static state serialized as String
• dynamic state as DynamicState

[ebruchez]

Data: initial state for Controls form:

• size: 107775
• versionedPathMatchers: 127
• pendingUploads: 4
• instances: 104294
• controls: 1446
• annotatedTemplate: 1
• lastAjaxResponse: 1

So we are talking about 100KB, mostly of instances.

Need more data of course.

[ebruchez]

One issue: unless there is a shared filesystem where temporary files, uploads which haven't been saved to a database would be lost when failing over.

[ebruchez]

This blog post from almost 6 years ago indicates about 120 MB/s with Ehcache via RMI for 60 KB objects, less as you increase object size (and about 50% of that with JGroups).

[ebruchez]

• We have to add the serialization time for DynamicState.
• Blog post doesn't mention CPU usage for the operation.

[ebruchez]

• If we replicate via Ehcache, we also have to replicate sessions, as we have essential stuff in the session.
• Ideally, replication would possibly happen via small diffs, not only full XML instances but as a change log: this value changed, etc., and be reconciled on the other side.

[ebruchez]

Unclear, based on this, whether using Ehcache for Tomcat session replication is possible independently from the rest of Shiro:

EHCache is also a nice choice if you quickly need container-independent session clustering. You can transparently plug in TerraCotta behind EHCache and have a container-independent clustered session cache. No more worrying about Tomcat, JBoss, Jetty, WebSphere or WebLogic specific session clustering ever again!

It seems that, maybe, Shiro creates its own sessions and you wouldn't use web sessions at all.

[ebruchez]

We have a few options:

1. Do everything with the session. Then we have the problem of the static state, and of configurations which might be different between app servers.
2. Do everything from Ehcache. In this case, we could move everything we store in the session to Ehcache, and index into Ehcache using the session id. Hopefully the session id is preserved across replicated servlet containers.
3. Do a mix. This seems like it would be a pain as servlet session AND Ehcache would need to replicate similarly but with different configurations.

So I think that option 2 might be the best so far.

[avernet]

I agree. Also, with the "everything in the session" option, where would we store what we can't afford to keep in memory, and right now serialize to disk through Ehcache? It seems to me that anyway, we can't keep everything in the session.

[ebruchez]

• [x] refactor session handling and forward to Ehcache cache
• [x] prototype: live XFormsContainingDocument in XFormsDocumentCache must also store their state in the Ehcache session
• [x] initial test of local replication
• [x] CHECK: Credentials in native session should be replicated
  • not a problem since we replicate the servlet session
• [x] handle concurrent access to session map and replication, as multiple concurrent requests can update the session Map [CHECK: not using a Map now but individual keys]
• [x] "Got unexpected request sequence number" when reverting to original server
• [x] configuration property to enable replication as that causes extra state serialization
• [x] XFormsServerSharedInstancesCache stores URI with port, which fails when loading missing instance
• [x] xforms.state cache must be initialized early so it can start replicating even before there is a first request coming in
• [x] clarify uploads which haven't been saved to a database (lost when failing over? if so must do it gracefully)
• [x] i18n upload message in persistence-model.xml
• [x] Q: Should last Ajax response be replicated? A: Probably, check logic and how to replicate.
  • A: This is already part of DynamicState so we should be good.
• [x] RMIAsynchronousCacheReplicator - Object with key orbeon.session.FA384F87C3A644A798B671164A5AF03E.oxf.xforms.state.manager.session-listeners-key.a1b7283d86d79c05cab03de89a6efbc5c7aefae2 is not Serializable and cannot be replicated
• [x] check isPE()
• [x] determine/refine ehcache.xml configuration
  • [x] maxElementsInMemory
  • [x] replicate xforms.resources
  • [x] xforms.xbl cache doesn't seem to be used!
• [x] NPE with clearMissingUnsavedDataAttachmentReturnFilenamesJava in Form Builder after replication, seems that data is null (replace with `xxf:instance())
• [x] move code from master for "sequence number is ok if user switched server but server was just slow"
• [x] solution for removing documents from cache following session expiration, as session listeners are not replicated right now
  • could we use a single listener which somehow find the list of all UUIDs in use for the session?
  • UPDATE: Now we serialize the listener. It only holds the UUID.
  • BUT there is a problem: our SessionListeners class uses custom serialization/deserialization, and deserializes to an empty list of listeners.
• [x] test setup
  • [x] Docker instances
  • [x] multicast network configuration
  • [x] HAProxy setup
  • [x] check whether if Tomcat's JVM must have more heap and increase if needed
• [x] tests
  • [x] uploads: test that missing uploads are detected and cleared and message to user shows
• [x] misc
  • [x] on master, check if xforms.xbl is in fact not used an remove if that's the case
• [ ] doc
  • [x] Orbeon Forms setup
  • [x] Tomcat setup
  • [x] HAProxy setup
  • [x] PE-only (also on web site)
  • [x] limitations (uploads, more?)
  • [ ] diagrams

[ebruchez]

Question raised by @avernet: if we do not store anything in the actual servlet session anymore, could we not use/check the session at all? Could this help with occasional issues users have with sessions in particular in embedded settings?

One idea is that the XForms document's UUID could be sufficient.

There are are few things where a setting by user can make sense, such as fr-language. It might be, at this point, the only setting shared per session.

[ebruchez]

We currently have a 2-level XForms document cache/store:

1. In-memory cache of live XFormsContainingDocument
2. When evicted from that cache, we put documents in the EhcacheStateStore, which creates an instance of DynamicState. This maps to the xforms.state cache, which is configured as a disk cache. Upon serializing to disk, Java serialization apples to DynamicState as it is a Scala case class which is a java.io.Serializable.

Given this, replication causes an issue, which is that after each Ajax request, the new state must be propagated.

Possibilities:

• store the new DynamicState after each request
  • this might be costly but we don't know how much yet exactly
• figure out a way to split the state into more than one part

(BTW as discussed with @avernet state serialization is likely much cheaper than deserialization, as deserialization entails recreating and initializing an XFormsContainingDocument including RRR and creating the whole tree of controls, evaluating lots of XPath expressions, etc.)

[ebruchez]

To clarify, it's not entirely true we don't need to replicate sessions even if we store everything in Ehcache: session replication needs to be enabled for the servlet containers, otherwise the container won't be able to accept the incoming session cookie from the client, and we won't have a session id at our disposal. However, no session objects will have to be replicated in this case, except maybe fr-language or secondary user settings like this.

The above, of course, if we keep checking the session at all. If we don't, then from Orbeon Forms we wouldn't even need to replicate the session at all, and a purely servlet session-less operation would be possible.

In practice, I would imagine that session would be needed by most deployments, be it only to handle logged in users, unless some third-party solutions like Apache Shiro are used.

[ebruchez]

Defer:

• portlet support
• Java servlet embedding support

[ebruchez]

We need 2 caches:

• XForms state store, which is a disk store only
• session cache, which is a memory cache only (probably)

Both need to be replicated the same way.

[ebruchez]

One question is that, since we do need session replication after all, should we use Ehcache for a session cache? What is the benefit?

One thing is that, for the XForms dynamic state, we do passivation to disk. Now we could use the servlet session, and configure that to do passivation as well, although that would apply to the entire content of the user's session, and the configuration would be different from app server to app server.

[avernet]

Only being able to passivate the whole session, and instead of being able to do it a document-by-document-basis, seems like a step back. Imagine you have fairly active users, that load many forms, over hours or work. In such a scenario, you would keep in memory the dynamic state for all the "old forms" for those users, and would require much more RAM for the same amount of load.

Do containers, say Tomcat, have the ability to only passivate part of the session for a given user?

[ebruchez]

Do containers, say Tomcat, have the ability to only passivate part of the session for a given user?

I doubt it, Tomcat is pretty basic.

I agree that just session passivation would be a step back.

[ebruchez]

Summary so far:

• We need to replicate the servlet session, even if it is empty, to:
  • allow clustering at the servlet container level
  • allow sticky sessions to work at the load balancer level
  • obtain a unique id to access other caches
• We need a separate state cache:
  • so that the static form state, AKA the compiled form definitions, can be cached and shared without being duplicated when being replicated
  • so that the dynamic state can be passivated for a given user when inactive without affecting the entire user's servlet session
• Since we replicate servlet sessions, we can keep storing non-form state items to the regular session, and all state items to the replicated xforms.state cache.
• The replicated xforms.state cache can keep being a disk store only.

[avernet]

Could you detail what you mean by "obtain a unique id to access other caches"?

Regarding "allow sticky sessions to work at the load balancer level", I don't think load balancer need the app server to maintain their own cookie. E.g. HAProxy can keep track of the server for a user by rewriting a given cookie, like JSESSIONID, doing 123 → s1~123 on responses, and s1~123 → 123 on requests, but it can also use its own cookie, e.g. SERVERID=s1.

[ebruchez]

Found out why I get an "unexpected request sequence number" when moving back to server A. It's because server A still has a live document around. It doesn't know that server B processed some requests.

One solution might be, in case of sequence number error, to check if we can find one in the store, as that might have been updated in the background.

[ebruchez]

Terminology from HAProxy doc:

Using persistence, we mean that we’re 100% sure that a user will get redirected to a single server. Using affinity, we mean that the user may be redirected to the same server…

[ebruchez]

Solution for problem above:

• Use Ehcache Element.version to store the sequence number.
• When getting document from XFormsDocumentCache, invalidate found entry if there is a newer sequence number in the store checked simply by looking at the element's version in cache, which does not require extracting the dynamic state to find its key (but does require accessing the Element.

[ebruchez]

So at this point we have shown that replication can work provided:

• we configure servlet container session replication
• we configure Ehcache replication of xforms.state
• we fix a few issues
  • XFormsDocumentCache cache invalidation
  • XFormsServerSharedInstancesCache
• we store DynamicState after every Ajax update

At this point, we don't yet know the performance implications for:

• increased DynamicState creation and serialization
• data transmission of the modified DynamicState to peers
• storage of updates to DynamicState in peers (using a disk cache)

So we would need some numbers.

[ebruchez]

storeDocumentState takes about 5 ms (average of 100 serializations) for form with:

• 14 Sections
• 11 Repeats
• 32 Grids
• 0 Section templates
• 191 Controls
• 248 All

The size is 53,732 bytes, out of which 46,290 are for instances, which breaks down to:

• 18,710 bytes for fr-initial-instance (see #3303)
• 18,715 bytes for fr-form-instance
• the other instances are smaller, at most 1,248 bytes for fr-error-summary-instance

fr-initial-instance is for keeping "initial data for clear button". It never changes after the initial load, so this would be a case where if we split the dynamic state into more than one cache key, we could prevent unneeded serializations and replication.

Editing that same form in Form Builder, storeDocumentState:

• serialization takes about 15 ms within Form Builder
• size is 500,944
• 495,649 in instances
• 472,716 for fb-form-instance

We should maybe consider disabling replication for Form Builder, for most deployment uses, as here replication might be more expensive. Seeing how we could reduce the size of serialized instances would be a good idea too. We should do space compression on instances (see #2751, #1715).

What about gzip compression of serialized data?

[ebruchez]

Dynamic state bytes compresses well by a factor of 7-8. For example:

• 51,004 → 7,186 (5 ms average)
• 501,014 → 62,238 (23 ms average, vs. 20 ms without gzip)

Not sure why I have 20 ms above vs. 15 ms in my previous test the day before!

Q: Should we, and if so when, compress dynamic state when putting it into Ehcache? Is the tradeoff worth it?

[ebruchez]

Property:

<property 
    as="xs:boolean" 
    name="oxf.xforms.replication"         
    value="true"/>

[ebruchez]

One question is: when should the caches be initialized? We would like replication can happen this at the earliest. The OrbeonServletContextListener is a possibility. But this is not XForms-aware. So we should probably introduce a new ServletContextListener for this purpose and add it to web.xml. Suggesting org.orbeon.oxf.xforms.ReplicationServletContextListener.

[ebruchez]

• [x] after each Ajax request, store XForms state if replication is enabled

[ebruchez]

XFormsServerSharedInstancesCache is used by Form Runner and Form Builder to cache resources, in particular form resources. Those loaded from oxf: are no problem, but those loaded with for example fr-get-fr-resources go through a local service.

What we should do is, if the URL is relative to the current host/port/context, store a path to be rewritten instead of the absolute URL. This way, when the path migrates to another server, it will be resolved again but against the current host.

[ebruchez]

Uploads are an issue. If an upload is in progress when a server goes down, then it will return an error, which is probably acceptable in most cases as the user will see the error and restart the upload. At least, I think this is ok in a first implementation.

If a server goes down after an upload has completed, then the form instance data points to a temporary file on disk. Unless the replica servers share the temp filesystem with the original server, the replica will now contain instance data with a temporary file URL which points to a missing file. When saving the data to the database, an error will probably happen.

So what should we do in this case?

1. Is it realistic to replicate uploaded files? They can be arbitrarily large, and we use Ehcache to replicate so we would have to find a way to replicate large amounts of data this way. Will Ehcache scale? On the other hand, if we switched to Ehcache for temporary attachments instead of using temporary files, this might work. But it is clearly more implementation work.
2. Another "solution" is to detect this situation, hoping that it will be rare and that most forms have at most a few attachments. At some point, maybe before saving, attachment paths should be checked. If a file is not present, then the user should be notified and saving should be interrupted.

[ebruchez]

I think that the sponsor for this feature doesn't need attachments at first. But we should probably at least do solution 2 above. We could make this part of require-uploads. This calls pending-uploads (tryPendingUploads). Either tryPendingUploads or another Form Runner action could check the status of unsaved attachment.

If unsaved attachments have missing files:

• reset the instance data (including metadata attributes)
• show an error
  • unclear how the error could be very precise as we would need to pass the file names and control ids maybe
  • as a first step, maybe just show a general error like "Some attachments have been removed due to an application error. Please reattach them."
  • if attachments are required, then saving/sending will fail after that, of course

Another possibility could be to have a standard validation for attachments which checks the existence of the associated file. If the file disappears, the control would show in error. But it could be costly to check all attachments all the time without optimization. With replication, this would only be needed upon state restoration.

[ebruchez]

In #1067 we discussed an xxforms-state-restored event which, possibly, could be also used to check attachments. (#3317)

[ebruchez]

For uploads, I have implemented a simple solution to detect missing attachments and show an error.

[ebruchez]

• [x] Getting this error upon Tomcat session serialization:

 java.io.NotSerializableException: org.orbeon.oxf.xforms.state.XFormsStateManager
    at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1183)
    at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
    at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
    at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
    at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
    at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)
    at org.apache.catalina.ha.session.DeltaRequest$AttributeInfo.writeExternal(DeltaRequest.java:401)
    at org.apache.catalina.ha.session.DeltaRequest.writeExternal(DeltaRequest.java:294)
    at org.apache.catalina.ha.session.DeltaRequest.serialize(DeltaRequest.java:308)
    at org.apache.catalina.ha.session.DeltaManager.serializeDeltaRequest(DeltaManager.java:585)
    at org.apache.catalina.ha.session.DeltaManager.requestCompleted(DeltaManager.java:966)
    at org.apache.catalina.ha.session.DeltaManager.requestCompleted(DeltaManager.java:933)
    at org.apache.catalina.ha.tcp.ReplicationValve.send(ReplicationValve.java:525)
    at org.apache.catalina.ha.tcp.ReplicationValve.sendMessage(ReplicationValve.java:513)
    at org.apache.catalina.ha.tcp.ReplicationValve.sendSessionReplicationMessage(ReplicationValve.java:495)
    at org.apache.catalina.ha.tcp.ReplicationValve.sendReplicationMessage(ReplicationValve.java:406)
    at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:329)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:502)
    at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1132)
    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:684)
    at org.apache.tomcat.util.net.AprEndpoint$SocketWithOptionsProcessor.run(AprEndpoint.java:2458)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:745)

[ebruchez]

New problem: I do manage one move from one server to the other. However if the session is moved back to the first server, I am getting an error:

Unable to retrieve XForms engine state. Unable to process incoming request.

[ebruchez]

It is as if, during cache bootstrap, the content was not obtained. Yet RMICacheManagerPeerListener appears to find its peer.

[ebruchez]

Adding this is necessary:

<bootstrapCacheLoaderFactory
    class="net.sf.ehcache.distribution.RMIBootstrapCacheLoaderFactory"
    properties="bootstrapAsynchronously=false" />

[ebruchez]

It would be great if we could write an automated test for this. This might justify the use of Docker instances. Here is an example test plan:

• start 2 Docker instances with Tomcat: s1 and s2
• start HAProxy
  • Q: Natively or within HAProxy Docker instance?
• run test within JSDOM (see #2743)
• request form page form1 into one context
• when loaded, request form page form2
  • because of round-robin configuration, should hit 2nd server if up
• simulate entering data in both forms form1 and form2
• stop s1 container
• enter data in form1
  • must work
• enter data in form2 (must obviously work)
• start server s1
• wait until s1 is "up" (how?)
• enter data in form1 and form2
• stop server s2
• enter data in form1 and form2
• ...

This can be extended to launching more than 2 instances and more clients.

[ebruchez]

Now we need to figure out how to run this, because there is interleaving of code on the server-side (launching/stopping containers) and code on the client-side (loading pages).

[ebruchez]

So there will be some code, somewhere, running the test above, most likely on the client. If on the client, it will have to remote-control the start/stop of containers.

This calls for having a small server which the client can remote control to start containers. I see these possibilities:

1. A simple servlet, created with an annotation, which is added as JAR/classes to the exploded orbeon-war. This means that at least one container with Orbeon Forms must be started before this works. Also, this requires the entire orbeon-war to be ready.
2. A separate server run strictly for the purpose of tests. This could be based on Undertow, or Docker, but it would be started/stopped once from sbt.

I think that option 2 would be cleaner. It is a single server/remote control which only needs to be started once and is independent from the specific tests that need to run.

[ebruchez]

So we implemented option 2 above for tests, but now thinking that we should use Node.js's child_process directly instead.

[ebruchez]

Current status is that we now have a test able to start/stop containers, with HAProxy, load forms and change state, shut down one server, make sure that forms initially hitting the server down works, and conversely that the state goes back to the original server when it is back up, and then this also works reversed.

We documented a possibility where state might be lost if it hasn't been replicated yet and the client got an Ajax response. Possible solutions:

1. Synchronous cache writes/replication with replicateAsynchronously=false
   • The worry here is that requests will now take longer to complete, possibly > 100 ms.
   • This will block the current thread within the limiter filter, unless we write a lot of code to handle this, as we don't yet have a nice async architecture for this.
2. Fancy state recovery
   • If server gets Ajax request for newer state, somehow rollback the client to the previous state (send original HTML, diff to previous state) and re-apply latest events, which must have been stored somewhere.
   • Not necessarily realistic, as the transition between states might have cause side effects (e.g. "Send" button).

• [x] We could give solution 1 a try. In our test, remove the delay before shutting down the server.

[ebruchez]

With testing locally (Docker images on the same machine), asynchronous state storing takes, from the caller's point of view, in the order of 10 ms. With replicateAsynchronously=false, using Form Builder as "form", this time climbs to 30-40 ms.

[ebruchez]

Documented but could benefit from diagrams.