Serialization Errors Cause Flaky UI Behaviour

[MichaelPluessErni]

Error description:

When an error occurs during session serialization, the UI often displays "Offline" / "Online" notifications on the client side that are not related to the actual state of the connection to the server side.

It is possible to use the UI, but sometimes certain operations randomly do not work as though the session was not available, even though it should be.

Expected behaviour:

Errors during session serialization should not impair users using the app. Session serialization errors should not cause other errors in the app.

Details:

We are using Spring Boot + Redis to store the serialized sessions. There was an error in Redis where the key expiration was not configured correctly and an Out-Of-Memory error was caused. This led to the following exception in our Vaadin application:

Servlet.service() for servlet [springServlet] in context with path [] threw exception
org.springframework.data.redis.RedisSystemException: Error in execution
    at org.springframework.data.redis.connection.lettuce.LettuceExceptionConverter.convert(LettuceExceptionConverter.java:52)
    at org.springframework.data.redis.connection.lettuce.LettuceExceptionConverter.convert(LettuceExceptionConverter.java:50)
    at org.springframework.data.redis.connection.lettuce.LettuceExceptionConverter.convert(LettuceExceptionConverter.java:41)
    at org.springframework.data.redis.PassThroughExceptionTranslationStrategy.translate(PassThroughExceptionTranslationStrategy.java:40)
    at org.springframework.data.redis.FallbackExceptionTranslationStrategy.translate(FallbackExceptionTranslationStrategy.java:38)
    at org.springframework.data.redis.connection.lettuce.LettuceConnection.convertLettuceAccessException(LettuceConnection.java:306)
    at org.springframework.data.redis.connection.lettuce.LettuceConnection.await(LettuceConnection.java:1010)
    at org.springframework.data.redis.connection.lettuce.LettuceConnection.lambda$doInvoke$3(LettuceConnection.java:443)
    at org.springframework.data.redis.connection.lettuce.LettuceInvoker$Synchronizer.invoke(LettuceInvoker.java:673)
    at org.springframework.data.redis.connection.lettuce.LettuceInvoker$DefaultSingleInvocationSpec.get(LettuceInvoker.java:589)
    at org.springframework.data.redis.connection.lettuce.LettuceStringCommands.set(LettuceStringCommands.java:102)
    at org.springframework.data.redis.connection.DefaultedRedisConnection.set(DefaultedRedisConnection.java:319)
    at com.vaadin.kubernetes.starter.sessiontracker.backend.RedisConnector.markSerializationStarted(RedisConnector.java:76)
    at com.vaadin.kubernetes.starter.sessiontracker.SessionSerializer.queueSerialization(SessionSerializer.java:225)
    at com.vaadin.kubernetes.starter.sessiontracker.SessionSerializer.serialize(SessionSerializer.java:185)
    at com.vaadin.kubernetes.starter.sessiontracker.SessionSerializer.serialize(SessionSerializer.java:171)
    at com.vaadin.kubernetes.starter.sessiontracker.SessionTrackerFilter.doFilter(SessionTrackerFilter.java:78)
    at jakarta.servlet.http.HttpFilter.doFilter(HttpFilter.java:53)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
[...]
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63)
    at java.base/java.lang.Thread.run(Unknown Source)
Caused by: io.lettuce.core.RedisCommandExecutionException: OOM command not allowed when used memory > 'maxmemory'.
    at io.lettuce.core.internal.ExceptionFactory.createExecutionException(ExceptionFactory.java:147)
[...]
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    ... 1 common frames omitted

Which caused this follow-up error:

Servlet.service() for servlet [springServlet] in context with path [] threw exception
org.springframework.dao.QueryTimeoutException: Redis command timed out
    at org.springframework.data.redis.connection.lettuce.LettuceExceptionConverter.convert(LettuceExceptionConverter.java:68)
    at org.springframework.data.redis.connection.lettuce.LettuceExceptionConverter.convert(LettuceExceptionConverter.java:41)
    at org.springframework.data.redis.PassThroughExceptionTranslationStrategy.translate(PassThroughExceptionTranslationStrategy.java:40)
    at org.springframework.data.redis.FallbackExceptionTranslationStrategy.translate(FallbackExceptionTranslationStrategy.java:38)
    at org.springframework.data.redis.connection.lettuce.LettuceConnection.convertLettuceAccessException(LettuceConnection.java:306)
    at org.springframework.data.redis.connection.lettuce.LettuceConnection.await(LettuceConnection.java:1010)
    at org.springframework.data.redis.connection.lettuce.LettuceConnection.lambda$doInvoke$3(LettuceConnection.java:443)
    at org.springframework.data.redis.connection.lettuce.LettuceInvoker$Synchronizer.invoke(LettuceInvoker.java:673)
    at org.springframework.data.redis.connection.lettuce.LettuceInvoker$DefaultSingleInvocationSpec.get(LettuceInvoker.java:589)
    at org.springframework.data.redis.connection.lettuce.LettuceStringCommands.set(LettuceStringCommands.java:102)
    at org.springframework.data.redis.connection.DefaultedRedisConnection.set(DefaultedRedisConnection.java:319)
    at com.vaadin.kubernetes.starter.sessiontracker.backend.RedisConnector.markSerializationStarted(RedisConnector.java:76)
    at com.vaadin.kubernetes.starter.sessiontracker.SessionSerializer.queueSerialization(SessionSerializer.java:225)
    at com.vaadin.kubernetes.starter.sessiontracker.SessionSerializer.serialize(SessionSerializer.java:185)
    at com.vaadin.kubernetes.starter.sessiontracker.SessionSerializer.serialize(SessionSerializer.java:171)
    at com.vaadin.kubernetes.starter.sessiontracker.SessionTrackerFilter.doFilter(SessionTrackerFilter.java:78)
    at jakarta.servlet.http.HttpFilter.doFilter(HttpFilter.java:53)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
[...]
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63)
    at java.base/java.lang.Thread.run(Unknown Source)
Caused by: io.lettuce.core.RedisCommandTimeoutException: Command timed out after 5 second(s)
    at io.lettuce.core.internal.ExceptionFactory.createTimeoutException(ExceptionFactory.java:59)
    at io.lettuce.core.internal.Futures.awaitOrCancel(Futures.java:246)
    at io.lettuce.core.LettuceFutures.awaitOrCancel(LettuceFutures.java:74)
    at org.springframework.data.redis.connection.lettuce.LettuceConnection.await(LettuceConnection.java:1008)
    ... 152 common frames omitted

The moment when the Redis memory was filled with keys:

During the time it took us to solve the Redis error, the Vaadin application UI behaved very strangely: It was constantly showing "Offline" / "Online" notifications. We were suspecting that the session serialization interfered with the Vaadin push system.

[mcollovati]

The stack trace indicates that the error happened when SessionSerializer tried to mark serialization started on the serialization backend. This operation happens in the request thread or background thread that is pushing changes, making the action fail.

@heruan we should probably handle errors when on backendConnector.markSerializationStarted(clusterKey), postponing the serialization to the next request. An alternative could be to reschedule the serialization after some delay in case of error, but this would require additional effort to detect and discard stale operation, e.g. when a new request successfully completes the serialization process before the postponed task is executed.