StorageExceptionDisruption Error

JustinVoitel commented 11 months ago

Environment Details

MicroStream Version: 08.01.01-MS-GA
JDK version: openjdk 21.0.1 2023-10-17
OS: Linux Alpine 3.19.0
Used frameworks: Spring Boot + Vaadin

Describe the bug

We encountered the following exception during Runtime:

Caused by: one.microstream.persistence.exceptions.PersistenceExceptionTransfer
        at one.microstream.storage.embedded.types.EmbeddedStorageBinaryTarget$Default.write(EmbeddedStorageBinaryTarget.java:90) ~[microstream-storage-embedded-08.01.01-MS-GA.jar:?]
        at one.microstream.storage.embedded.types.EmbeddedStorageBinaryTarget$Default.write(EmbeddedStorageBinaryTarget.java:49) ~[microstream-storage-embedded-08.01.01-MS-GA.jar:?]
        at one.microstream.persistence.binary.types.BinaryStorer$Default.commit(BinaryStorer.java:522) ~[microstream-persistence-binary-08.01.01-MS-GA.jar:?]
        at one.microstream.persistence.types.PersistenceManager$Default.store(PersistenceManager.java:305) ~[microstream-persistence-08.01.01-MS-GA.jar:?]
        at one.microstream.storage.types.StorageConnection.store(StorageConnection.java:401) ~[microstream-storage-08.01.01-MS-GA.jar:?]
        at de.app.db.data.UsersQuery.storeCollectionItem(UsersQuery.java:58) ~[classes/:2.1.0-beta.27]
        ... more
Caused by: one.microstream.storage.exceptions.StorageExceptionDisruptingExceptions: Disruptions: {one.microstream.storage.exceptions.StorageExceptionGarbageCollector:Exception in channel #0,}
        at one.microstream.storage.types.StorageOperationController$Default.checkProcessingEnabled(StorageOperationController.java:139) ~[microstream-storage-08.01.01-MS-GA.jar:?]
        at one.microstream.storage.types.StorageTaskBroker$Default.enqueueTask(StorageTaskBroker.java:209) ~[microstream-storage-08.01.01-MS-GA.jar:?]
        at one.microstream.storage.types.StorageTaskBroker$Default.enqueueTask(StorageTaskBroker.java:200) ~[microstream-storage-08.01.01-MS-GA.jar:?]
        at one.microstream.storage.types.StorageTaskBroker$Default.enqueueTaskAndNotifyAll(StorageTaskBroker.java:191) ~[microstream-storage-08.01.01-MS-GA.jar:?]
        at one.microstream.storage.types.StorageTaskBroker$Default.enqueueStoreTask(StorageTaskBroker.java:410) ~[microstream-storage-08.01.01-MS-GA.jar:?]
        at one.microstream.storage.types.StorageRequestAcceptor$Default.storeData(StorageRequestAcceptor.java:180) ~[microstream-storage-08.01.01-MS-GA.jar:?]
        at one.microstream.storage.embedded.types.EmbeddedStorageBinaryTarget$Default.write(EmbeddedStorageBinaryTarget.java:86) ~[microstream-storage-embedded-08.01.01-MS-GA.jar:?]
        at one.microstream.storage.embedded.types.EmbeddedStorageBinaryTarget$Default.write(EmbeddedStorageBinaryTarget.java:49) ~[microstream-storage-embedded-08.01.01-MS-GA.jar:?]
        at one.microstream.persistence.binary.types.BinaryStorer$Default.commit(BinaryStorer.java:522) ~[microstream-persistence-binary-08.01.01-MS-GA.jar:?]
        at one.microstream.persistence.types.PersistenceManager$Default.store(PersistenceManager.java:305) ~[microstream-persistence-08.01.01-MS-GA.jar:?]
        at one.microstream.storage.types.StorageConnection.store(StorageConnection.java:401) ~[microstream-storage-08.01.01-MS-GA.jar:?]
        at de.app.db.data.UsersQuery.storeCollectionItem(UsersQuery.java:58) ~[classes/:2.1.0-beta.27]
    ... more

After restarting the Servlet we can no longer initialize MicroStream: Caused by: org.springframework.context.ApplicationContextException: Could not initiate Microstream properly: Problem in channel #0

Is there a way to figure out:

what exactly happened ?
whether the database can be "repaired" ? Or do we sorely rely on backups in this scenario ?
prevent this Error in the Future ?

To Reproduce

We have no Idea how it happened since it happened in a production database on runtime without further logging that can help us. We have a clue that a bug in the application (and a possible halt to a thead) caused the Garbage Collector to disrupt the DB.

fh-ms commented 11 months ago

We need more information to find out what happened and how to avoid this in the future.

Is there a longer stack trace? Because the underlying error is missing.

Setting the log level to 'debug' should produce a more usable output.

The thread halt could be the culprit if the housekeeping thread was involved. This is where the GC happens.

If you can provide the affected storage and source code, we could investigate on our side.

JustinVoitel commented 10 months ago

@fh-ms I think we know what the problem is. If you look at the error when you restart the Database:

11:17:23 Logging  INFO  -  MicroStream Version 08.01.01-MS-GA
11:17:23 EmbeddedStorageFoundation$Default  INFO  - Creating embedded storage manager
11:17:23 PersistenceTypeHandlerManager$Default  INFO  -  Initializing type handler manager
11:17:23 EmbeddedStorageManager$Default  INFO  -  Starting embedded storage manager
11:17:23 StorageSystem$Default  INFO  - Starting storage system
11:17:23 StorageStructureValidator$Default  INFO  - Storage structure validated successfully.
11:17:23 StorageChannelTask$Abstract  ERROR - Error occurred in storage channel#0
one.microstream.storage.exceptions.StorageExceptionConsistency: 0 Length 5886354 of file 134194 is inconsistent with the transactions entry's length of 5886824
        at one.microstream.storage.types.StorageFileManager$Default.validateStorageDataFilesLength(StorageFileManager.java:781) ~[microstream-storage-08.01.01-MS-GA.jar:?]
        at one.microstream.storage.types.StorageFileManager$Default.initializeStorage(StorageFileManager.java:847) ~[microstream-storage-08.01.01-MS-GA.jar:?]
        at one.microstream.storage.types.StorageChannel$Default.initializeStorage(StorageChannel.java:738) ~[microstream-storage-08.01.01-MS-GA.jar:?]
        at one.microstream.storage.types.StorageChannelTaskInitialize$Default.succeed(StorageChannelTaskInitialize.java:201) ~[microstream-storage-08.01.01-MS-GA.jar:?]
        at one.microstream.storage.types.StorageChannelTaskInitialize$Default.succeed(StorageChannelTaskInitialize.java:36) ~[microstream-storage-08.01.01-MS-GA.jar:?]
        at one.microstream.storage.types.StorageChannelSynchronizingTask$AbstractCompletingTask.synchronizedComplete(StorageChannelSynchronizingTask.java:84) ~[microstream-storage-08.01.01-MS-GA.jar:?]
        at one.microstream.storage.types.StorageChannelSynchronizingTask$AbstractCompletingTask.complete(StorageChannelSynchronizingTask.java:132) ~[microstream-storage-08.01.01-MS-GA.jar:?]
        at one.microstream.storage.types.StorageChannelTask$Abstract.processBy(StorageChannelTask.java:268) ~[microstream-storage-08.01.01-MS-GA.jar:?]
        at one.microstream.storage.types.StorageChannel$Default.work(StorageChannel.java:409) ~[microstream-storage-08.01.01-MS-GA.jar:?]
        at one.microstream.storage.types.StorageChannel$Default.run(StorageChannel.java:492) ~[microstream-storage-08.01.01-MS-GA.jar:?]
        at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
11:17:23 DiStorageService  ERROR - Could not initiate Microstream properly: Problem in channel #0
one.microstream.storage.exceptions.StorageException: Problem in channel #0
        at one.microstream.storage.types.StorageChannelTask$Abstract.checkForProblems(StorageChannelTask.java:114)
        at one.microstream.storage.types.StorageChannelTask$Abstract.waitOnCompletion(StorageChannelTask.java:173)
        at one.microstream.storage.types.StorageSystem$Default.startThreads(StorageSystem.java:336)
        at one.microstream.storage.types.StorageSystem$Default.internalStartUp(StorageSystem.java:516)
        at one.microstream.storage.types.StorageSystem$Default.start(StorageSystem.java:600)
        at one.microstream.storage.types.StorageSystem$Default.start(StorageSystem.java:78)
        at one.microstream.storage.embedded.types.EmbeddedStorageManager$Default.start(EmbeddedStorageManager.java:245)
        at one.microstream.storage.embedded.types.EmbeddedStorageManager$Default.start(EmbeddedStorageManager.java:97)

... it looks like the file size is inconsistent. We are currently testing our testing environment as a Docker container and we copied the production Database onto the staging server which is mounted into the container. Maybe its a permission problem ?

fh-ms commented 10 months ago

One possibility is that the storage files were being updated as you copied them over.

As a quick fix, you can delete the transaction files (transactions_*.sft) in each channel directory and restart the application. These are getting rebuilt automatically.

I don't think that it is a permission problem. If so, an exception would be thrown by the IO layer.

JustinVoitel commented 10 months ago

@fh-ms We used the issueFullBackup method of the storageManager to create the copy, does this make a difference or can it still happen to corrupt the storage ?

This is good to know with the deletion of the files, we will test this out. What are the transactions_*.sft used for ?

JustinVoitel commented 10 months ago

@fh-ms Unfortunately when deleting the transactions files we get the following error after startup:

12:58:21 Logging  INFO - MicroStream Version 08.01.01-MS-GA
12:58:21 EmbeddedStorageFoundation$Default  INFO - Creating embedded storage manager
12:58:21 PersistenceTypeHandlerManager$Default  INFO - Initializing type handler manager
12:58:21 EmbeddedStorageManager$Default  INFO - Starting embedded storage manager
12:58:21 StorageSystem$Default  INFO - Starting storage system
12:58:21 StorageStructureValidator$Default  INFO - Storage structure validated successfully.
12:58:21 StorageChannelTask$Abstract  ERROR - Error occurred in storage channel#0
one.microstream.storage.exceptions.StorageException: TypeId not resolvable via type dictionary: 8444249301319680
        at one.microstream.storage.types.StorageTypeDictionary.lookupTypeHandlerChecked(StorageTypeDictionary.java:53) ~[microstream-storage-08.01.01-MS-GA.jar:?]
        at one.microstream.storage.types.StorageEntityCache$Default.addNewType(StorageEntityCache.java:351) ~[microstream-storage-08.01.01-MS-GA.jar:?]
        at one.microstream.storage.types.StorageEntityCache$Default.getType(StorageEntityCache.java:340) ~[microstream-storage-08.01.01-MS-GA.jar:?]
        at one.microstream.storage.types.StorageEntityCache$Default.initialCreateEntity(StorageEntityCache.java:664) ~[microstream-storage-08.01.01-MS-GA.jar:?]
        at one.microstream.storage.types.StorageEntityInitializer$Default.registerFileEntities(StorageEntityInitializer.java:154) ~[microstream-storage-08.01.01-MS-GA.jar:?]
        at one.microstream.storage.types.StorageEntityInitializer$Default.registerEntities(StorageEntityInitializer.java:115) ~[microstream-storage-08.01.01-MS-GA.jar:?]
        at one.microstream.storage.types.StorageEntityInitializer$Default.registerEntities(StorageEntityInitializer.java:91) ~[microstream-storage-08.01.01-MS-GA.jar:?]
        at one.microstream.storage.types.StorageEntityInitializer$Default.registerEntities(StorageEntityInitializer.java:54) ~[microstream-storage-08.01.01-MS-GA.jar:?]
        at one.microstream.storage.types.StorageFileManager$Default.initializeForExistingFiles(StorageFileManager.java:986) ~[microstream-storage-08.01.01-MS-GA.jar:?]
        at one.microstream.storage.types.StorageFileManager$Default.initializeStorage(StorageFileManager.java:878) ~[microstream-storage-08.01.01-MS-GA.jar:?]
        at one.microstream.storage.types.StorageChannel$Default.initializeStorage(StorageChannel.java:738) ~[microstream-storage-08.01.01-MS-GA.jar:?]
        at one.microstream.storage.types.StorageChannelTaskInitialize$Default.succeed(StorageChannelTaskInitialize.java:201) ~[microstream-storage-08.01.01-MS-GA.jar:?]
        at one.microstream.storage.types.StorageChannelTaskInitialize$Default.succeed(StorageChannelTaskInitialize.java:36) ~[microstream-storage-08.01.01-MS-GA.jar:?]
        at one.microstream.storage.types.StorageChannelSynchronizingTask$AbstractCompletingTask.synchronizedComplete(StorageChannelSynchronizingTask.java:84) ~[microstream-storage-08.01.01-MS-GA.jar:?]
        at one.microstream.storage.types.StorageChannelSynchronizingTask$AbstractCompletingTask.complete(StorageChannelSynchronizingTask.java:132) ~[microstream-storage-08.01.01-MS-GA.jar:?]
        at one.microstream.storage.types.StorageChannelTask$Abstract.processBy(StorageChannelTask.java:268) ~[microstream-storage-08.01.01-MS-GA.jar:?]
        at one.microstream.storage.types.StorageChannel$Default.work(StorageChannel.java:409) ~[microstream-storage-08.01.01-MS-GA.jar:?]
        at one.microstream.storage.types.StorageChannel$Default.run(StorageChannel.java:492) ~[microstream-storage-08.01.01-MS-GA.jar:?]
        at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
12:58:21 StorageChannelTask$Abstract  ERROR - Error occurred in storage channel#2
one.microstream.storage.exceptions.StorageException: TypeId not resolvable via type dictionary: 1966080
        at one.microstream.storage.types.StorageTypeDictionary.lookupTypeHandlerChecked(StorageTypeDictionary.java:53) ~[microstream-storage-08.01.01-MS-GA.jar:?]
        at one.microstream.storage.types.StorageEntityCache$Default.addNewType(StorageEntityCache.java:351) ~[microstream-storage-08.01.01-MS-GA.jar:?]
        at one.microstream.storage.types.StorageEntityCache$Default.getType(StorageEntityCache.java:340) ~[microstream-storage-08.01.01-MS-GA.jar:?]
        at one.microstream.storage.types.StorageEntityCache$Default.initialCreateEntity(StorageEntityCache.java:664) ~[microstream-storage-08.01.01-MS-GA.jar:?]
        at one.microstream.storage.types.StorageEntityInitializer$Default.registerFileEntities(StorageEntityInitializer.java:154) ~[microstream-storage-08.01.01-MS-GA.jar:?]
        at one.microstream.storage.types.StorageEntityInitializer$Default.registerEntities(StorageEntityInitializer.java:115) ~[microstream-storage-08.01.01-MS-GA.jar:?]
        at one.microstream.storage.types.StorageEntityInitializer$Default.registerEntities(StorageEntityInitializer.java:91) ~[microstream-storage-08.01.01-MS-GA.jar:?]
        at one.microstream.storage.types.StorageEntityInitializer$Default.registerEntities(StorageEntityInitializer.java:54) ~[microstream-storage-08.01.01-MS-GA.jar:?]
        at one.microstream.storage.types.StorageFileManager$Default.initializeForExistingFiles(StorageFileManager.java:986) ~[microstream-storage-08.01.01-MS-GA.jar:?]
        at one.microstream.storage.types.StorageFileManager$Default.initializeStorage(StorageFileManager.java:878) ~[microstream-storage-08.01.01-MS-GA.jar:?]
        at one.microstream.storage.types.StorageChannel$Default.initializeStorage(StorageChannel.java:738) ~[microstream-storage-08.01.01-MS-GA.jar:?]
        at one.microstream.storage.types.StorageChannelTaskInitialize$Default.succeed(StorageChannelTaskInitialize.java:201) ~[microstream-storage-08.01.01-MS-GA.jar:?]
        at one.microstream.storage.types.StorageChannelTaskInitialize$Default.succeed(StorageChannelTaskInitialize.java:36) ~[microstream-storage-08.01.01-MS-GA.jar:?]
        at one.microstream.storage.types.StorageChannelSynchronizingTask$AbstractCompletingTask.synchronizedComplete(StorageChannelSynchronizingTask.java:84) ~[microstream-storage-08.01.01-MS-GA.jar:?]
        at one.microstream.storage.types.StorageChannelSynchronizingTask$AbstractCompletingTask.complete(StorageChannelSynchronizingTask.java:132) ~[microstream-storage-08.01.01-MS-GA.jar:?]
        at one.microstream.storage.types.StorageChannelTask$Abstract.processBy(StorageChannelTask.java:268) ~[microstream-storage-08.01.01-MS-GA.jar:?]
        at one.microstream.storage.types.StorageChannel$Default.work(StorageChannel.java:409) ~[microstream-storage-08.01.01-MS-GA.jar:?]
        at one.microstream.storage.types.StorageChannel$Default.run(StorageChannel.java:492) ~[microstream-storage-08.01.01-MS-GA.jar:?]
        at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]

hg-ms commented 10 months ago

Hello,

I had a look at the problem too. From my point of view the root cause for the corrupted files is the first exception (StorageExceptionGarbageCollector). Unfortunately, the StorageExceptionDisruptingExceptions class only prints the message, but not the cause. So is not possible to see what really happened, except that the channel 0 tread has been interrupted by some error in the StorageGC during a store operation. All other exceptions after restart are just consequences of this incomplete IO.

Created #687 to track the StorageExceptionDisruptingExceptions issue.

JustinVoitel commented 10 months ago

@hg-ms Unfortunately we were not able to recreate the StorageExceptionDisruption Error but we have some other information that might be relevant:

We have limited access to the Prod and Staging Servers due to the high security circumstances of our client
The production DB we want to copy is approximately 4gb big
The production App on Server A is still running on a local tomcat (no Docker involved yet) using Java 17
The staging App on Server B is running in a Docker container (Alpine image) using Java 21 where the MS storage is mounted into the container using a Bind mount (e.g -v /webapps/app1/data:/data)
We tried using multiple exports (copies) of the DB even when we were sure that no writing actions happened
We are always able to recreate the StorageExceptionConsistency error, we procees as follows:
1. we copy a freshly exported DB from Server A to Server B using scp
2. We start the docker container and it starts successfully -> reading and writing operations are working as expected
3. Now when restarting the docker container, the error happens: StorageExceptionConsistency.log
4. When we now delete the transactions files as mentioned by @fh-ms we get the following error: after restart StorageException.log

We really hope there is a solution to this Problem.

hg-ms commented 10 months ago

Can you reproduce the problem without docker?

It’s interesting that the error occurs after restarting the docker container. This indicates that the copy was ok at the first start. May it be possible that the docker restart causes in incomplete write of storage files?

The StorageExceptionConsistency: 2 Length 8388092 of file 116011 is inconsistent with the transactions entry's length of 8388596 also indicates that a storage file has not been completely written.

The exception says that file No. 116011 of channel 2 is shorter then expected. The files size is 8388092 Bytes, but the transaction log says that it should be 8388596 Bytes.

The Transaction Log is always written after a storage file IO has succeeded.

JustinVoitel commented 10 months ago

@hg-ms it seems like when running docker stop app its not able to shutdown correctly:

12-Jan-2024 10:22:40.433 INFORMATION [Thread-3] org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler ["http-nio-8080"]
12-Jan-2024 10:22:40.435 INFORMATION [Thread-3] org.apache.catalina.core.StandardService.stopInternal Stopping service [Catalina]
10:22:40 DiAppInitListener  INFO  - DESTROY SERVICE
10:22:40 ForkJoinPool  INFO  - Using ForkJoinPool  java.util.concurrent.ForkJoinPool. Set the org.atmosphere.cpr.broadcaster.maxAsyncWriteThreads to -1 to fully use its power.
10:22:40 DefaultBroadcaster  WARN  - AtmosphereResource 4b36c887-f3da-409e-a097-af0e352aa88e is not suspended. If cached messages exists, this may cause unexpected situation. Suspend first
10:22:40 DiAppInitListener  INFO  - DESTROY SESSION
10:22:40 QuartzScheduler  INFO  - Scheduler DefaultQuartzScheduler_$_NON_CLUSTERED shutting down.
10:22:40 QuartzScheduler  INFO  - Scheduler DefaultQuartzScheduler_$_NON_CLUSTERED paused.
10:22:40 QuartzScheduler  INFO  - Scheduler DefaultQuartzScheduler_$_NON_CLUSTERED shutdown complete.
10:22:40 StorageSystem$Default  INFO  - Stopping storage system
10:22:40 StorageSystem$Default  INFO  - Storage system stopped
10:22:40 DiStorageService  INFO  - Shutting down MicroStream StorageManager
12-Jan-2024 10:22:40.507 INFORMATION [Thread-3] org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean.destroy Closing JPA EntityManagerFactory for persistence unit 'default'
12-Jan-2024 10:22:40.513 WARNUNG [Thread-3] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [App] appears to have started a thread named [MicroStream-StorageChannel-0] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base/java.lang.Object.wait0(Native Method)
 java.base/java.lang.Object.wait(Object.java:366)
 one.microstream.storage.types.StorageTask$Abstract.awaitNext(StorageTask.java:86)
 one.microstream.storage.types.StorageChannel$Default.work(StorageChannel.java:450)
 one.microstream.storage.types.StorageChannel$Default.run(StorageChannel.java:492)
 java.base/java.lang.Thread.run(Thread.java:1583)
12-Jan-2024 10:22:40.513 WARNUNG [Thread-3] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [App] appears to have started a thread named [MicroStream-StorageChannel-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base/java.lang.Object.wait0(Native Method)
 java.base/java.lang.Object.wait(Object.java:366)
 one.microstream.storage.types.StorageTask$Abstract.awaitNext(StorageTask.java:86)
 one.microstream.storage.types.StorageChannel$Default.work(StorageChannel.java:450)
 one.microstream.storage.types.StorageChannel$Default.run(StorageChannel.java:492)
 java.base/java.lang.Thread.run(Thread.java:1583)
12-Jan-2024 10:22:40.513 WARNUNG [Thread-3] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [App] appears to have started a thread named [MicroStream-StorageChannel-2] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base/java.lang.Object.wait0(Native Method)
 java.base/java.lang.Object.wait(Object.java:366)
 one.microstream.storage.types.StorageTask$Abstract.awaitNext(StorageTask.java:86)
 one.microstream.storage.types.StorageChannel$Default.work(StorageChannel.java:450)
 one.microstream.storage.types.StorageChannel$Default.run(StorageChannel.java:492)
 java.base/java.lang.Thread.run(Thread.java:1583)
12-Jan-2024 10:22:40.513 WARNUNG [Thread-3] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [App] appears to have started a thread named [MicroStream-StorageChannel-3] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base/java.lang.Object.wait0(Native Method)
 java.base/java.lang.Object.wait(Object.java:366)
 one.microstream.storage.types.StorageTask$Abstract.awaitNext(StorageTask.java:86)
 one.microstream.storage.types.StorageChannel$Default.work(StorageChannel.java:450)
 one.microstream.storage.types.StorageChannel$Default.run(StorageChannel.java:492)
 java.base/java.lang.Thread.run(Thread.java:1583)
12-Jan-2024 10:22:40.513 WARNUNG [Thread-3] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [App] appears to have started a thread named [DefaultQuartzScheduler_Worker-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base/java.lang.Object.wait0(Native Method)
 java.base/java.lang.Object.wait(Object.java:366)
 org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)
12-Jan-2024 10:22:40.513 WARNUNG [Thread-3] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [App] appears to have started a thread named [DefaultQuartzScheduler_Worker-2] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base/java.lang.Object.wait0(Native Method)
 java.base/java.lang.Object.wait(Object.java:366)
 org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)
12-Jan-2024 10:22:40.513 WARNUNG [Thread-3] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [App] appears to have started a thread named [DefaultQuartzScheduler_Worker-3] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base/java.lang.Object.wait0(Native Method)
 java.base/java.lang.Object.wait(Object.java:366)
 org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)
12-Jan-2024 10:22:40.513 WARNUNG [Thread-3] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [App] appears to have started a thread named [DefaultQuartzScheduler_Worker-4] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base/java.lang.Object.wait0(Native Method)
 java.base/java.lang.Object.wait(Object.java:366)
 org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)
12-Jan-2024 10:22:40.514 WARNUNG [Thread-3] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [App] appears to have started a thread named [DefaultQuartzScheduler_Worker-5] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base/java.lang.Object.wait0(Native Method)
 java.base/java.lang.Object.wait(Object.java:366)
 org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)
12-Jan-2024 10:22:40.514 WARNUNG [Thread-3] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [App] appears to have started a thread named [DefaultQuartzScheduler_Worker-6] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base/java.lang.Object.wait0(Native Method)
 java.base/java.lang.Object.wait(Object.java:366)
 org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)
12-Jan-2024 10:22:40.514 WARNUNG [Thread-3] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [App] appears to have started a thread named [DefaultQuartzScheduler_Worker-7] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base/java.lang.Object.wait0(Native Method)
 java.base/java.lang.Object.wait(Object.java:366)
 org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)
12-Jan-2024 10:22:40.514 WARNUNG [Thread-3] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [App] appears to have started a thread named [DefaultQuartzScheduler_Worker-8] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base/java.lang.Object.wait0(Native Method)
 java.base/java.lang.Object.wait(Object.java:366)
 org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)
12-Jan-2024 10:22:40.514 WARNUNG [Thread-3] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [App] appears to have started a thread named [DefaultQuartzScheduler_Worker-9] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base/java.lang.Object.wait0(Native Method)
 java.base/java.lang.Object.wait(Object.java:366)
 org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)
12-Jan-2024 10:22:40.515 WARNUNG [Thread-3] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [App] appears to have started a thread named [DefaultQuartzScheduler_Worker-10] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base/java.lang.Object.wait0(Native Method)
 java.base/java.lang.Object.wait(Object.java:366)
 org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)
12-Jan-2024 10:22:40.515 WARNUNG [Thread-3] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [App] appears to have started a thread named [Atmosphere-Scheduler-0] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base/jdk.internal.misc.Unsafe.park(Native Method)
 java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:269)
 java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1758)
 java.base/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.poll(ScheduledThreadPoolExecutor.java:1223)
 java.base/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.poll(ScheduledThreadPoolExecutor.java:899)
 java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069)
 java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
 java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
 java.base/java.lang.Thread.run(Thread.java:1583)
12-Jan-2024 10:22:40.515 WARNUNG [Thread-3] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [App] appears to have started a thread named [Atmosphere-Shared-0] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base/jdk.internal.misc.Unsafe.park(Native Method)
 java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:269)
 java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1758)
 java.base/java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:460)
 org.atmosphere.cpr.DefaultBroadcaster$1.run(DefaultBroadcaster.java:405)
 java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
 java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
 java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1423)
 java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:387)
 java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1312)
 java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1843)
 java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1808)
 java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:188)
12-Jan-2024 10:22:40.515 WARNUNG [Thread-3] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [App] appears to have started a thread named [Atmosphere-Scheduler-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base/jdk.internal.misc.Unsafe.park(Native Method)
 java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:269)
 java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1758)
 java.base/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.poll(ScheduledThreadPoolExecutor.java:1218)
 java.base/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.poll(ScheduledThreadPoolExecutor.java:899)
 java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069)
 java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
 java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
 java.base/java.lang.Thread.run(Thread.java:1583)
12-Jan-2024 10:22:40.523 INFORMATION [Thread-3] org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler ["http-nio-8080"]
12-Jan-2024 10:22:40.530 INFORMATION [Thread-3] org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler ["http-nio-8080"]

The same happens on my local tomcat server. We initiate and destroy MS similar to the Bookstore Demo as a Spring Bean:

  @Bean(destroyMethod = "shutdown")
   public DiStorageService createStorageService() {
      return new StorageService();
   }

Do you have any idea why this is happening ?

microstream-one / microstream