search

eclipse-store / store

High-Performance Java-Native-Persistence. Store and load any Java Object Graph or Subgraphs partially, Relieved of Heavy-weight JPA. Microsecond Response Time. Ultra-High Throughput. Minimum of Latencies. Create Ultra-Fast In-Memory Database Applications & Microservices.
https://eclipsestore.io/
Eclipse Public License 2.0
196 stars 17 forks source link

Storage file size exceeds Java technical IO limitations #321

Open Bios-Marcel opened 1 week ago

Bios-Marcel commented 1 week ago

Similar to #319 I am getting an issue on start-up. The thing that wonders me, is that EclipseStore happily writes to the file and "breaks" the database, resulting in not being able to access the data anymore. What are the steps to recover or avoid this altogether.

i assume this happened because I wrote one giant transaction (store and commit)?

I am getting the following issue on startup now:

Caused by: org.eclipse.store.storage.exceptions.StorageException: Problem in channel #0
    at org.eclipse.store.storage.types.StorageChannelTask$Abstract.checkForProblems(StorageChannelTask.java:106) ~[storage-2.0.0.jar:?]
    at org.eclipse.store.storage.types.StorageChannelTask$Abstract.waitOnCompletion(StorageChannelTask.java:168) ~[storage-2.0.0.jar:?]
    at org.eclipse.store.storage.types.StorageSystem$Default.startThreads(StorageSystem.java:332) ~[storage-2.0.0.jar:?]
    at org.eclipse.store.storage.types.StorageSystem$Default.internalStartUp(StorageSystem.java:520) ~[storage-2.0.0.jar:?]
    at org.eclipse.store.storage.types.StorageSystem$Default.start(StorageSystem.java:608) ~[storage-2.0.0.jar:?]
    at org.eclipse.store.storage.types.StorageSystem$Default.start(StorageSystem.java:71) ~[storage-2.0.0.jar:?]
    at org.eclipse.store.storage.embedded.types.EmbeddedStorageManager$Default.start(EmbeddedStorageManager.java:250) ~[storage-embedded-2.0.0.jar:?]
    at org.eclipse.store.storage.embedded.types.EmbeddedStorageManager$Default.start(EmbeddedStorageManager.java:96) ~[storage-embedded-2.0.0.jar:?]
    at de.ispinsoft.kkg.server.persistence.StorageImpl.<init>(StorageImpl.java:47) ~[classes/:?]
    at de.ispinsoft.kkg.server.persistence.StorageConfiguration.storageImpl(StorageConfiguration.java:61) ~[classes/:?]
    at de.ispinsoft.kkg.server.persistence.StorageConfiguration$$SpringCGLIB$$0.CGLIB$storageImpl$1(<generated>) ~[classes/:?]
    at de.ispinsoft.kkg.server.persistence.StorageConfiguration$$SpringCGLIB$$FastClass$$1.invoke(<generated>) ~[classes/:?]
    at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:258) ~[spring-core-6.1.6.jar:6.1.6]
    at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:331) ~[spring-context-6.1.6.jar:6.1.6]
    at de.ispinsoft.kkg.server.persistence.StorageConfiguration$$SpringCGLIB$$0.storageImpl(<generated>) ~[classes/:?]
    at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) ~[?:?]
    at java.base/java.lang.reflect.Method.invoke(Method.java:580) ~[?:?]
    at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:140) ~[spring-beans-6.1.6.jar:6.1.6]
    at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:644) ~[spring-beans-6.1.6.jar:6.1.6]
    at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:636) ~[spring-beans-6.1.6.jar:6.1.6]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1335) ~[spring-beans-6.1.6.jar:6.1.6]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1165) ~[spring-beans-6.1.6.jar:6.1.6]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:562) ~[spring-beans-6.1.6.jar:6.1.6]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:522) ~[spring-beans-6.1.6.jar:6.1.6]
    at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:326) ~[spring-beans-6.1.6.jar:6.1.6]
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) ~[spring-beans-6.1.6.jar:6.1.6]
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:324) ~[spring-beans-6.1.6.jar:6.1.6]
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:225) ~[spring-beans-6.1.6.jar:6.1.6]
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveNamedBean(DefaultListableBeanFactory.java:1323) ~[spring-beans-6.1.6.jar:6.1.6]
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveNamedBean(DefaultListableBeanFactory.java:1284) ~[spring-beans-6.1.6.jar:6.1.6]
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveBean(DefaultListableBeanFactory.java:486) ~[spring-beans-6.1.6.jar:6.1.6]
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:341) ~[spring-beans-6.1.6.jar:6.1.6]
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:334) ~[spring-beans-6.1.6.jar:6.1.6]
    at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1252) ~[spring-context-6.1.6.jar:6.1.6]
    at de.ispinsoft.kkg.server.KkgServer.start(KkgServer.java:196) ~[classes/:?]
    at de.ispinsoft.kkg.server.KkgServer.<init>(KkgServer.java:174) ~[classes/:?]
    at de.ispinsoft.kkg.server.KkgServer$$SpringCGLIB$$0.<init>(<generated>) ~[classes/:?]
    at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62) ~[?:?]
    at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) ~[?:?]
    at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486) ~[?:?]
    at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:208) ~[spring-beans-6.1.6.jar:6.1.6]
    at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:111) ~[spring-beans-6.1.6.jar:6.1.6]
    at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:315) ~[spring-beans-6.1.6.jar:6.1.6]
    ... 18 more
Caused by: org.eclipse.store.storage.exceptions.StorageExceptionIoReading: Storage file size exceeds Java technical IO limitations: storage$channel_0$channel_0_2.dat
    at org.eclipse.store.storage.types.StorageEntityInitializer$Default.determineLargestFileSize(StorageEntityInitializer.java:304) ~[storage-2.0.0.jar:?]
    at org.eclipse.store.storage.types.StorageEntityInitializer$Default.allocateInitializationBuffer(StorageEntityInitializer.java:253) ~[storage-2.0.0.jar:?]
    at org.eclipse.store.storage.types.StorageEntityInitializer$Default.registerEntities(StorageEntityInitializer.java:95) ~[storage-2.0.0.jar:?]
    at org.eclipse.store.storage.types.StorageEntityInitializer$Default.registerEntities(StorageEntityInitializer.java:85) ~[storage-2.0.0.jar:?]
    at org.eclipse.store.storage.types.StorageEntityInitializer$Default.registerEntities(StorageEntityInitializer.java:48) ~[storage-2.0.0.jar:?]
    at org.eclipse.store.storage.types.StorageFileManager$Default.initializeForExistingFiles(StorageFileManager.java:1002) ~[storage-2.0.0.jar:?]
    at org.eclipse.store.storage.types.StorageFileManager$Default.initializeStorage(StorageFileManager.java:886) ~[storage-2.0.0.jar:?]
    at org.eclipse.store.storage.types.StorageChannel$Default.initializeStorage(StorageChannel.java:782) ~[storage-2.0.0.jar:?]
    at org.eclipse.store.storage.types.StorageChannelTaskInitialize$Default.succeed(StorageChannelTaskInitialize.java:200) ~[storage-2.0.0.jar:?]
    at org.eclipse.store.storage.types.StorageChannelTaskInitialize$Default.succeed(StorageChannelTaskInitialize.java:34) ~[storage-2.0.0.jar:?]
    at org.eclipse.store.storage.types.StorageChannelSynchronizingTask$AbstractCompletingTask.synchronizedComplete(StorageChannelSynchronizingTask.java:78) ~[storage-2.0.0.jar:?]
    at org.eclipse.store.storage.types.StorageChannelSynchronizingTask$AbstractCompletingTask.complete(StorageChannelSynchronizingTask.java:126) ~[storage-2.0.0.jar:?]
    at org.eclipse.store.storage.types.StorageChannelTask$Abstract.processBy(StorageChannelTask.java:260) ~[storage-2.0.0.jar:?]
    at org.eclipse.store.storage.types.StorageChannel$Default.work(StorageChannel.java:453) ~[storage-2.0.0.jar:?]
    at org.eclipse.store.storage.types.StorageChannel$Default.run(StorageChannel.java:536) ~[storage-2.0.0.jar:?]
    at java.base/java.lang.Thread.run(Thread.java:1583) ~[?:?]

Is this even still a real limitation?

Bios-Marcel commented 1 week ago

Given https://github.com/microstream-one/microstream/issues/611 is still open, I assume this is the same bug?