Open useryq8 opened 1 year ago
Hello, The exception is most likely caused by the lazy references in the source storage. Lazy references are tricky to copy from one storage to another as they are bound to a storage. To do so they must be loaded, and you need to unlink (one.microstream.reference.Lazy.Default.$unlink()) the lazy references from the source storage.
To reduce the size of a storage you can enforce cleanup by manually calling issueFullCacheCheck()
and issueFullFileCheck()
from the storage manager.
In case you want create a “cleaned up” copy of a storage you may use the StorageConverter to copy a non running storage. This will not copy unreferenced (deleted) data.
Environment Details
Describe the bug
the store performance become being slow when the storage size is about 1.2G, after cleaning some data , the size does not has any change. so I stop my app, and write a new tools, which read the existing storage the write to new storageManager, after copying , then validate it , but when I open the new storageManager, it reports the following error
one.microstream.storage.exceptions.StorageExceptionConsistency: No entity found for objectId 1000000000001447523
To Reproduce
copy the existing microstream storageManager data to a new one storageManager, then open it and validate it
my core code is like this
Expected behavior
when validating , it works well ,it can read data from new storageManager
Screenshots
If applicable, add screenshots to help explain your problem.
Additional context
2023-11-02 13:39:42,322 ERROR a.m.osmdb.cellconn.base.PListContainer.readOp [pool-1-thread-5] Error while reading value, errMsg:Problem in channel #0 one.microstream.storage.exceptions.StorageException: Problem in channel #0 at one.microstream.storage.types.StorageChannelTask$Abstract.checkForProblems(StorageChannelTask.java:114) at one.microstream.storage.types.StorageChannelTask$Abstract.waitOnCompletion(StorageChannelTask.java:176) at one.microstream.storage.types.StorageRequestAcceptor$Default.waitOnTask(StorageRequestAcceptor.java:162) at one.microstream.storage.types.StorageRequestAcceptor$Default.queryByObjectIds(StorageRequestAcceptor.java:191) at one.microstream.storage.embedded.types.EmbeddedStorageBinarySource$Default.readByObjectIds(EmbeddedStorageBinarySource.java:104) at one.microstream.persistence.binary.types.BinaryLoader$Default.readLoadOidData(BinaryLoader.java:790) at one.microstream.persistence.binary.types.BinaryLoader$Default.getObject(BinaryLoader.java:858) at one.microstream.persistence.types.PersistenceManager$Default.getObject(PersistenceManager.java:417) at one.microstream.storage.types.StorageConnection.getObject(StorageConnection.java:455) at one.microstream.storage.types.Database$Default.getObject(Database.java:165) at one.microstream.reference.Lazy$Default.load(Lazy.java:527) at one.microstream.reference.Lazy$Default.get(Lazy.java:508) at one.microstream.collections.lazy.LazyArrayList$Segment.getData(LazyArrayList.java:1223) at one.microstream.collections.lazy.LazyArrayList.get(LazyArrayList.java:260) at java.base/java.util.AbstractList$Itr.next(AbstractList.java:371) at xxx.cellconn.CopyMigrationMetaListValidator.lambda$checkBase$7(CopyMigrationMetaListValidator.java:155) at xxx.cellconn.base.PListContainer.readOp(PListContainer.java:67) at xxx.cellconn.CopyMigrationMetaListValidator.lambda$checkBase$8(CopyMigrationMetaListValidator.java:165) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:833) Caused by: one.microstream.storage.exceptions.StorageExceptionConsistency: No entity found for objectId 1000000000001447523 at one.microstream.storage.types.StorageChannel$EntityCollectorByOid.accept(StorageChannel.java:826) at one.microstream.persistence.binary.types.LoadItemsChain$ChannelHashing$ChainItemObjectIdSet.iterate(LoadItemsChain.java:330) at one.microstream.storage.types.StorageChannel$Default.collectLoadByOids(StorageChannel.java:591) at one.microstream.storage.types.StorageRequestTaskLoadByOids$Default.internalProcessBy(StorageRequestTaskLoadByOids.java:58) at one.microstream.storage.types.StorageRequestTaskLoadByOids$Default.internalProcessBy(StorageRequestTaskLoadByOids.java:28) at one.microstream.storage.types.StorageChannelTask$Abstract.processBy(StorageChannelTask.java:252) at one.microstream.storage.types.StorageChannel$Default.work(StorageChannel.java:409) at one.microstream.storage.types.StorageChannel$Default.run(StorageChannel.java:492) ... 1 common frames omitted