microstream-one / microstream

High-Performance Java-Native-Persistence. Store and load any Java Object Graph or Subgraphs partially, Relieved of Heavy-weight JPA. Microsecond Response Time. Ultra-High Throughput. Minimum of Latencies. Create Ultra-Fast In-Memory Database Applications & Microservices.
https://microstream.one/
Eclipse Public License 2.0
560 stars 44 forks source link

microstream reports one.microstream.storage.exceptions.StorageExceptionConsistency: No entity found for objectId 1000000000001447523 #676

Open useryq8 opened 1 year ago

useryq8 commented 1 year ago

Environment Details

Describe the bug

the store performance become being slow when the storage size is about 1.2G, after cleaning some data , the size does not has any change. so I stop my app, and write a new tools, which read the existing storage the write to new storageManager, after copying , then validate it , but when I open the new storageManager, it reports the following error

one.microstream.storage.exceptions.StorageExceptionConsistency: No entity found for objectId 1000000000001447523

To Reproduce

copy the existing microstream storageManager data to a new one storageManager, then open it and validate it

my core code is like this

1, open the existing and new storageManagers by try
    the DataRoot has some LazyArrayLists

2,
      PListContainer<CellMeta> newCellMetaList = new PListContainer<>(newMeta.getStorage());
      PListContainer<CellMeta> existingCellMetaList = existingTables.get(branch); // existingTables is one LazyArrayList of the existing storageManager

      newCellMetaList.writeOp(metaList -> {
        CellMeta[] metaArray = existingCellMetaList.toArray(new CellMeta[0]);
        metaList.addAll(Arrays.asList(metaArray));
      });

      newTables.put(branch, newCellMetaList);

    newMeta.store(newTables); //  newMeta is new storageManager object
    newTables.store(); // newTables is a a member for DataRoot of new storageManager

Expected behavior

when validating , it works well ,it can read data from new storageManager

Screenshots

If applicable, add screenshots to help explain your problem.

Additional context

2023-11-02 13:39:42,322 ERROR a.m.osmdb.cellconn.base.PListContainer.readOp [pool-1-thread-5] Error while reading value, errMsg:Problem in channel #0 one.microstream.storage.exceptions.StorageException: Problem in channel #0 at one.microstream.storage.types.StorageChannelTask$Abstract.checkForProblems(StorageChannelTask.java:114) at one.microstream.storage.types.StorageChannelTask$Abstract.waitOnCompletion(StorageChannelTask.java:176) at one.microstream.storage.types.StorageRequestAcceptor$Default.waitOnTask(StorageRequestAcceptor.java:162) at one.microstream.storage.types.StorageRequestAcceptor$Default.queryByObjectIds(StorageRequestAcceptor.java:191) at one.microstream.storage.embedded.types.EmbeddedStorageBinarySource$Default.readByObjectIds(EmbeddedStorageBinarySource.java:104) at one.microstream.persistence.binary.types.BinaryLoader$Default.readLoadOidData(BinaryLoader.java:790) at one.microstream.persistence.binary.types.BinaryLoader$Default.getObject(BinaryLoader.java:858) at one.microstream.persistence.types.PersistenceManager$Default.getObject(PersistenceManager.java:417) at one.microstream.storage.types.StorageConnection.getObject(StorageConnection.java:455) at one.microstream.storage.types.Database$Default.getObject(Database.java:165) at one.microstream.reference.Lazy$Default.load(Lazy.java:527) at one.microstream.reference.Lazy$Default.get(Lazy.java:508) at one.microstream.collections.lazy.LazyArrayList$Segment.getData(LazyArrayList.java:1223) at one.microstream.collections.lazy.LazyArrayList.get(LazyArrayList.java:260) at java.base/java.util.AbstractList$Itr.next(AbstractList.java:371) at xxx.cellconn.CopyMigrationMetaListValidator.lambda$checkBase$7(CopyMigrationMetaListValidator.java:155) at xxx.cellconn.base.PListContainer.readOp(PListContainer.java:67) at xxx.cellconn.CopyMigrationMetaListValidator.lambda$checkBase$8(CopyMigrationMetaListValidator.java:165) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:833) Caused by: one.microstream.storage.exceptions.StorageExceptionConsistency: No entity found for objectId 1000000000001447523 at one.microstream.storage.types.StorageChannel$EntityCollectorByOid.accept(StorageChannel.java:826) at one.microstream.persistence.binary.types.LoadItemsChain$ChannelHashing$ChainItemObjectIdSet.iterate(LoadItemsChain.java:330) at one.microstream.storage.types.StorageChannel$Default.collectLoadByOids(StorageChannel.java:591) at one.microstream.storage.types.StorageRequestTaskLoadByOids$Default.internalProcessBy(StorageRequestTaskLoadByOids.java:58) at one.microstream.storage.types.StorageRequestTaskLoadByOids$Default.internalProcessBy(StorageRequestTaskLoadByOids.java:28) at one.microstream.storage.types.StorageChannelTask$Abstract.processBy(StorageChannelTask.java:252) at one.microstream.storage.types.StorageChannel$Default.work(StorageChannel.java:409) at one.microstream.storage.types.StorageChannel$Default.run(StorageChannel.java:492) ... 1 common frames omitted

hg-ms commented 1 year ago

Hello, The exception is most likely caused by the lazy references in the source storage. Lazy references are tricky to copy from one storage to another as they are bound to a storage. To do so they must be loaded, and you need to unlink (one.microstream.reference.Lazy.Default.$unlink()) the lazy references from the source storage.

To reduce the size of a storage you can enforce cleanup by manually calling issueFullCacheCheck() and issueFullFileCheck() from the storage manager.

In case you want create a “cleaned up” copy of a storage you may use the StorageConverter to copy a non running storage. This will not copy unreferenced (deleted) data.