"Repair - Reconcile component database from blob store" task fails if blobs are non existent

[Alphayeeeet]

As described above, the taks fails to reconcile the database, if a blob referenced by the database does not exist anymore (which may be the case after restoring a backup.

Issue could be resolved, by checking for non existent blobs, and remove them from the component database. So if a blob is does not exist, the task should remove all references in the database. This means to remove all assets contained by the blob, but keeping the repository configuration.

Environment: OS: Windows Server 2022 Build 20348.2031 JVM: Nexus Built-In Nexus Version: OSS 3.61.0-02 Database Backup was from Version OSS 3.60.0-02

Task log: 2023-10-16 09:21:56,137+0200 WARN [quartz-15-thread-20] *SYSTEM org.sonatype.nexus.blobstore.file.FileBlobStore - Attempt to access non-existent blob attributes file E:\sonatype-work\nexus3\blobs\default\content\vol-05\chap-44\2651dcac-6df6-45de-9a52-e11b1e3ef5cd.properties for blob 2651dcac-6df6-45de-9a52-e11b1e3ef5cd 2023-10-16 09:22:07,541+0200 WARN [quartz-15-thread-20] *SYSTEM org.sonatype.nexus.blobstore.file.FileBlobStore - Attempt to access non-existent blob attributes file E:\sonatype-work\nexus3\blobs\default\content\vol-05\chap-44\2651dcac-6df6-45de-9a52-e11b1e3ef5cd.properties for blob 2651dcac-6df6-45de-9a52-e11b1e3ef5cd 2023-10-16 09:22:07,541+0200 WARN [quartz-15-thread-20] *SYSTEM org.sonatype.nexus.transaction.RetryController - Exceeded retry limit: 8/8 (org.sonatype.nexus.repository.storage.MissingBlobException: Blob default@2651dcac-6df6-45de-9a52-e11b1e3ef5cd exists in metadata, but is missing from the blobstore) 2023-10-16 09:22:07,542+0200 ERROR [quartz-15-thread-20] *SYSTEM org.sonatype.nexus.blobstore.restore.orient.OrientRestoreMetadataTask - Failed to run task 'null' org.sonatype.nexus.repository.storage.RetryDeniedException: Exceeded retry limit at org.sonatype.nexus.repository.storage.StorageTxImpl.allowRetry(StorageTxImpl.java:228) at sun.reflect.GeneratedMethodAccessor528.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.sonatype.nexus.common.stateguard.SimpleMethodInvocation.proceed(SimpleMethodInvocation.java:53) at org.sonatype.nexus.common.stateguard.StateGuardAspect$1.invoke(StateGuardAspect.java:69) at com.sun.proxy.$Proxy229.allowRetry(Unknown Source) at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:83) at org.sonatype.nexus.transaction.TransactionInterceptor.proceedWithTransaction(TransactionInterceptor.java:66) at org.sonatype.nexus.transaction.TransactionInterceptor.invoke(TransactionInterceptor.java:55) at org.sonatype.nexus.repository.storage.DefaultComponentMaintenanceImpl.deleteAsset(DefaultComponentMaintenanceImpl.java:94) at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39) at org.sonatype.nexus.common.stateguard.StateGuard$GuardImpl.run(StateGuard.java:272) at org.sonatype.nexus.common.stateguard.GuardedInterceptor.invoke(GuardedInterceptor.java:54) at org.sonatype.nexus.repository.storage.DefaultComponentMaintenanceImpl.deleteAsset(DefaultComponentMaintenanceImpl.java:85) at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39) at org.sonatype.nexus.common.stateguard.StateGuard$GuardImpl.run(StateGuard.java:272) at org.sonatype.nexus.common.stateguard.GuardedInterceptor.invoke(GuardedInterceptor.java:54) at org.sonatype.nexus.repository.maintenance.internal.MaintenanceServiceImpl.deleteAsset(MaintenanceServiceImpl.java:85) at org.sonatype.nexus.blobstore.restore.orient.OrientRestoreMetadataTask.integrityCheckFailedHandler(OrientRestoreMetadataTask.java:291) at org.sonatype.nexus.blobstore.restore.orient.DefaultOrientIntegrityCheckStrategy.check(DefaultOrientIntegrityCheckStrategy.java:143) at org.sonatype.nexus.blobstore.restore.orient.OrientRestoreMetadataTask.lambda$8(OrientRestoreMetadataTask.java:279) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175) at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175) at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) at org.sonatype.nexus.blobstore.restore.orient.OrientRestoreMetadataTask.blobStoreIntegrityCheck(OrientRestoreMetadataTask.java:276) at org.sonatype.nexus.blobstore.restore.orient.OrientRestoreMetadataTask.execute(OrientRestoreMetadataTask.java:168) at org.sonatype.nexus.blobstore.restore.orient.OrientRestoreMetadataTask.execute(OrientRestoreMetadataTask.java:1) at org.sonatype.nexus.scheduling.TaskSupport.call(TaskSupport.java:100) at org.sonatype.nexus.quartz.internal.task.QuartzTaskJob.doExecute(QuartzTaskJob.java:143) at org.sonatype.nexus.quartz.internal.task.QuartzTaskJob.execute(QuartzTaskJob.java:106) at org.quartz.core.JobRunShell.run(JobRunShell.java:202) at org.sonatype.nexus.quartz.internal.QuartzThreadPool.lambda$0(QuartzThreadPool.java:145) at org.sonatype.nexus.thread.internal.MDCAwareRunnable.run(MDCAwareRunnable.java:40) at org.apache.shiro.subject.support.SubjectRunnable.doRun(SubjectRunnable.java:120) at org.apache.shiro.subject.support.SubjectRunnable.run(SubjectRunnable.java:108) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: org.sonatype.nexus.repository.storage.MissingBlobException: Blob default@2651dcac-6df6-45de-9a52-e11b1e3ef5cd exists in metadata, but is missing from the blobstore at org.sonatype.nexus.repository.storage.StorageTxImpl.requireBlob(StorageTxImpl.java:1067) at sun.reflect.GeneratedMethodAccessor526.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.sonatype.nexus.common.stateguard.SimpleMethodInvocation.proceed(SimpleMethodInvocation.java:53) at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39) at org.sonatype.nexus.common.stateguard.StateGuard$GuardImpl.run(StateGuard.java:272) at org.sonatype.nexus.common.stateguard.GuardedInterceptor.invoke(GuardedInterceptor.java:54) at org.sonatype.nexus.common.stateguard.StateGuardAspect$1.invoke(StateGuardAspect.java:63) at com.sun.proxy.$Proxy229.requireBlob(Unknown Source) at com.sonatype.nexus.repository.npm.internal.orient.NpmFacetUtils.loadPackageRoot(NpmFacetUtils.java:359) at com.sonatype.nexus.repository.npm.internal.orient.NpmHostedComponentMaintenanceImpl.deleteTarball(NpmHostedComponentMaintenanceImpl.java:123) at com.sonatype.nexus.repository.npm.internal.orient.NpmHostedComponentMaintenanceImpl.deleteAssetTx(NpmHostedComponentMaintenanceImpl.java:81) at com.sonatype.nexus.repository.npm.internal.orient.NpmHostedComponentMaintenanceImpl.deleteAssetTx(NpmHostedComponentMaintenanceImpl.java:67) at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:58) ... 41 common frames omitted 2023-10-16 09:22:07,542+0200 INFO [quartz-15-thread-20] *SYSTEM org.sonatype.nexus.blobstore.restore.orient.OrientRestoreMetadataTask - Task complete 2023-10-16 09:22:38,931+0200 WARN [Thread-358 <command>sql.select from asset where bucket = :bucket</command>] *SYSTEM org.sonatype.nexus.repository.storage.OrientAsyncHelper$QueueFeedingResultListener - Timed out adding query result to queue 8d6fba9 after 60 seconds, aborting query 2023-10-16 09:23:38,942+0200 WARN [Thread-358 <command>sql.select from asset where bucket = :bucket</command>] *SYSTEM org.sonatype.nexus.repository.storage.OrientAsyncHelper$QueueFeedingResultListener - Timed out adding end marker to queue 8d6fba9 after 60 seconds

[mrprescott]

Thanks for filing this. Your request makes sense, although it's hard to know what might be going wrong, normally we'd need a support engagement to understand the state of the system. We've made a note about your suggested improvement, it does make sense that if your database backup is fresher than your blob store backup, you might have some stray asset records to prune when you reconcile them.

[Alphayeeeet]

Thank you for the reply. Support engagement would be difficult anyway as we use the OSS version of nexus. The problem is, as we tried to migrate our existing sonatype instance onto a new system, we wanted to cleanup our proxied repos, which are currently on the same blobstore as the hosted ones. Therefore we decided to reimport our hosted packages and prune the proxied ones. The above mentioned problem occured, when we restored the database but not the blobstores (because they contain proxied repositories, which we now moved to different blobstores during reimport) and tried to run the repair-task.

[Alphayeeeet]

@nblair @mrprescott Are there any updates?

[Alphayeeeet]

Any updates regarding this issue?

[Alphayeeeet]

Still waiting for any updates. @nblair @mrprescott

[github-actions[bot]]

This issue is stale because it has been open for 60 days with no activity.

[Alphayeeeet]

Still asking for updates and beeing ignored so far. @nblair @mrprescott

[github-actions[bot]]

This issue is stale because it has been open for 60 days with no activity.

[Alphayeeeet]

Still active