All Blob stores are offline after migration to v3.73.0: unable to migrate secrets error

[azuregos]

We have just recently migrated our Nexus OSS service from v3.72.0 to v3.73.0. Immediately after Nexus has started we noticed that all S3 blob stores are reported as not started. Further investigation uncovered the following error message for each blob store we have. The same error appears when I attempt to modify blob store configuration via API or UI.

2024-10-11 15:58:26,282+0000 ERROR [quartz-9-thread-5]  *SYSTEM org.sonatype.nexus.internal.security.secrets.task.SecretsMigrationTask - Failed to run task 'Migrate existing secrets into a single source (secrets table).'
org.sonatype.nexus.security.secrets.SecretMigrationException: unable to migrate secrets for blobstore: xxxxxxxx
    at org.sonatype.nexus.repository.internal.blobstore.secrets.migration.BlobStoreConfigSecretsMigrator.migrate(BlobStoreConfigSecretsMigrator.java:69)
    at org.sonatype.nexus.internal.security.secrets.task.SecretsMigrationTask.execute(SecretsMigrationTask.java:51)
    at org.sonatype.nexus.scheduling.TaskSupport.call(TaskSupport.java:105)
    at org.sonatype.nexus.quartz.internal.task.QuartzTaskJob.doExecute(QuartzTaskJob.java:143)
    at org.sonatype.nexus.quartz.internal.task.QuartzTaskJob.execute(QuartzTaskJob.java:106)
    at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
    at org.sonatype.nexus.quartz.internal.QuartzThreadPool.lambda$0(QuartzThreadPool.java:145)
    at org.sonatype.nexus.thread.internal.MDCAwareRunnable.run(MDCAwareRunnable.java:40)
    at org.apache.shiro.subject.support.SubjectRunnable.doRun(SubjectRunnable.java:120)
    at org.apache.shiro.subject.support.SubjectRunnable.run(SubjectRunnable.java:108)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
    at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: org.sonatype.nexus.crypto.internal.error.CipherException: last block incomplete in decryption
    at org.sonatype.nexus.crypto.internal.LegacyCipherFactoryImpl$PbeCipherImpl.transform(LegacyCipherFactoryImpl.java:114)
    at org.sonatype.nexus.crypto.internal.LegacyCipherFactoryImpl$PbeCipherImpl.decrypt(LegacyCipherFactoryImpl.java:103)
    at org.sonatype.nexus.crypto.secrets.internal.SecretsServiceImpl.decryptLegacy(SecretsServiceImpl.java:306)
    at org.sonatype.nexus.crypto.secrets.internal.SecretsServiceImpl.doDecrypt(SecretsServiceImpl.java:229)
    at org.sonatype.nexus.crypto.secrets.internal.SecretsServiceImpl.access$0(SecretsServiceImpl.java:227)
    at org.sonatype.nexus.crypto.secrets.internal.SecretsServiceImpl$SecretImpl.decrypt(SecretsServiceImpl.java:364)
    at org.sonatype.nexus.repository.internal.blobstore.secrets.migration.BlobStoreConfigSecretsMigrator.maybeMigrateSecret(BlobStoreConfigSecretsMigrator.java:85)
    at org.sonatype.nexus.repository.internal.blobstore.secrets.migration.BlobStoreConfigSecretsMigrator.migrate(BlobStoreConfigSecretsMigrator.java:65)
    ... 14 common frames omitted
Caused by: javax.crypto.IllegalBlockSizeException: last block incomplete in decryption
    at org.bouncycastle.jcajce.provider.symmetric.util.BaseBlockCipher.engineDoFinal(Unknown Source)
    at java.base/javax.crypto.Cipher.doFinal(Cipher.java:2263)
    at org.sonatype.nexus.crypto.internal.LegacyCipherFactoryImpl$PbeCipherImpl.transform(LegacyCipherFactoryImpl.java:110)
    ... 21 common frames omitted
2024-10-11 15:58:26,283+0000 INFO  [quartz-9-thread-5]  *SYSTEM org.sonatype.nexus.internal.security.secrets.task.SecretsMigrationTask - Task complete

I have not found a workaround. I have additionally followed the release note instructions and created an encryption key using a nexus.secrets.file property and migrated secrets using REST API call. But the problem with blob store persists.

[garesgit]

Having same issue. Executed http://localhost:8081/service/rest/v1/secrets/encryption/re-encrypt but s3 blobstore are all down.

Got this log:

2024-10-12 15:32:45,232+0000 INFO [quartz-9-thread-6] SYSTEM org.sonatype.nexus.internal.security.secrets.tasks.ReEncryptTask - Task information: 2024-10-12 15:32:45,233+0000 INFO [quartz-9-thread-6] SYSTEM org.sonatype.nexus.internal.security.secrets.tasks.ReEncryptTask - ID: 4c7e580d-c5ef-4366-9971-5e4900880e7d 2024-10-12 15:32:45,233+0000 INFO [quartz-9-thread-6] SYSTEM org.sonatype.nexus.internal.security.secrets.tasks.ReEncryptTask - Type: security.secrets.re-encrypt 2024-10-12 15:32:45,233+0000 INFO [quartz-9-thread-6] SYSTEM org.sonatype.nexus.internal.security.secrets.tasks.ReEncryptTask - Name: Admin - Re-encrypt secrets with the specified key 2024-10-12 15:32:45,233+0000 INFO [quartz-9-thread-6] SYSTEM org.sonatype.nexus.internal.security.secrets.tasks.ReEncryptTask - Description: Re-encrypting secrets with specified key id 2024-10-12 15:32:55,234+0000 INFO [quartz-9-thread-6] SYSTEM org.sonatype.nexus.internal.security.secrets.tasks.ReEncryptTask - Started re-encrypting secrets with provided keyId 2024-10-12 15:32:55,234+0000 INFO [quartz-9-thread-6] SYSTEM org.sonatype.nexus.internal.security.secrets.tasks.ReEncryptTask - Completed re-encryption of secrets with keyId 'xxxxx'. Processed 0 secrets 2024-10-12 15:32:55,234+0000 INFO [quartz-9-thread-6] SYSTEM org.sonatype.nexus.internal.security.secrets.tasks.ReEncryptTask - Task complete

[garesgit]

Actually, having same error as azuregos.

[Xeroxxx]

Got the same issue.

Health data claims:

• Nexus was not configured with an encryption key and is using the Default key.
• Blob store 'minio-default' reports as not started

[garesgit]

Just a clue of what could be going on here. I tried to find the diffs and it looks like the issue is happening on the AmazonS3Factory.java (I having issues with the AWS S3 repos). Doing a diff with the previous version I can see that there is decrypt method call over the AWS secretAccessKey. In previous version there was not and it was passed thorugh clear (see below diff with previous version). I checked the DB and the secretAccessKey is not encrypted (it is in cleartext). Due just code change in the AmazonS3Factory.java and as a matter of test, I have replaced the AmazonS3Factory.class into the nexus-blobstore-s3-3.73.0-12.jar with the class of v3.72-04 and the S3 repos started fine with only this change. So, wondering if the code is trying to decrypt something that is not encrypted and that is why we are getting that error.

$ diff ./nexus-public-release-3.73.0-12/plugins/nexus-blobstore-s3/src/main/java/org/sonatype/nexus/blobstore/s3/internal/AmazonS3Factory.java nexus-public-release-3.72.0-04/plugins/nexus-blobstore-s3/src/main/java/org/sonatype/nexus/blobstore/s3/internal/AmazonS3Fa
ctory.java
26d25
< import org.sonatype.nexus.crypto.secrets.SecretsFactory;
47d45
< import static com.google.common.base.Preconditions.checkNotNull;
71,72d68
<   private final SecretsFactory secretsFactory;
<
77,78c73
<                          @Named("${nexus.s3.cloudwatchmetrics.namespace:-nexus-blobstore-s3}") final String cloudWatchMetricsNamespace,
<                          final SecretsFactory secretsFactory) {
---
>                          @Named("${nexus.s3.cloudwatchmetrics.namespace:-nexus-blobstore-s3}") final String cloudWatchMetricsNamespace) {
83d77
<     this.secretsFactory = checkNotNull(secretsFactory);
104,105c98
<       String decryptedAccessKey = new String(secretsFactory.from(secretAccessKey).decrypt());
<       AWSCredentials credentials = buildCredentials(accessKeyId, decryptedAccessKey, sessionToken);
---
>       AWSCredentials credentials = buildCredentials(accessKeyId, secretAccessKey, sessionToken);

[Xeroxxx]

Ended up downgrading and restoring the database.

For replacing AmazonS3Factory.java I would need to build an own container.

[rmartcas]

Same error here. Unable to use nexus in the 3.73.0 release:

2024-10-15T11:42:41.322529258Z org.sonatype.nexus.blobstore.api.BlobStoreException: Unable to initialize blob store bucket: xxxxxxxxxxxxxxxxxxx, Cause: last block incomplete in decryption

[vramperez]

Same error here. Downgrading to version 3.72.0.

[Xeroxxx]

The thing is, I would be fine if the re-encrypt would work with the default key, but it doesn't. I don't want to create an own secret. Its an internal system without any connection to the outside world. I stay on 3.72.0 till this is solved.

[xepa]

Same error here als downgraded.

[mrprescott]

@garesgit is correct, there's a bug here that causes Repo to sometimes try to decrypt something that's not encrypted, so the upgrade step fails. Apologies for the pain here, we have a fix in progress that is scheduled to come out in our next monthly release (scheduled for the first week of November).

[mokrinsky]

I had the same issue and with the new 3.74.0 release everything went back to normal.

[mrprescott]

Okay, 3.74 is out with a fix. Apologies for the friction trying to upgrade. Thanks for coming back and posting that it worked for you, @mokrinsky.

[wnobres-sr]

We migrated from 3.70.1 to 3.74.0 and we are getting the same error message but just for one random repository for the multiple repos we have. Every time we restart nexus, only one random Blob Repository stops to work because of this message during initialization, and all others work fine:

2024-11-22 17:41:32,427+0000 INFO [quartz-9-thread-4] *SYSTEM org.sonatype.nexus.internal.security.secrets.task.SecretsMigrationTask - Task log: /nexus-data/log/tasks/secrets.migration-20241122174132424.log 2024-11-22 17:41:32,579+0000 WARN [quartz-9-thread-4] *SYSTEM org.sonatype.nexus.quartz.internal.task.QuartzTaskJob - Task 2e10be16-b36f-4091-8a5c-76a9c1685821 : 'Secrets - Migrate secrets' [secrets.migration] execution failure org.sonatype.nexus.security.secrets.SecretMigrationException: unable to migrate secrets for blobstore: raw-golden-images at org.sonatype.nexus.repository.internal.blobstore.secrets.migration.BlobStoreConfigSecretsMigrator.migrate(BlobStoreConfigSecretsMigrator.java:69) at org.sonatype.nexus.internal.security.secrets.task.SecretsMigrationTask.execute(SecretsMigrationTask.java:51) at org.sonatype.nexus.scheduling.TaskSupport.call(TaskSupport.java:105) at org.sonatype.nexus.quartz.internal.task.QuartzTaskJob.doExecute(QuartzTaskJob.java:143) at org.sonatype.nexus.quartz.internal.task.QuartzTaskJob.execute(QuartzTaskJob.java:106) at org.quartz.core.JobRunShell.run(JobRunShell.java:202) at org.sonatype.nexus.quartz.internal.QuartzThreadPool.lambda$0(QuartzThreadPool.java:145) at org.sonatype.nexus.thread.internal.MDCAwareRunnable.run(MDCAwareRunnable.java:40) at org.apache.shiro.subject.support.SubjectRunnable.doRun(SubjectRunnable.java:120) at org.apache.shiro.subject.support.SubjectRunnable.run(SubjectRunnable.java:108) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:840) Caused by: org.sonatype.nexus.blobstore.s3.internal.S3BlobStoreException: Bucket exists but is not owned by you. at org.sonatype.nexus.blobstore.s3.internal.S3BlobStoreException.bucketOwnershipError(S3BlobStoreException.java:90) at org.sonatype.nexus.blobstore.s3.internal.BucketManager.checkBucketOwner(BucketManager.java:261) at org.sonatype.nexus.blobstore.s3.internal.BucketManager.checkPermissions(BucketManager.java:233) at org.sonatype.nexus.blobstore.s3.internal.BucketManager.prepareStorageLocation(BucketManager.java:84) at org.sonatype.nexus.blobstore.s3.internal.S3BlobStore.doInit(S3BlobStore.java:628) at org.sonatype.nexus.blobstore.BlobStoreSupport.init(BlobStoreSupport.java:276) at org.sonatype.nexus.repository.internal.blobstore.BlobStoreManagerImpl.startWithConfig(BlobStoreManagerImpl.java:382) at org.sonatype.nexus.repository.internal.blobstore.BlobStoreManagerImpl.doUpdate(BlobStoreManagerImpl.java:348) at org.sonatype.nexus.repository.internal.blobstore.BlobStoreManagerImpl.update(BlobStoreManagerImpl.java:315) at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39) at org.sonatype.nexus.common.stateguard.StateGuard$GuardImpl.run(StateGuard.java:287) at org.sonatype.nexus.common.stateguard.GuardedInterceptor.invoke(GuardedInterceptor.java:54) at org.sonatype.nexus.repository.internal.blobstore.secrets.migration.BlobStoreConfigSecretsMigrator.maybeMigrateSecret(BlobStoreConfigSecretsMigrator.java:86) at org.sonatype.nexus.repository.internal.blobstore.secrets.migration.BlobStoreConfigSecretsMigrator.migrate(BlobStoreConfigSecretsMigrator.java:65) ... 14 common frames omitted 2024-11-22 17:41:32,580+0000 INFO [quartz-9-thread-4] *SYSTEM org.sonatype.nexus.quartz.internal.task.QuartzTaskInfo - Task 'Secrets - Migrate secrets' [secrets.migration] state change RUNNING -> OK

Any ideas?