search

sonatype / docker-nexus

Docker image for Sonatype Nexus
Other
383 stars 203 forks source link

Container failing to initialize when storage exists #56

Open jamesrenaud opened 7 years ago

jamesrenaud commented 7 years ago

Hey folks,

I'm using an NFS drive to persist data for a Nexus 3 container. When the container starts, there is an existing blobs directory and configuration in place. As the container starts, it fails with the following:

2017-07-05 15:07:41,933+0000 INFO [FelixStartLevel] *SYSTEM org.sonatype.nexus.quartz.internal.QuartzSchedulerSPI - Quartz Scheduler v2.2.2 2017-07-05 15:07:42,454+0000 INFO [FelixStartLevel] *SYSTEM org.sonatype.nexus.quartz.internal.task.QuartzTaskInfo - Task 'Backup' [db.backup] : state=WAITING 2017-07-05 15:07:42,543+0000 INFO [FelixStartLevel] *SYSTEM org.sonatype.nexus.quartz.internal.task.QuartzTaskInfo - Task 'Storage facet cleanup' [repository.storage-facet-cleanup] : state=WAITING 2017-07-05 15:07:42,557+0000 INFO [FelixStartLevel] *SYSTEM org.sonatype.nexus.quartz.internal.task.QuartzTaskInfo - Task 'Rebuild' [repository.rebuild-index] : state=WAITING 2017-07-05 15:07:42,684+0000 ERROR [FelixStartLevel] *SYSTEM org.sonatype.nexus.repository.internal.blobstore.BlobStoreManagerImpl - Failed transition: NEW -> STARTED org.sonatype.nexus.blobstore.api.BlobStoreException: BlobId: null, Unable to initialize blob store directory structure: default at org.sonatype.nexus.blobstore.file.internal.FileBlobStore.init(FileBlobStore.java:584) [na:na] at org.sonatype.nexus.repository.internal.blobstore.BlobStoreManagerImpl.newBlobStore(BlobStoreManagerImpl.java:201) [na:na] at org.sonatype.nexus.repository.internal.blobstore.BlobStoreManagerImpl.doStart(BlobStoreManagerImpl.java:102) [na:na] at org.sonatype.nexus.common.stateguard.StateGuardLifecycleSupport.start(StateGuardLifecycleSupport.java:67) [org.sonatype.nexus.common:3.3.2.02] at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39) [org.sonatype.nexus.common:3.3.2.02] at org.sonatype.nexus.common.stateguard.StateGuard$TransitionImpl.run(StateGuard.java:191) [org.sonatype.nexus.common:3.3.2.02] at org.sonatype.nexus.common.stateguard.TransitionsInterceptor.invoke(TransitionsInterceptor.java:56) [org.sonatype.nexus.common:3.3.2.02] at org.sonatype.nexus.repository.manager.internal.RepositoryManagerImpl.doStart(RepositoryManagerImpl.java:208) [na:na] at org.sonatype.nexus.common.stateguard.StateGuardLifecycleSupport.start(StateGuardLifecycleSupport.java:67) [org.sonatype.nexus.common:3.3.2.02] at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39) [org.sonatype.nexus.common:3.3.2.02] at org.sonatype.nexus.common.stateguard.StateGuard$TransitionImpl.run(StateGuard.java:191) [org.sonatype.nexus.common:3.3.2.02] at org.sonatype.nexus.common.stateguard.TransitionsInterceptor.invoke(TransitionsInterceptor.java:56) [org.sonatype.nexus.common:3.3.2.02] at org.sonatype.nexus.extender.NexusLifecycleManager.startComponent(NexusLifecycleManager.java:153) [org.sonatype.nexus.extender:3.3.2.02] at org.sonatype.nexus.extender.NexusLifecycleManager.to(NexusLifecycleManager.java:93) [org.sonatype.nexus.extender:3.3.2.02] at org.sonatype.nexus.extender.NexusContextListener.frameworkEvent(NexusContextListener.java:188) [org.sonatype.nexus.extender:3.3.2.02] at org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1429) [org.apache.felix.framework-5.6.1.jar:na] at org.apache.felix.framework.FrameworkStartLevelImpl.run(FrameworkStartLevelImpl.java:308) [org.apache.felix.framework-5.6.1.jar:na] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131] Caused by: java.nio.file.AccessDeniedException: /nexus-data/blobs/default at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) [na:1.8.0_131] at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) [na:1.8.0_131] at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) [na:1.8.0_131] at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384) [na:1.8.0_131] at java.nio.file.Files.createDirectory(Files.java:674) [na:1.8.0_131] at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781) [na:1.8.0_131] at java.nio.file.Files.createDirectories(Files.java:767) [na:1.8.0_131] at org.sonatype.nexus.common.io.DirectoryHelper.mkdir(DirectoryHelper.java:143) [na:na] at org.sonatype.nexus.blobstore.file.internal.FileBlobStore.init(FileBlobStore.java:579) [na:na] ... 17 common frames omitted

So far as I can tell it's expecting an empty or clean install default store in the blobs directory, but because I have an existing installation this fails. How is the container supposed to handle an existing persistent data volume?

staylorx commented 6 years ago

I have the same issue but with quieter logs: "Nexus data directory already in use: /nexus-data". It's a clear error but as @JamesrenaudMitel asks how do containers share existing volumes?

jamesrenaud commented 6 years ago

@staylorx I vaguely remember this issue and it may have been a permissions problem on the directory you're mounting to nexus-data. I believe I solved this by doing a CHOWN -R 200:200 /nexus-data

joedragons commented 6 years ago

A data directory should only be used by one NXRM instance at the time for the record. Using it against multiple NXRM instances at a time will cause trouble.

lopf commented 6 years ago

Same here. The use-case is a nexus container running in a Kubernetes Deployment and the node where the pod is running on abruptly dies. A replica count of 1 makes the Deployment to start a new pod but it then crash-loops because of the "lock". The PV is mounted via NFS.

ericleu commented 6 years ago

I am noticing the same issue in nexus3 (3.8.0) as well; I am using docker container image, not sure if that makes a diff. I found a work around to bring nexus back by removing /nexus-data/lock and /nexus-data/tmp/*. Since this is a POC, repos are still empty. So I am not sure what the actual effect is when there is live data.

chtompki commented 5 years ago

I've also seen this particularly with regards to having an efs system symlinked into /nexus-data/blobs. We get exceptions that the file system is already created. Must the java process create the filesystem for things to be valid?

Edit. I've found that the best plan is to chown -R 200:200 the efs file system and volume mount it into the container separately. After the container is running is when the chown need take place.