Nexus 3.9.0-01 with 1.0.5 Failed Transition state causes Nexus to fail to restart

ju2wheels commented 6 years ago

Thanks for creating an issue! Please fill out this form so we can be sure to have all the information we need, and to minimize back and forth.

What are you trying to do? I installed 1.0.5 on Nexus 3.9.0-01 which went fine and Nexus started OK. I configured a bunch of repositories including four apt hosted repos (all repos were empty and had no artifacts, although my S3 bucket has two property files: metrics and metadata). For the GPG key, I concatted the output of the public/secret amored GPG output for the RSA signing key (no idea if thats correct or if it only wants the private key only since its not documented anywhere).

Upon restarting the Nexus service in order to expose some additional ports on the Docker container, I get the following error:

2018-03-09 21:24:11,088+0000 ERROR [FelixStartLevel] *SYSTEM org.sonatype.nexus.repository.manager.internal.RepositoryManagerImpl - Failed transition: NEW -> STARTED
java.lang.IllegalStateException: Missing recipe: apt-hosted
    at com.google.common.base.Preconditions.checkState(Preconditions.java:518)
    at org.sonatype.nexus.repository.manager.internal.RepositoryManagerImpl.recipe(RepositoryManagerImpl.java:138)
    at org.sonatype.nexus.repository.manager.internal.RepositoryManagerImpl.newRepository(RepositoryManagerImpl.java:156)
    at org.sonatype.nexus.repository.manager.internal.RepositoryManagerImpl.restoreRepositories(RepositoryManagerImpl.java:246)
    at org.sonatype.nexus.repository.manager.internal.RepositoryManagerImpl.doStart(RepositoryManagerImpl.java:228)
    at org.sonatype.nexus.common.stateguard.StateGuardLifecycleSupport.start(StateGuardLifecycleSupport.java:67)
    at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)
    at org.sonatype.nexus.common.stateguard.StateGuard$TransitionImpl.run(StateGuard.java:191)
    at org.sonatype.nexus.common.stateguard.TransitionsInterceptor.invoke(TransitionsInterceptor.java:56)
    at org.sonatype.nexus.extender.NexusLifecycleManager.startComponent(NexusLifecycleManager.java:155)
    at org.sonatype.nexus.extender.NexusLifecycleManager.to(NexusLifecycleManager.java:95)
    at org.sonatype.nexus.extender.NexusContextListener.frameworkEvent(NexusContextListener.java:189)
    at org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1429)
    at org.apache.felix.framework.FrameworkStartLevelImpl.run(FrameworkStartLevelImpl.java:308)
    at java.lang.Thread.run(Thread.java:748)
2018-03-09 21:24:11,098+0000 ERROR [FelixStartLevel] *SYSTEM org.sonatype.nexus.extender.NexusContextListener - Failed to start nexus
java.lang.IllegalStateException: Missing recipe: apt-hosted
    at com.google.common.base.Preconditions.checkState(Preconditions.java:518)
    at org.sonatype.nexus.repository.manager.internal.RepositoryManagerImpl.recipe(RepositoryManagerImpl.java:138)
    at org.sonatype.nexus.repository.manager.internal.RepositoryManagerImpl.newRepository(RepositoryManagerImpl.java:156)
    at org.sonatype.nexus.repository.manager.internal.RepositoryManagerImpl.restoreRepositories(RepositoryManagerImpl.java:246)
    at org.sonatype.nexus.repository.manager.internal.RepositoryManagerImpl.doStart(RepositoryManagerImpl.java:228)
    at org.sonatype.nexus.common.stateguard.StateGuardLifecycleSupport.start(StateGuardLifecycleSupport.java:67)
    at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)
    at org.sonatype.nexus.common.stateguard.StateGuard$TransitionImpl.run(StateGuard.java:191)
    at org.sonatype.nexus.common.stateguard.TransitionsInterceptor.invoke(TransitionsInterceptor.java:56)
    at org.sonatype.nexus.extender.NexusLifecycleManager.startComponent(NexusLifecycleManager.java:155)
    at org.sonatype.nexus.extender.NexusLifecycleManager.to(NexusLifecycleManager.java:95)
    at org.sonatype.nexus.extender.NexusContextListener.frameworkEvent(NexusContextListener.java:189)
    at org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1429)
    at org.apache.felix.framework.FrameworkStartLevelImpl.run(FrameworkStartLevelImpl.java:308)
    at java.lang.Thread.run(Thread.java:748)

Note that im also doing all of this with the S3 backend plugin.

What feature or behavior is this required for? hosted apt repo
How could we solve this issue? (Not knowing is okay!) Identify if the above bug is with apt or s3 backend plugin or the combination of both (I dont think it is S3 though, im not getting permissions errors and theres already metadata files on the bucket).
Anything else?

DarthHater commented 6 years ago

How did you install apt in Docker? The error indicates the recipe is missing which seems to indicate that apt was installed in a temporary fashion.

ju2wheels commented 6 years ago

Its definitely installed into the image itself, but ill rebuild it again and see if it can start again when pointed at the original data dir.

Repo Branch Docker file Docker Compose File

I basically just took upstream and changed the Dockerfiles on the plugin repos to spit out the JAR into a target dir, and the Nexus repo contains them as submodules and includes their output JAR in the image (at build not at runtime, so it shouldnt be temporary).

I have the plugin building and Nexus image building separately now but will merge them into one Dockerfile the modern way later but it shouldnt make a difference.

Is that the plugin loader routine thats failing to find the JAR for the apt plugin or could it be related to the service being abruptly shutdown and the db is in an inconsistent state (since Docker compose only gives it 10 secs to stop by default)? If its failing to find it then that would be odd since all I did was restart the container and I would think the apt-hosted repo would not have been an option at all if it werent in my image when I started it initially.

ju2wheels commented 6 years ago

Rebuilt images and reconfigured all my repos/roles again and still get the same issue after stopping and restarting.

I started up my Docker image and manually went in to double check whats in it:

$ docker run -it nexus3:with_apt_s3_blobstore /bin/bash
bash-4.2$ cd /opt/sonatype/nexus/system
bash-4.2$ find . -type f -name "*repository-apt*"
./net/staticsnow/nexus-repository-apt/1.0.5/nexus-repository-apt-1.0.5.jar
bash-4.2$ find . -type f -name "*blobstore-s3*"  
./org/sonatype/nexus/nexus-blobstore-s3/1.2.1-SNAPSHOT/nexus-blobstore-s3-1.2.1-SNAPSHOT.jar
bash-4.2$ find . -type f -exec grep blobstore-s3 {} +
./org/sonatype/nexus/assemblies/nexus-core-feature/3.9.0-01/nexus-core-feature-3.9.0-01-features.xml:<bundle>mvn:org.sonatype.nexus/nexus-blobstore-s3/1.2.1-SNAPSHOT</bundle>
./org/sonatype/nexus/assemblies/nexus-base-feature/3.9.0-01/nexus-base-feature-3.9.0-01-features.xml:<bundle>mvn:org.sonatype.nexus/nexus-blobstore-s3/1.2.1-SNAPSHOT</bundle>
Binary file ./org/sonatype/nexus/nexus-blobstore-s3/1.2.1-SNAPSHOT/nexus-blobstore-s3-1.2.1-SNAPSHOT.jar matches
bash-4.2$ find . -type f -exec grep repository-apt {} +  
Binary file ./net/staticsnow/nexus-repository-apt/1.0.5/nexus-repository-apt-1.0.5.jar matches
./com/sonatype/nexus/assemblies/nexus-oss-feature/3.9.0-01/nexus-oss-feature-3.9.0-01-features.xml:        <feature prerequisite="false" dependency="false">nexus-repository-apt</feature>
./com/sonatype/nexus/assemblies/nexus-oss-feature/3.9.0-01/nexus-oss-feature-3.9.0-01-features.xml:    <feature name="nexus-repository-apt" description="net.staticsnow:nexus-repository-apt" version="1.0.5">
./com/sonatype/nexus/assemblies/nexus-oss-feature/3.9.0-01/nexus-oss-feature-3.9.0-01-features.xml:        <details>net.staticsnow:nexus-repository-apt</details>
./com/sonatype/nexus/assemblies/nexus-oss-feature/3.9.0-01/nexus-oss-feature-3.9.0-01-features.xml:        <bundle>mvn:net.staticsnow/nexus-repository-apt/1.0.5</bundle>

Is the Nexus OSS feature the right file to put that repository-apt metadata into or does it need to go into core (not sure if one gets loaded before the other and that could make a difference)?

ju2wheels commented 6 years ago

@DarthHater : I figured out what triggers it. Adding the PRO evaluation license!

I was able to add 4 apt hosted repos + roles with different permissions for each of these and restart ok. Then when I added the PRO eval license and restarted, it wouldnt come back up.

Is there already apt support expose by the PRO eval license that conflicts with this open source plugin?

DarthHater commented 6 years ago

Not sure, but I’ll take a gander later today. Thanks for the detective work and good info 🕵️‍♂️ On Mon, Mar 12, 2018 at 9:54 AM Julio Lajara notifications@github.com wrote:

@DarthHater https://github.com/darthhater : I figured out what triggers it. Adding the PRO evaluation license!

I was able to add 4 apt hosted repos + roles with different permissions for each of these and restart ok. Then when I added the PRO eval license and restarted, it wouldnt come back up.

Is there already apt support expose by the PRO eval license that conflicts with this open source plugin?

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/sonatype-nexus-community/nexus-repository-apt/issues/41#issuecomment-372404095, or mute the thread https://github.com/notifications/unsubscribe-auth/AFSZhkhGakAGZpGz6JEuTGapB3biYsiJks5tdrZLgaJpZM4Sk5kn .

ju2wheels commented 6 years ago

Closing, I got it fixed. Please update this Dockerfile in this repo to use * instead of oss in the paths. I copied it wholesale without reading the README and it doesnt configure the plugin for pro hence it not loading instead of conflicting with a pro alternative like I thought.

    sed -i 's@nexus-repository-npm</feature>@nexus-repository-npm</feature>\n        <feature prerequisite="false" dependency="false">nexus-repository-apt</feature>@g' /opt/sonatype/nexus/system/com/sonatype/nexus/assemblies/nexus-*-feature/${NEXUS_VERSION}-${NEXUS_BUILD}/nexus-*-feature-${NEXUS_VERSION}-${NEXUS_BUILD}-features.xml; \
    sed -i 's@<feature name="nexus-repository-npm"@<feature name="nexus-repository-apt" description="net.staticsnow:nexus-repository-apt" version="1.0.5">\n        <details>net.staticsnow:nexus-repository-apt</details>\n        <bundle>mvn:net.staticsnow/nexus-repository-apt/1.0.5</bundle>\n    </feature>\n    <feature name="nexus-repository-npm"@g' /opt/sonatype/nexus/system/com/sonatype/nexus/assemblies/nexus-*-feature/${NEXUS_VERSION}-${NEXUS_BUILD}/nexus-*-feature-${NEXUS_VERSION}-${NEXUS_BUILD}-features.xml;

DarthHater commented 6 years ago

I think we can actually modify the base instead of OSS. Pro and OSS both include that file, so that would be the better one to modify. I'll make a PR in a bit once I've tested it.

DarthHater commented 6 years ago

At any rate, you got it @ju2wheels , it was missing from the pro feature file. I'm working through a PR where we add it to core so that it just gets added automatically regardless of how you are running Nexus Repo.

mpoindexter commented 6 years ago

Yeah, my company ran into this same thing as soon as we enabled the Pro license. Maybe at some point it would be good to look into: https://issues.sonatype.org/browse/NEXUS-14673? Seems like the root problem here is that installing/uninstalling plugins is very complex and easy to mess up.

DarthHater commented 6 years ago

It totally is the worst pain point at time being. I would love if we could load KAR files or something akin.

sonatype-nexus-community / nexus-repository-apt

Nexus 3.9.0-01 with 1.0.5 Failed Transition state causes Nexus to fail to restart #41