search

sonatype-nexus-community / nexus-repository-apt

A Nexus Repository 3 plugin that allows usage of apt repositories
Eclipse Public License 1.0
105 stars 50 forks source link

500 Error when Uploading Ubuntu Kernel Packages #52

Closed derjohn closed 6 years ago

derjohn commented 6 years ago

Hi, I try to upload an AMD64 kernel from Ubuntu's Mainline Repo. I worked until 4.16.5 kernel and stopped with 4.16.6 kernel.

(To reproduce look here: http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.16.6/ )

I did set the logger to

And noticed that I looks like having to do with the xz compression format, which apache commons can't handle. I bet ubuntu changed that internal compression format to something more hipsterish inside the deb, but didn't really dig deeper.

I am on 3.9.0/1.0.5.

Here is the log output:

#2018-05-01 10:00:31,778+0000 DEBUG [qtp1790585161-43]  *SYSTEM org.sonatype.nexus.internal.web.ErrorPageServlet - Attaching cause
javax.servlet.ServletException: Filtered request failed.
    at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:384)
    at org.sonatype.nexus.security.SecurityFilter.doFilterInternal(SecurityFilter.java:101)
    at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
    at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
    at org.sonatype.nexus.repository.httpbridge.internal.ExhaustRequestFilter.doFilter(ExhaustRequestFilter.java:71)
    at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
    at com.sonatype.nexus.licensing.internal.LicensingRedirectFilter.doFilter(LicensingRedirectFilter.java:108)
    at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
    at com.codahale.metrics.servlet.AbstractInstrumentedFilter.doFilter(AbstractInstrumentedFilter.java:97)
    at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
    at org.sonatype.nexus.internal.web.ErrorPageFilter.doFilter(ErrorPageFilter.java:68)
    at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
    at org.sonatype.nexus.internal.web.EnvironmentFilter.doFilter(EnvironmentFilter.java:102)
    at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
    at org.sonatype.nexus.internal.web.HeaderPatternFilter.doFilter(HeaderPatternFilter.java:98)
    at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
    at com.google.inject.servlet.DynamicFilterPipeline.dispatch(DynamicFilterPipeline.java:104)
    at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:135)
    at org.sonatype.nexus.bootstrap.osgi.DelegatingFilter.doFilter(DelegatingFilter.java:73)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1629)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
    at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.java:175)
    at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
    at org.eclipse.jetty.server.Server.handle(Server.java:530)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:347)
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:256)
    at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
    at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:247)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
    at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:382)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NoClassDefFoundError: org/tukaani/xz/XZInputStream
    at net.staticsnow.nexus.repository.apt.internal.hosted.AptHostedFacet.ingestAsset(AptHostedFacet.java:144)
    at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:56)
    at org.sonatype.nexus.transaction.TransactionInterceptor.invoke(TransactionInterceptor.java:54)
    at net.staticsnow.nexus.repository.apt.internal.hosted.AptHostedHandler.handle(AptHostedHandler.java:59)
    at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
    at net.staticsnow.nexus.repository.apt.internal.gpg.AptSigningHandler.handle(AptSigningHandler.java:43)
    at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
    at net.staticsnow.nexus.repository.apt.internal.snapshot.AptSnapshotHandler.handle(AptSnapshotHandler.java:60)
    at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
    at org.sonatype.nexus.repository.storage.UnitOfWorkHandler.handle(UnitOfWorkHandler.java:39)
    at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
    at org.sonatype.nexus.repository.view.handlers.ContentHeadersHandler.handle(ContentHeadersHandler.java:44)
    at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
    at org.sonatype.nexus.repository.http.PartialFetchHandler.handle(PartialFetchHandler.java:55)
    at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
    at org.sonatype.nexus.repository.view.handlers.ConditionalRequestHandler.handle(ConditionalRequestHandler.java:72)
    at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
    at org.sonatype.nexus.repository.view.handlers.ExceptionHandler.handle(ExceptionHandler.java:44)
    at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
    at org.sonatype.nexus.repository.security.SecurityHandler.handle(SecurityHandler.java:52)
    at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
    at org.sonatype.nexus.repository.view.handlers.TimingHandler.handle(TimingHandler.java:46)
    at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
    at org.sonatype.nexus.repository.view.Context.start(Context.java:114)
    at org.sonatype.nexus.repository.view.Router.dispatch(Router.java:63)
    at org.sonatype.nexus.repository.view.ConfigurableViewFacet.dispatch(ConfigurableViewFacet.java:52)
    at org.sonatype.nexus.repository.view.ConfigurableViewFacet.dispatch(ConfigurableViewFacet.java:43)
    at org.sonatype.nexus.repository.httpbridge.internal.ViewServlet.dispatchAndSend(ViewServlet.java:211)
    at org.sonatype.nexus.repository.httpbridge.internal.ViewServlet.doService(ViewServlet.java:173)
    at org.sonatype.nexus.repository.httpbridge.internal.ViewServlet.service(ViewServlet.java:126)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
    at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:286)
    at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:276)
    at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:181)
    at com.google.inject.servlet.DynamicServletPipeline.service(DynamicServletPipeline.java:71)
    at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:112)
    at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61)
    at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
    at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
    at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
    at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
    at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
    at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
    at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
    at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
    at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
    at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
    at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
    at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
    at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)
    at org.sonatype.nexus.security.SecurityFilter.executeChain(SecurityFilter.java:85)
    at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
    at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
    at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
    at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)
    at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
    at org.sonatype.nexus.security.SecurityFilter.doFilterInternal(SecurityFilter.java:101)
    at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
    at org.sonatype.nexus.repository.httpbridge.internal.ExhaustRequestFilter.doFilter(ExhaustRequestFilter.java:71)
    at com.sonatype.nexus.licensing.internal.LicensingRedirectFilter.doFilter(LicensingRedirectFilter.java:108)
    at com.codahale.metrics.servlet.AbstractInstrumentedFilter.doFilter(AbstractInstrumentedFilter.java:97)
    at org.sonatype.nexus.internal.web.ErrorPageFilter.doFilter(ErrorPageFilter.java:68)
    at org.sonatype.nexus.internal.web.EnvironmentFilter.doFilter(EnvironmentFilter.java:102)
    at org.sonatype.nexus.internal.web.HeaderPatternFilter.doFilter(HeaderPatternFilter.java:98)
    ... 35 common frames omitted
Caused by: java.lang.ClassNotFoundException: org.tukaani.xz.XZInputStream not found by org.apache.commons.compress [191]
    at org.apache.felix.framework.BundleWiringImpl.findClassOrResourceByDelegation(BundleWiringImpl.java:1550)
    at org.apache.felix.framework.BundleWiringImpl.access$200(BundleWiringImpl.java:79)
    at org.apache.felix.framework.BundleWiringImpl$BundleClassLoader.loadClass(BundleWiringImpl.java:1958)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 99 common frames omitted
2018-05-01 10:00:31,788+0000 DEBUG [qtp179
derjohn commented 6 years ago

Hm, it looks like I have xz V 1.5 : ./sonatype/nexus/system/org/tukaani/xz/1.5/xz-1.5.jar

According to Stackoverflow https://stackoverflow.com/questions/28229095/apache-commons-compress-using-7zip we need at least 1.6 ...

It looks like, the 1.5 comes with the nexus 3.10.0 release, so changing the pom.xml here in the plugin repo doesn't fetch it. (It's inside nexus-3.10.0-04-unix.tar.gz .... an outdated .m2 cache form the release engineer ?)

jsoref commented 6 years ago

Fwiw, as a workaround, you can: download the file.deb, and then manually perform:

ar xf file.deb
mkdir c d
(cd c; tar xf ../control.xz; tar czf ../control.gz *)
(cd d; tar xf ../data.xz; tar czf ../data.gz *)
ar -r file.deb debian-binary control.tar.gz data.tar.gz

And then publish the file.

I've done that, because I needed a repo w/ the file and haven't had the energy to fight the java side (although I will probably look at some point).

derjohn commented 6 years ago

@jsoref Thx for that workaround! I am quite sure, the issue will be resolved with nexus 3.12, as the XZ / Tukani ans commons compress will be updated. With 3.11 we still have the old s*t inside. I was able to build a nexus-3.12 for testing that, but I have no clue, how the nexus-base-template.foo is converted into nexus-3.12xxxx.unix.tar.gz. Both look similar inside , but in detail certain things are missing (e.g. the initscripts) and I was not able to find that "transformation step" in any open repo on sonatype.

@DarthHater Can you help? Is there maybe a download URL for the 3.12-SNAPSHOT-unix.tar.gz ?

mpoindexter commented 6 years ago

The pom file does specify that xz is a dependency, and Nexus 3.11 seems to ship with version 1.8. Maybe try adding

<bundle>mvn:org.tukaani/xz/1.8</bundle>

to this bit in the features file:

+ <feature name="nexus-repository-apt" description="net.staticsnow:nexus-repository-apt" version="1.0.6">
+     <details>net.staticsnow:nexus-repository-apt</details>
+     <bundle>mvn:net.staticsnow/nexus-repository-apt/1.0.6</bundle>
+     <bundle>mvn:org.apache.commons/commons-compress/1.16.1</bundle>
+ </feature>

where it looks like 

+ <feature name="nexus-repository-apt" description="net.staticsnow:nexus-repository-apt" version="1.0.6">
+     <details>net.staticsnow:nexus-repository-apt</details>
+     <bundle>mvn:net.staticsnow/nexus-repository-apt/1.0.6</bundle>
+     <bundle>mvn:org.apache.commons/commons-compress/1.16.1</bundle>
+     <bundle>mvn:org.tukaani/xz/1.8</bundle>
+ </feature>

Not sure if it will work, but if it does we can update the readme

DarthHater commented 6 years ago

@derjohn snapshots I believe are not public, if they were they would be on RSO.

derjohn commented 6 years ago

@mpoindexter I tried what you proposed - nexus started, but I was still not able to upload the Debian packages. But I don't see the stracktrace anymore, just "curl: (22) The requested URL returned error: 400 Invalid Debian package supplied". Besides the 1.8 library do we need to allow the packing algo in the plugin code somewhere?

mpoindexter commented 6 years ago

In addition to needing the XZ bundle declared in the feature, there was a stupid bug (I forgot a break statement in a switch) in the parsing code. I've fixed it, it works now. Thanks for the bug report and testing out the feature changes.