sonatype-nexus-community / nexus-repository-composer

Composer support for the Nexus Repository Manager (work in progress!)
Eclipse Public License 1.0
220 stars 87 forks source link

Implement Composer group repositories #14

Closed fjmilens3 closed 6 years ago

fjmilens3 commented 6 years ago

Fixes #8 by adding basic support for merging (flattened) packages.json files and provider JSON files, along with the standard group handler for obtaining the correct zip file based on repository ordering.

TheBay0r commented 6 years ago

Hey @fjmilens3 ! I just gave your branch a shot. I have composer (group) now in the recipes, but I can not select it. Not sure if I'm to early with testing it, but just wanted to let you know 🙂

fjmilens3 commented 6 years ago

@TheBay0r, that concerns me somewhat because I'm able to select the composer (group) repo and create one without an issue. A couple of questions:

  1. Do you get any error message/popup/log entry when you try to click on the "composer (group)" entry in the list?
  2. Have you tried doing a hard refresh in your browser or clearing the browser cache?
TheBay0r commented 6 years ago

@fjmilens3, you are right. I was able to create a grouped repository now. I decided to combine composer-proxy and composer-hosted and decided to just name it composer. In the user interface it is fine and I can see all the packages from proxy and hosted.

When I try to do a composer install using the grouped repository only, I retrieve a NullPointerException. Accessing the same package via composer-hosted works

2018-04-18 08:07:52,287+0000 WARN  [qtp1963253024-60] *UNKNOWN org.sonatype.nexus.repository.httpbridge.internal.ViewServlet - Failure servicing: GET /repository/composer/{private_vendor}/{private_package}/{version}/{filename}.zip
java.lang.NullPointerException: null
    at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:770)
    at org.sonatype.nexus.repository.composer.internal.ComposerProxyFacetImpl.getZipballUrl(ComposerProxyFacetImpl.java:174)
    at org.sonatype.nexus.repository.composer.internal.ComposerProxyFacetImpl.getUrl(ComposerProxyFacetImpl.java:133)
    at org.sonatype.nexus.repository.proxy.ProxyFacetSupport.fetch(ProxyFacetSupport.java:345)
    at org.sonatype.nexus.repository.proxy.ProxyFacetSupport.doGet(ProxyFacetSupport.java:219)
    at org.sonatype.nexus.repository.proxy.ProxyFacetSupport.lambda$1(ProxyFacetSupport.java:208)
    at org.sonatype.nexus.repository.proxy.Cooperation$CooperatingFuture.download(Cooperation.java:259)
    at org.sonatype.nexus.repository.proxy.Cooperation.download(Cooperation.java:191)
    at org.sonatype.nexus.repository.proxy.Cooperation.cooperate(Cooperation.java:90)
    at org.sonatype.nexus.repository.proxy.ProxyFacetSupport.get(ProxyFacetSupport.java:200)
    at org.sonatype.nexus.repository.proxy.ProxyHandler.handle(ProxyHandler.java:49)
    at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
    at org.sonatype.nexus.repository.storage.UnitOfWorkHandler.handle(UnitOfWorkHandler.java:39)
    at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
    at org.sonatype.nexus.repository.view.handlers.ContentHeadersHandler.handle(ContentHeadersHandler.java:44)
    at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
    at org.sonatype.nexus.repository.http.PartialFetchHandler.handle(PartialFetchHandler.java:55)
    at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
    at org.sonatype.nexus.repository.view.handlers.ConditionalRequestHandler.handle(ConditionalRequestHandler.java:72)
    at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
    at org.sonatype.nexus.repository.cache.NegativeCacheHandler.handle(NegativeCacheHandler.java:50)
    at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
    at org.sonatype.nexus.repository.assetdownloadcount.internal.AssetDownloadCountContributedHandler.handle(AssetDownloadCountContributedHandler.java:53)
    at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
    at org.sonatype.nexus.repository.view.handlers.HandlerContributor.handle(HandlerContributor.java:67)
    at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
    at org.sonatype.nexus.repository.view.handlers.ExceptionHandler.handle(ExceptionHandler.java:44)
    at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
    at org.sonatype.nexus.repository.security.SecurityHandler.handle(SecurityHandler.java:52)
    at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
    at org.sonatype.nexus.repository.view.Context$proceed.call(Unknown Source)
    at org.sonatype.nexus.repository.composer.internal.ComposerRecipeSupport$_closure1.doCall(ComposerRecipeSupport.groovy:110)
    at sun.reflect.GeneratedMethodAccessor221.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
    at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1087)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
    at groovy.lang.Closure.call(Closure.java:414)
    at org.codehaus.groovy.runtime.ConvertedClosure.invokeCustom(ConvertedClosure.java:54)
    at org.codehaus.groovy.runtime.ConversionHandler.invoke(ConversionHandler.java:124)
    at com.sun.proxy.$Proxy157.handle(Unknown Source)
    at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
    at org.sonatype.nexus.repository.view.handlers.TimingHandler.handle(TimingHandler.java:46)
    at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
    at org.sonatype.nexus.repository.view.Context.start(Context.java:114)
    at org.sonatype.nexus.repository.view.Router.dispatch(Router.java:63)
    at org.sonatype.nexus.repository.view.ConfigurableViewFacet.dispatch(ConfigurableViewFacet.java:52)
    at org.sonatype.nexus.repository.group.GroupHandler.getFirst(GroupHandler.java:120)
    at org.sonatype.nexus.repository.group.GroupHandler.doGet(GroupHandler.java:97)
    at org.sonatype.nexus.repository.group.GroupHandler.handle(GroupHandler.java:81)
    at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
    at org.sonatype.nexus.repository.assetdownloadcount.internal.AssetDownloadCountContributedHandler.handle(AssetDownloadCountContributedHandler.java:53)
    at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
    at org.sonatype.nexus.repository.view.handlers.HandlerContributor.handle(HandlerContributor.java:67)
    at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
    at org.sonatype.nexus.repository.view.handlers.ExceptionHandler.handle(ExceptionHandler.java:44)
    at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
    at org.sonatype.nexus.repository.security.SecurityHandler.handle(SecurityHandler.java:52)
    at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
    at org.sonatype.nexus.repository.view.Context$proceed.call(Unknown Source)
    at org.sonatype.nexus.repository.composer.internal.ComposerRecipeSupport$_closure1.doCall(ComposerRecipeSupport.groovy:110)
    at sun.reflect.GeneratedMethodAccessor221.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
    at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1087)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
    at groovy.lang.Closure.call(Closure.java:414)
    at org.codehaus.groovy.runtime.ConvertedClosure.invokeCustom(ConvertedClosure.java:54)
    at org.codehaus.groovy.runtime.ConversionHandler.invoke(ConversionHandler.java:124)
    at com.sun.proxy.$Proxy157.handle(Unknown Source)
    at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
    at org.sonatype.nexus.repository.view.handlers.TimingHandler.handle(TimingHandler.java:46)
    at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
    at org.sonatype.nexus.repository.view.Context.start(Context.java:114)
    at org.sonatype.nexus.repository.view.Router.dispatch(Router.java:63)
    at org.sonatype.nexus.repository.view.ConfigurableViewFacet.dispatch(ConfigurableViewFacet.java:52)
    at org.sonatype.nexus.repository.view.ConfigurableViewFacet.dispatch(ConfigurableViewFacet.java:43)
    at org.sonatype.nexus.repository.httpbridge.internal.ViewServlet.dispatchAndSend(ViewServlet.java:211)
    at org.sonatype.nexus.repository.httpbridge.internal.ViewServlet.doService(ViewServlet.java:173)
    at org.sonatype.nexus.repository.httpbridge.internal.ViewServlet.service(ViewServlet.java:126)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
    at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:286)
    at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:276)
    at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:181)
    at com.google.inject.servlet.DynamicServletPipeline.service(DynamicServletPipeline.java:71)
    at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
    at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:112)
    at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
    at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61)
    at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
    at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
    at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
    at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
    at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
    at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
    at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
    at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
    at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
    at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
    at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
    at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
    at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)
    at org.sonatype.nexus.security.SecurityFilter.executeChain(SecurityFilter.java:85)
    at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
    at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
    at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
    at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)
    at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
    at org.sonatype.nexus.security.SecurityFilter.doFilterInternal(SecurityFilter.java:101)
    at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
    at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
    at org.sonatype.nexus.repository.httpbridge.internal.ExhaustRequestFilter.doFilter(ExhaustRequestFilter.java:71)
    at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
    at com.sonatype.nexus.licensing.internal.LicensingRedirectFilter.doFilter(LicensingRedirectFilter.java:108)
    at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
    at com.codahale.metrics.servlet.AbstractInstrumentedFilter.doFilter(AbstractInstrumentedFilter.java:97)
    at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
    at org.sonatype.nexus.internal.web.ErrorPageFilter.doFilter(ErrorPageFilter.java:68)
    at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
    at org.sonatype.nexus.internal.web.EnvironmentFilter.doFilter(EnvironmentFilter.java:102)
    at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
    at org.sonatype.nexus.internal.web.HeaderPatternFilter.doFilter(HeaderPatternFilter.java:98)
    at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
    at com.google.inject.servlet.DynamicFilterPipeline.dispatch(DynamicFilterPipeline.java:104)
    at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:135)
    at org.sonatype.nexus.bootstrap.osgi.DelegatingFilter.doFilter(DelegatingFilter.java:73)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1629)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
    at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.java:175)
    at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
    at org.eclipse.jetty.server.Server.handle(Server.java:530)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:347)
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:256)
    at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
    at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:247)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
    at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:382)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
    at java.lang.Thread.run(Thread.java:748)
fjmilens3 commented 6 years ago

When I try to do a composer install using the grouped repository only, I retrieve a NullPointerException. Accessing the same package via composer-hosted works

@TheBay0r, this is a problem caused by the hackery I did in the proxy implementation to look up the provider JSON for a particular vendor and project in order to obtain the dist url. This probably wouldn't happen if you had your hosted repo before your proxy repo in the configuration, but it's still something we need to fix because it manifests a larger issue. The short version is that we're not robustly dealing with certain download situations in proxy (where a file that does not exist is requested, but it still checks upstream to reference any existing provider JSON first).

For some background, Nexus Repo does group merge in two ways. For metadata, indices, and the like, it has to fetch all upstream responses from the member repos and merge them together. For actual content, we basically just hit each repository in succession until we find something and return that as the result. (You can see https://github.com/sonatype/nexus-public/blob/ffc3d00c05778ee415bfa69135d982bac2f38450/components/nexus-repository/src/main/java/org/sonatype/nexus/repository/group/GroupHandler.java and subclasses as the crown jewel of this particular dynasty.)

In this case, you're running into a quirk of the Composer implementation I've done. The merge for all the metadata/indices works just fine, but when you're to fetching the zip archive, the code in proxy goes and tries to find the provider JSON to obtain the url indirectly in case it hasn't been stored yet or needs updated (this is largely a result of how the proxy support works in Nexus Repo). In that case, we're not dealing with the missing provider content gracefully, and the result is the stack trace you see before you.

I have a pretty good idea of how to fix this as some other formats (npm) end up with similar problems, fortunately, and as today is our "improvement day" I think I can get something together quickly. I can't thank you enough for how involved you've been in contributing feedback and testing, it counts a lot to know what we do is valued!

fjmilens3 commented 6 years ago

@TheBay0r, when you get a chance, give it a try with https://github.com/sonatype-nexus-community/nexus-repository-composer/pull/14/commits/95a20b566a540126c33b10ede3c95758bcf7f2f9 and see if that works any better for your scenario.

TheBay0r commented 6 years ago

@fjmilens3 I'm sry that it took me so long to have another look at it. And I'm happy when my little testing helps you. Wish I could do even more, it is a great plugin 🙂

Downloading the packages.json works with your fix. I tried downloading the file manually and just had a quick look and also tried installing the dependencies using the grouped repository only. I didn't have any problem that would be related to the composer-group branch! So your fix looks good for me 🙂

TheBay0r commented 6 years ago

I'm just about to build that branch and will test it this week. I'll let you know how it looks as soon as possible 🙂

TheBay0r commented 6 years ago

@fjmilens3 it looks like there is some regression. 🤔

The composer-lock is missing a couple of informations now, that were there in the composer-hosted branch still. (Although the information is coming from composer-proxy). I also tried just using the composer-proxy without the composer-group feature as well.

As you can see as an example for phpunit in the screenshot it is missing the information of the binary, which causes the vendor/bin folder to not be created and therefore vendor/bin/phpunit to not be available.

Also the extra part is missing were I'm not sure what it is used for, next to some meta information below.

What is also a little bit concerning is, that we lost the reference for the package. Which means if somebody (for whatever) reason would puplish the same version with a different commit hash, composer wouldn't realise that there was a change.

screen shot 2018-06-07 at 09 37 45

Hope it is readable enough for you. If you have any questions, just let me know.

I even deployed latest master again to verify that this information wasn't missing after adding the composer-hosted feature.

fjmilens3 commented 6 years ago

@TheBay0r, I've taken a quick look at this and made what I think are the necessary changes, but I have not tested it beyond unit testing. I'll probably try and take a look at this for further testing sometime next week as we're traveling this weekend, but you may end up getting to it before I do (and probably have more context than myself by far, in that I'd trust you more than I'd trust myself to find things wrong with it).

As far as what happened here, as part of working on group I tried to be more consistent in terms of how we handled different fields in building and merging JSON, and the minimal subset that I chose was too minimal to support some of these additional features; when I do test this I'm usually testing against very trivial composer.json files rather than building a huge project, so a lot of that sneaks through. Note that to be consistent with adding (or re-adding) some of these fields, I've also gone ahead and added them for hosted since they were missing there.

Also note that I added your name to our contributors list on https://github.com/sonatype-nexus-community/nexus-repository-composer/pull/14/commits/b3e9b805660f046d7986c05c66d5058ba89f3c98 because of all the QA work you've done on this project over these months; if you want me to take that out I will, but I'd like to recognize your support if you're okay with that.

TheBay0r commented 6 years ago

I can verify that the regression issue is seems to be resolved, although I currently have problems building one of our containers. I'll have to take a deeper look and will let you know if it is related to this branch.

Something to clarify in the meanwhile. I understood you added the missing fields to composer-hosted as well. But when I have a look at one of our private packages, it doesn't look like it. See screenshot: screen shot 2018-06-08 at 11 37 14 (sorry that I anonymise a lot, guess it is still readable 🙂 )

TheBay0r commented 6 years ago

Okay… it seems to be that the failing build is related to the "target-dir" property still missing the JSON.

screen shot 2018-06-08 at 12 01 07
fjmilens3 commented 6 years ago

Something to clarify in the meanwhile. I understood you added the missing fields to composer-hosted as well. But when I have a look at one of our private packages, it doesn't look like it.

@TheBay0r, I will take a look and test some hosted uploads myself to see what's going on. The changes only affect newly-deployed packages, was this one that you deployed after these changes or one that you had around previously in your Nexus storage?

TheBay0r commented 6 years ago

The changes only affect newly-deployed packages, was this one that you deployed after these changes or one that you had around previously in your Nexus storage?

I already thought that question would come up, so I tested a package that was there before and also a package I just uploaded. Although I am not sure if it might interfere because I just replaced an existing version?

fjmilens3 commented 6 years ago

I already thought that question would come up, so I tested a package that was there before and also a package I just uploaded. Although I am not sure if it might interfere because I just replaced an existing version?

@TheBay0r:

It shouldn't, if it did that, then it's a bug (my first thought would center on whether or not we're correctly handling update events in terms of regenerating the metadata files).

For what it's worth, I've tried it locally with a new upload and it seems to be working correctly (I've also added a few more fields to be stored), so I'm wondering if there is something more going on here.

Could you put together a sample composer.json or entire package, suitably anonymized, but based on yours, so that I can use that in my testing (for upload purposes)? I only had a few minutes to look at this between tasks today at work, so that might help me work it through when I do have more time.

TheBay0r commented 6 years ago

@fjmilens3 I just tried a few more things. Re-uploading an existing package -> Downloading the JSON from /p/vendor/{package}.json => None of the new fields is available

Uploading a completely new version of the package -> Downloading JSON from /p/vendor/{package}.json => All previous versions also have the new fields, not just the one just uploaded

But I still have to have a look at fcbc942937489a63e6f482bfca9a9dc8a261286c because those fields were obviously still missing.

Providing you with a full composer.json might take me until Monday

TheBay0r commented 6 years ago

I guess one of the best composer.json you can find is actually the one from the composer repo itself. I just extendid it slightly by adding the repository field and adding an additional way for scripts. Also I removed the names. But I guess this covers most of the fields 🤔

{
    "name": "composer/composer",
    "description": "Composer helps you declare, manage and install dependencies of PHP projects, ensuring you have the right stack everywhere.",
    "keywords": ["package", "dependency", "autoload"],
    "homepage": "https://getcomposer.org/",
    "type": "library",
    "license": "MIT",
    "authors": [
        {
            "name": "Not Person",
            "email": "not.person@examplehomepagenotvalid.it",
            "homepage": "http://www.examplehomepagenotvalid.it"
        },
        {
            "name": "Another Not Person",
            "email": "another@examplehomepagenotvalid.it",
            "homepage": "http://www.examplehomepagenotvalid.it"
        }
    ],
    "support": {
        "irc": "irc://irc.examplehomepagenotvalid.it/composer",
        "issues": "https://github.com/composer/composer/issues"
    },
    "repositories": [
        {
            "type": "git",
            "url": "https://github.com/foobar/intermediate.git",
            "no-api": true
        }
    ],
    "require": {
        "php": "^5.3.2 || ^7.0",
        "justinrainbow/json-schema": "^3.0 || ^4.0 || ^5.0",
        "composer/ca-bundle": "^1.0",
        "composer/semver": "^1.0",
        "composer/spdx-licenses": "^1.2",
        "composer/xdebug-handler": "^1.1",
        "seld/jsonlint": "^1.4",
        "symfony/console": "^2.7 || ^3.0 || ^4.0",
        "symfony/finder": "^2.7 || ^3.0 || ^4.0",
        "symfony/process": "^2.7 || ^3.0 || ^4.0",
        "symfony/filesystem": "^2.7 || ^3.0 || ^4.0",
        "seld/phar-utils": "^1.0",
        "psr/log": "^1.0"
    },
    "require-dev": {
        "phpunit/phpunit": "^4.8.35 || ^5.7",
        "phpunit/phpunit-mock-objects": "^2.3 || ^3.0"
    },
    "conflict": {
        "symfony/console": "2.8.38"
    },
    "config": {
        "platform": {
            "php": "5.3.9"
        }
    },
    "suggest": {
        "ext-zip": "Enabling the zip extension allows you to unzip archives",
        "ext-zlib": "Allow gzip compression of HTTP requests",
        "ext-openssl": "Enabling the openssl extension allows you to access https URLs for repositories and packages"
    },
    "autoload": {
        "psr-4": { "Composer\\": "src/Composer" }
    },
    "autoload-dev": {
        "psr-4": { "Composer\\Test\\": "tests/Composer/Test" }
    },
    "bin": ["bin/composer"],
    "extra": {
        "branch-alias": {
            "dev-master": "1.7-dev"
        }
    },
    "scripts": {
        "test": "phpunit",
        "custom-cmd": [
            "chmod -R 777 *"
        ]
    }
}
TheBay0r commented 6 years ago

Also had a look at fcbc942937489a63e6f482bfca9a9dc8a261286c. It looks very good. The only thing is, that the .json generated in nexus is only updated when a new version is uploaded, but not when a existing version should be replaced 🤔

fjmilens3 commented 6 years ago

@TheBay0r:

I guess one of the best composer.json you can find is actually the one from the composer repo itself. I just extendid it slightly by adding the repository field and adding an additional way for scripts. Also I removed the names. But I guess this covers most of the fields

Thanks for getting that together, I ran it through once and it looks like we're in good shape. It did make me realize that I'm probably not handling merge correctly with respect to the time in for provider JSON files in groups, which I changed in https://github.com/sonatype-nexus-community/nexus-repository-composer/pull/14/commits/2780bc307bc79fd748477c83391254f1d12e7108 (if we use the time string from the JSON we're merging, we're more stable in terms of timestamps and UIDs than if we don't, in which case they'll be different for each and every request!).

Also had a look at fcbc942. It looks very good.

I appreciate you retesting those changes; at this point I'm going to merge this PR in as it should be sufficient for group.

The only thing is, that the .json generated in nexus is only updated when a new version is uploaded, but not when a existing version should be replaced

I want to do some more investigation on this as time permits, for now I'm breaking it out into https://github.com/sonatype-nexus-community/nexus-repository-composer/issues/20 so we can discuss it more there; whatever is going on here is more related to hosted than to group so should probably be worked separately.

Thanks for all your help on this!

sonatypecla[bot] commented 4 years ago

Thanks for the contribution! Unfortunately we can't verify if the committer(s), Frederick John Milens III fmilens@sonatype.com, signed the CLA because they have not associated their commits with their GitHub user. Please follow these instructions to associate your commits with your GitHub user. Then sign the Sonatype Contributor License Agreement and this Pull Request will be revalidated.

sonatypecla[bot] commented 3 years ago

Thanks for the contribution! Unfortunately we can't verify if the committer(s), Frederick John Milens III fmilens@sonatype.com, signed the CLA because they have not associated their commits with their GitHub user. Please follow these instructions to associate your commits with your GitHub user. Then sign the Sonatype Contributor License Agreement and this Pull Request will be revalidated.