search

sonatype / nexus-ruby-support

26 stars 7 forks source link

group repo + specs.4.8.gz + memory usage + garbage collection #28

Closed flyinprogrammer closed 10 years ago

flyinprogrammer commented 11 years ago

So - we've noticed that as we accumulate more gems the length of time to build the specs.4.8.gz in our group repo (comprised of 2 internal (hosted) repos and rubygems.org (proxy)) increases. we were wondering what type of caching policy around this file exists, how we might be able to tweak this caching policy, and maybe some information about how this file is built?

The bigger problem is that it seems we are able to completely seize nexus from running by simply doing N number of 'bundle update's which is an issue. Where N is an unknown number to date, but similar to how many licks it takes to get to the center of a lollipop :smile:

And I apologize for opening yet-another-issue; if there's any information or data I can capture for you, please don't hesitate to ask, I'll do my absolute best to get it asap.

flyinprogrammer commented 11 years ago

https://www.evernote.com/shard/s9/sh/4607bf08-e521-49fb-8ce7-e1d7281f825a/b629351609204c488ea6b24a2575a22f

flyinprogrammer commented 11 years ago

This is the kind of trend we see; only the drops in usuage only occur when we manually perform a GC via jconsole

http://puu.sh/3J2vb/5985ed74e7.png

mkristian commented 11 years ago

rubygems.org proxy will download the spec.4.8.gz after an expiration time - you can configure that in the webinterface under "metadata max age".

I guess you are using the latest version of the plugin !?

you could do me favour of testing that memory "leak" against the gemsproxy only - the group should not cache anything for the bundler API. it does merge the spec.4.8.gz files from all its members and writes it out on disk. that is only used by bundler if the bundler API is too slow (which can happen).

that info would help me to start on that tomorrow morning . . .

flyinprogrammer commented 11 years ago

So I've done some load testing, hitting nexus 1000+ times against both our group repo and hosted repo, in an attempted multi-threaded fashion with MRI and jruby; none of them generated memory spikes or any real cpu load. However, after i've stopped testing i've noticed that we have 300+ threads remaining open, and our effectiveness of GC is gone, and we've crept up to 2.5gb of memory usage.

I'm beginning to think maybe this isn't a plugin problem, but a Nexus problem.

How is the pluggin written to handle getting asked for the group specs gz simultaneously 25 times? does it block on the request, or would it go to build the gz from scratch for each request? or what rules exist for this operation: 'it does merge the spec.4.8.gz files from all its members and writes it out on disk.'

mkristian commented 11 years ago

nexus uses read-write lock on any files. for the group repo it first needs to read the 'attributes' of its group members, i.e. the timestamp and if anything is newer then the cached spec.4.8.gz file on disk then it will be regenerated. otherwise it just reads the file from disk. just have a look at the timestamps of those files in the sonatype-work/nexus/storage/ directory (or the attribute files /usr/local/sonatype-work/nexus/storage/gems/.nexus/attributes/ here the repo name is gems)

when I do concurrent 'gem list -ra maven-tools' (which triggers specs.4.8.gz download) or 'bundle update' on any group or hosted or proxy I do see a slight increase in memory but it shrinks again after some idle time. I am using nexus-2.5.0-04 and the current HEAD for the plugin. so currently I am not able to reproduce your observations.

what max age of meta-data do you have on the proxy ? what nexus version ?

but I will have look at the situation with jconsole as well

flyinprogrammer commented 11 years ago

So we were using 2.2-01, with the latest plugin. Now we're using 2.5.1-01, and I'm running on an instance separate from our java repos after monitoring for 6hrs, this issue is not with this plugin, but definitely with something else. We are hosting 1000+ gems with little issue.

Sorry for opening this, I really thought I might have found something.

mkristian commented 11 years ago

please reopen that issue if you see the "leak" coming back - maybe it happens with having both java and ruby on one nexus instance. there were leaks in the past with the ruby plugin so who knows what else is hidden somewhere ;):

flyinprogrammer commented 11 years ago

So we've been running nexus with the ruby plugin by itself for 5 days or so now; I have noticed that we get a couple of these every so often:

2013-07-30 21:23:32 ERROR [qtp631120219-1050] anonymous org.sonatype.nexus.rest.ContentPlexusResource - Got exception during processing request "GET http://nexus.pilot.cs.ha:8081/nexus/content/groups/gems/api/v1/dependencies?gems=selenium-webdriver,activeresource,gem-release,active_support,geminabox,nexus,treetop,httparty,mail,hoe,weakling,mini_portile,net-ssh,hydra-head,jettywrapper,active-fedora,rails,om,solrizer,rsolr,blacklight,rubydora,nom-xml,mime-types,selenium-client,rspec-expectations,rspec-mocks,term-ansicolor,diff-lcs,polyglot,json_pure,gherkin,multi_json,multi_test,i18n,scribe,parallel,activemodel,arel,tzinfo,activerecord-deprecated_finders,activerecord-jdbc-adapter,jdbc-jtds,minitest,thread_safe,memcache-client": 
java.lang.NullPointerException: null

And eventually over time we start running into these:

2013-07-30 21:56:51 ERROR [qtp631120219-1123] anonymous org.sonatype.nexus.rest.ContentPlexusResource - Got exception during processing request "GET http://nexus.pilot.cs.ha:8081/nexus/content/groups/gems/specs.4.8.gz": 
org.jruby.embed.InvokeFailedException: java.lang.OutOfMemoryError: Java heap space
    at org.jruby.embed.internal.EmbedRubyObjectAdapterImpl.call(EmbedRubyObjectAdapterImpl.java:319) ~[na:na]
    at org.jruby.embed.internal.EmbedRubyObjectAdapterImpl.callMethod(EmbedRubyObjectAdapterImpl.java:169) ~[na:na]
    at org.jruby.embed.ScriptingContainer.callMethod(ScriptingContainer.java:1431) ~[na:na]
    at org.sonatype.nexus.ruby.DefaultRubygemsGateway.mergeSpecs(DefaultRubygemsGateway.java:151) ~[na:na]
    at org.sonatype.nexus.plugins.ruby.group.GroupRubygemsFacade.mergeSpecsIndex(GroupRubygemsFacade.java:36) ~[na:na]
    at org.sonatype.nexus.plugins.ruby.fs.RubyFSLocalRepositoryStorage.storeSpecsIndices(RubyFSLocalRepositoryStorage.java:356) ~[na:na]
    at org.sonatype.nexus.plugins.ruby.group.DefaultRubyGroupRepository.retrieveItem(DefaultRubyGroupRepository.java:129) ~[na:na]
    at org.sonatype.nexus.proxy.router.DefaultRepositoryRouter.retrieveItem(DefaultRepositoryRouter.java:176) ~[nexus-core-2.5.1-01.jar:2.5.1-01]
    at org.sonatype.nexus.rest.AbstractResourceStoreContentPlexusResource.get(AbstractResourceStoreContentPlexusResource.java:174) ~[nexus-restlet1x-plugin-2.5.1-01.jar:2.5.1-01]
    at org.sonatype.plexus.rest.resource.RestletResource.represent(RestletResource.java:269) [plexus-restlet-bridge-1.25.jar:1.25]
    at org.sonatype.nexus.rest.NexusRestletResource.represent(NexusRestletResource.java:43) [nexus-restlet1x-plugin-2.5.1-01.jar:2.5.1-01]
    at org.restlet.resource.Resource.getRepresentation(Resource.java:302) [org.restlet-1.1.6-SONATYPE-5348-V8.jar:na]
    at org.restlet.resource.Resource.handleGet(Resource.java:464) [org.restlet-1.1.6-SONATYPE-5348-V8.jar:na]
    at org.restlet.Finder.handle(Finder.java:353) [org.restlet-1.1.6-SONATYPE-5348-V8.jar:na]
    at org.restlet.Filter.doHandle(Filter.java:150) [org.restlet-1.1.6-SONATYPE-5348-V8.jar:na]

I'm using the latest plugin code; and nexus 2.5.1-01.

Unfortunately we're running in ERROR log mode; would DEBUG help you? or what can I do to help track down the cause of this?

mkristian commented 11 years ago

for the NullPointerException a debug log would help or maybe there is the stacktrace. the request itself does not produce such an error with me.

for the OutOfMemory at least there is the indication that is has something to do with DefaultRubygemsGateway.mergeSpecs

flyinprogrammer commented 11 years ago

ok -- i've setup our server to 'debug' -- i'll watch for these to happen again, and get back to you ASAP

mkristian commented 11 years ago

commit 8168e5a21deb6e336bb800ecfd000e2e63ce45a0 did indeed fix some leaks with mergeSpecs - maybe that was your leak ;)

flyinprogrammer commented 11 years ago

I'll rebuild the plugin; and test with this new code :+1:

flyinprogrammer commented 11 years ago

your new code works :smiley: -- i'll keep watching the logs

is it possible that the process might not be able to ask for more heap while processing:
http://puu.sh/3QfpO/032385e461.png That spike is from a specs.4.8.gz request from our Gems group repo. You can see it runs dangerously close to the heap max for a couple seconds.

flyinprogrammer commented 11 years ago

attached is a zipped log; download the png; change it to .zip; decompress; have fun :wink: nexus log zipped

And prior to 09:00:00 is before the patch; after 09:00:00 is with the patch

mkristian commented 11 years ago

is the later with my patch ?

it does show almost "normal" until those java.rmi.NotBoundException

how do you create such "problems" in such a short time ?

flyinprogrammer commented 11 years ago

we have builds that are hammering Nexus with 'bundle update' every couple of minutes;

java.rmi.NotBoundException <--- interesting; we enabled jmxremote so that we could monitor the process; I wonder if this is causing that issue.

Before setting the heap upper-bounds to 2gb; it would garbage collect often; now it doesn't as much; so here's a dump from when it's grown upwards of 1.2gb: https://www.dropbox.com/s/t75gegyyrejtl6r/heapdump-1375287225712.hprof.zip

It should be noted we're running nexus with the following 'extra' flags:

-XX:MaxPermSize=512m -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9010 -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -XX:+CMSPermGenSweepingEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseConcMarkSweepGC -Xms2048m -Xmx2048m

flyinprogrammer commented 11 years ago

https://github.com/sonatype/nexus-ruby-support/blob/master/nexus-ruby-plugin/src/main/java/org/sonatype/nexus/plugins/ruby/fs/RubyFSLocalRepositoryStorage.java#L309-L330

If the copy fails; out will never close; will it not?

mkristian commented 11 years ago

I will fix the copy thing.

and I also finally found some time for the heapdump. was that a heapdump just after a OutOfMemoryError ? I do see that the RubyProxy part uses more than a third of the heap but that is genuine. I have already an idea how to improve that by using some diskcache instead of keeping a hush hash-map in memory.

would it be possible for you to provide a heapdump after an OutOfMemory occured (assuming a no answer above ;) ?

flyinprogrammer commented 11 years ago

https://www.dropbox.com/s/c8fafbp3453u0zc/java_pid5114.hprof.zip

Fresh from this morning :smiley_cat:

mkristian commented 11 years ago

that last dump shows a lot of rubyobjects around the mergeSpecs methods. it kind of looks like that this is code before https://github.com/sonatype/nexus-ruby-support/commit/8168e5a21deb6e336bb800ecfd000e2e63ce45a0 - can that be the case ?

anyways I put in a crude synchronization to ensure only one thread can trigger the actual merging.

flyinprogrammer commented 11 years ago

crap you might be right about not running the code with that close in it :| -- i suck :(

flyinprogrammer commented 11 years ago 
| org.jruby.embed.InvokeFailedException: java.io.EOFException: Unexpected end of ZLIB input stream
jvm 1    |      at org.jruby.embed.internal.EmbedRubyObjectAdapterImpl.call(EmbedRubyObjectAdapterImpl.java:319) ~[na:na]
jvm 1    |      at org.jruby.embed.internal.EmbedRubyObjectAdapterImpl.callMethod(EmbedRubyObjectAdapterImpl.java:169) ~[na:na]
jvm 1    |      at org.jruby.embed.ScriptingContainer.callMethod(ScriptingContainer.java:1431) ~[na:na]
jvm 1    |      at org.sonatype.nexus.ruby.DefaultRubygemsGateway.mergeSpecs(DefaultRubygemsGateway.java:151) ~[na:na]
jvm 1    |      at org.sonatype.nexus.plugins.ruby.group.GroupRubygemsFacade.mergeSpecsIndex(GroupRubygemsFacade.java:36) ~[na:na]
jvm 1    |      at org.sonatype.nexus.plugins.ruby.fs.RubyFSLocalRepositoryStorage.storeSpecsIndices(RubyFSLocalRepositoryStorage.java:361) ~[na:na]
jvm 1    |      at org.sonatype.nexus.plugins.ruby.group.DefaultRubyGroupRepository.retrieveItem(DefaultRubyGroupRepository.java:131) ~[na:na]
jvm 1    |      at org.sonatype.nexus.proxy.router.DefaultRepositoryRouter.retrieveItem(DefaultRepositoryRouter.java:176) ~[nexus-core-2.5.1-01.jar:2.5.1-01]
jvm 1    |      at org.sonatype.nexus.rest.AbstractResourceStoreContentPlexusResource.get(AbstractResourceStoreContentPlexusResource.java:174) ~[nexus-restlet1x-plugin-2.5.1-01.jar:2.5.1-01]
jvm 1    |      at org.sonatype.plexus.rest.resource.RestletResource.represent(RestletResource.java:269) [plexus-restlet-bridge-1.25.jar:1.25]
jvm 1    |      at org.sonatype.nexus.rest.NexusRestletResource.represent(NexusRestletResource.java:43) [nexus-restlet1x-plugin-2.5.1-01.jar:2.5.1-01]
jvm 1    |      at org.restlet.resource.Resource.getRepresentation(Resource.java:302) [org.restlet-1.1.6-SONATYPE-5348-V8.jar:na]
jvm 1    |      at org.restlet.resource.Resource.handleGet(Resource.java:464) [org.restlet-1.1.6-SONATYPE-5348-V8.jar:na]
jvm 1    |      at org.restlet.Finder.handle(Finder.java:353) [org.restlet-1.1.6-SONATYPE-5348-V8.jar:na]
jvm 1    |      at org.restlet.Filter.doHandle(Filter.java:150) [org.restlet-1.1.6-SONATYPE-5348-V8.jar:na]
jvm 1    |      at org.restlet.Filter.handle(Filter.java:195) [org.restlet-1.1.6-SONATYPE-5348-V8.jar:na]
jvm 1    |      at org.restlet.Filter.doHandle(Filter.java:150) [org.restlet-1.1.6-SONATYPE-5348-V8.jar:na]
jvm 1    |      at org.restlet.Filter.handle(Filter.java:195) [org.restlet-1.1.6-SONATYPE-5348-V8.jar:na]
jvm 1    |      at org.restlet.Router.handle(Router.java:504) [org.restlet-1.1.6-SONATYPE-5348-V8.jar:na]
jvm 1    |      at org.restlet.Filter.doHandle(Filter.java:150) [org.restlet-1.1.6-SONATYPE-5348-V8.jar:na]
jvm 1    |      at org.restlet.Filter.handle(Filter.java:195) [org.restlet-1.1.6-SONATYPE-5348-V8.jar:na]
jvm 1    |      at org.restlet.Filter.doHandle(Filter.java:150) [org.restlet-1.1.6-SONATYPE-5348-V8.jar:na]
jvm 1    |      at org.sonatype.plexus.rest.RetargetableRestlet.doHandle(RetargetableRestlet.java:39) [plexus-restlet-bridge-1.25.jar:1.25]
jvm 1    |      at org.restlet.Filter.handle(Filter.java:195) [org.restlet-1.1.6-SONATYPE-5348-V8.jar:na]
jvm 1    |      at org.restlet.Filter.doHandle(Filter.java:150) [org.restlet-1.1.6-SONATYPE-5348-V8.jar:na]
jvm 1    |      at org.sonatype.plexus.rest.RetargetableRestlet.doHandle(RetargetableRestlet.java:39) [plexus-restlet-bridge-1.25.jar:1.25]
jvm 1    |      at org.sonatype.nexus.plugins.ui.RedirectingNexusApplicationCustomizer$Redirector.doHandle(RedirectingNexusApplicationCustomizer.java:58) [nexus-ui-extjs3-plugin-2.5.1-01.jar:2.5.1-01]
jvm 1    |      at org.restlet.Filter.handle(Filter.java:195) [org.restlet-1.1.6-SONATYPE-5348-V8.jar:na]
jvm 1    |      at org.restlet.Filter.doHandle(Filter.java:150) [org.restlet-1.1.6-SONATYPE-5348-V8.jar:na]
jvm 1    |      at org.restlet.Filter.handle(Filter.java:195) [org.restlet-1.1.6-SONATYPE-5348-V8.jar:na]
jvm 1    |      at org.restlet.Filter.doHandle(Filter.java:150) [org.restlet-1.1.6-SONATYPE-5348-V8.jar:na]
jvm 1    |      at org.restlet.Filter.handle(Filter.java:195) [org.restlet-1.1.6-SONATYPE-5348-V8.jar:na]
jvm 1    |      at org.restlet.Filter.doHandle(Filter.java:150) [org.restlet-1.1.6-SONATYPE-5348-V8.jar:na]
jvm 1    |      at com.noelios.restlet.StatusFilter.doHandle(StatusFilter.java:130) [com.noelios.restlet-1.1.6-SONATYPE-5348-V8.jar:na]
jvm 1    |      at org.restlet.Filter.handle(Filter.java:195) [org.restlet-1.1.6-SONATYPE-5348-V8.jar:na]
jvm 1    |      at org.restlet.Filter.doHandle(Filter.java:150) [org.restlet-1.1.6-SONATYPE-5348-V8.jar:na]

| org.jruby.embed.InvokeFailedException: java.io.EOFException: Unexpected end of ZLIB input stream
jvm 1    |      at org.jruby.embed.internal.EmbedRubyObjectAdapterImpl.call(EmbedRubyObjectAdapterImpl.java:319) ~[na:na]
jvm 1    |      at org.jruby.embed.internal.EmbedRubyObjectAdapterImpl.callMethod(EmbedRubyObjectAdapterImpl.java:169) ~[na:na]
jvm 1    |      at org.jruby.embed.ScriptingContainer.callMethod(ScriptingContainer.java:1431) ~[na:na]
jvm 1    |      at org.sonatype.nexus.ruby.DefaultRubygemsGateway.mergeSpecs(DefaultRubygemsGateway.java:151) ~[na:na]
jvm 1    |      at org.sonatype.nexus.plugins.ruby.group.GroupRubygemsFacade.mergeSpecsIndex(GroupRubygemsFacade.java:36) ~[na:na]
jvm 1    |      at org.sonatype.nexus.plugins.ruby.fs.RubyFSLocalRepositoryStorage.storeSpecsIndices(RubyFSLocalRepositoryStorage.java:361) ~[na:na]
jvm 1    |      at org.sonatype.nexus.plugins.ruby.group.DefaultRubyGroupRepository.retrieveItem(DefaultRubyGroupRepository.java:131) ~[na:na]
jvm 1    |      at org.sonatype.nexus.proxy.router.DefaultRepositoryRouter.retrieveItem(DefaultRepositoryRouter.java:176) ~[nexus-core-2.5.1-01.jar:2.5.1-01]
jvm 1    |      at org.sonatype.nexus.rest.AbstractResourceStoreContentPlexusResource.get(AbstractResourceStoreContentPlexusResource.java:174) ~[nexus-restlet1x-plugin-2.5.1-01.jar:2.5.1-01]
jvm 1    |      at org.sonatype.plexus.rest.resource.RestletResource.represent(RestletResource.java:269) [plexus-restlet-bridge-1.25.jar:1.25]
jvm 1    |      at org.sonatype.nexus.rest.NexusRestletResource.represent(NexusRestletResource.java:43) [nexus-restlet1x-plugin-2.5.1-01.jar:2.5.1-01]
jvm 1    |      at org.restlet.resource.Resource.getRepresentation(Resource.java:302) [org.restlet-1.1.6-SONATYPE-5348-V8.jar:na]
jvm 1    |      at org.restlet.resource.Resource.handleGet(Resource.java:464) [org.restlet-1.1.6-SONATYPE-5348-V8.jar:na]
jvm 1    |      at org.restlet.Finder.handle(Finder.java:353) [org.restlet-1.1.6-SONATYPE-5348-V8.jar:na]
jvm 1    |      at org.restlet.Filter.doHandle(Filter.java:150) [org.restlet-1.1.6-SONATYPE-5348-V8.jar:na]
jvm 1    |      at org.restlet.Filter.handle(Filter.java:195) [org.restlet-1.1.6-SONATYPE-5348-V8.jar:na]
jvm 1    |      at org.restlet.Filter.doHandle(Filter.java:150) [org.restlet-1.1.6-SONATYPE-5348-V8.jar:na]
jvm 1    |      at org.restlet.Filter.handle(Filter.java:195) [org.restlet-1.1.6-SONATYPE-5348-V8.jar:na]
jvm 1    |      at org.restlet.Router.handle(Router.java:504) [org.restlet-1.1.6-SONATYPE-5348-V8.jar:na]
jvm 1    |      at org.restlet.Filter.doHandle(Filter.java:150) [org.restlet-1.1.6-SONATYPE-5348-V8.jar:na]
jvm 1    |      at org.restlet.Filter.handle(Filter.java:195) [org.restlet-1.1.6-SONATYPE-5348-V8.jar:na]
jvm 1    |      at org.restlet.Filter.doHandle(Filter.java:150) [org.restlet-1.1.6-SONATYPE-5348-V8.jar:na]
jvm 1    |      at org.sonatype.plexus.rest.RetargetableRestlet.doHandle(RetargetableRestlet.java:39) [plexus-restlet-bridge-1.25.jar:1.25]
jvm 1    |      at org.restlet.Filter.handle(Filter.java:195) [org.restlet-1.1.6-SONATYPE-5348-V8.jar:na]
jvm 1    |      at org.restlet.Filter.doHandle(Filter.java:150) [org.restlet-1.1.6-SONATYPE-5348-V8.jar:na]
jvm 1    |      at org.sonatype.plexus.rest.RetargetableRestlet.doHandle(RetargetableRestlet.java:39) [plexus-restlet-bridge-1.25.jar:1.25]
jvm 1    |      at org.sonatype.nexus.plugins.ui.RedirectingNexusApplicationCustomizer$Redirector.doHandle(RedirectingNexusApplicationCustomizer.java:58) [nexus-ui-extjs3-plugin-2.5.1-01.jar:2.5.1-01]
jvm 1    |      at org.restlet.Filter.handle(Filter.java:195) [org.restlet-1.1.6-SONATYPE-5348-V8.jar:na]
jvm 1    |      at org.restlet.Filter.doHandle(Filter.java:150) [org.restlet-1.1.6-SONATYPE-5348-V8.jar:na]
jvm 1    |      at org.restlet.Filter.handle(Filter.java:195) [org.restlet-1.1.6-SONATYPE-5348-V8.jar:na]
jvm 1    |      at org.restlet.Filter.doHandle(Filter.java:150) [org.restlet-1.1.6-SONATYPE-5348-V8.jar:na]
jvm 1    |      at org.restlet.Filter.handle(Filter.java:195) [org.restlet-1.1.6-SONATYPE-5348-V8.jar:na]
jvm 1    |      at org.restlet.Filter.doHandle(Filter.java:150) [org.restlet-1.1.6-SONATYPE-5348-V8.jar:na]
jvm 1    |      at com.noelios.restlet.StatusFilter.doHandle(StatusFilter.java:130) [com.noelios.restlet-1.1.6-SONATYPE-5348-V8.jar:na]
jvm 1    |      at org.restlet.Filter.handle(Filter.java:195) [org.restlet-1.1.6-SONATYPE-5348-V8.jar:na]
jvm 1    |      at org.restlet.Filter.doHandle(Filter.java:150) [org.restlet-1.1.6-SONATYPE-5348-V8.jar:na]

This started popping up with the latest code.

flyinprogrammer commented 11 years ago

getting them instantly without the sync commit running the plugin built at and including commit 9d814a7c28f0ac08a796d2bb17e92b2b71ad640d:

jvm 1    | 2013-08-08 11:16:54 ERROR [qtp948252856-49] anonymous org.sonatype.nexus.rest.ContentPlexusResource - Got exception during processing request "GET http://nexus.pilot.cs.ha:8081/nexus/content/groups/gems/specs.4.8.gz":
jvm 1    | org.jruby.embed.InvokeFailedException: java.io.EOFException: Unexpected end of ZLIB input stream
jvm 1    |      at org.jruby.embed.internal.EmbedRubyObjectAdapterImpl.call(EmbedRubyObjectAdapterImpl.java:319) ~[na:na]
jvm 1    |      at org.jruby.embed.internal.EmbedRubyObjectAdapterImpl.callMethod(EmbedRubyObjectAdapterImpl.java:169) ~[na:na]
jvm 1    |      at org.jruby.embed.ScriptingContainer.callMethod(ScriptingContainer.java:1431) ~[na:na]
jvm 1    |      at org.sonatype.nexus.ruby.DefaultRubygemsGateway.mergeSpecs(DefaultRubygemsGateway.java:151) ~[na:na]
jvm 1    |      at org.sonatype.nexus.plugins.ruby.group.GroupRubygemsFacade.mergeSpecsIndex(GroupRubygemsFacade.java:36) ~[na:na]
jvm 1    |      at org.sonatype.nexus.plugins.ruby.fs.RubyFSLocalRepositoryStorage.storeSpecsIndices(RubyFSLocalRepositoryStorage.java:362) ~[na:na]
jvm 1    |      at org.sonatype.nexus.plugins.ruby.group.DefaultRubyGroupRepository.retrieveItem(DefaultRubyGroupRepository.java:129) ~[na:na]
jvm 1    |      at org.sonatype.nexus.proxy.router.DefaultRepositoryRouter.retrieveItem(DefaultRepositoryRouter.java:176) ~[nexus-core-2.5.1-01.jar:2.5.1-01]
jvm 1    |      at org.sonatype.nexus.rest.AbstractResourceStoreContentPlexusResource.get(AbstractResourceStoreContentPlexusResource.java:174) ~[nexus-restlet1x-plugin-2.5.1-01.jar:2.5.1-01]
jvm 1    |      at org.sonatype.plexus.rest.resource.RestletResource.represent(RestletResource.java:269) [plexus-restlet-bridge-1.25.jar:1.25]
jvm 1    |      at org.sonatype.nexus.rest.NexusRestletResource.represent(NexusRestletResource.java:43) [nexus-restlet1x-plugin-2.5.1-01.jar:2.5.1-01]
jvm 1    |      at org.restlet.resource.Resource.getRepresentation(Resource.java:302) [org.restlet-1.1.6-SONATYPE-5348-V8.jar:na]
jvm 1    |      at org.restlet.resource.Resource.handleGet(Resource.java:464) [org.restlet-1.1.6-SONATYPE-5348-V8.jar:na]
mkristian commented 11 years ago

strange since I am running the code now for some hours with some load as well. anyways I am about to look into those stacktraces.

flyinprogrammer commented 11 years ago

really weird too -- i reverted back to the original code before all of this; it was still broken; deleted the specs.4.8.gz file in the group repo storage, and things are working again.

I'm back to running the code just before the sync commit again; seems fine atm.

flyinprogrammer commented 11 years ago

seems like this might be a weird CentOS6 / jvm file locking issue ?

mkristian commented 11 years ago

well, I am running fedora with OpenJDK 64-Bit Server VM (build 23.7-b01, mixed mode)

which is not so far away from CentOS, but who knows. but the last hours I did see some increase in HEAP and thread count, so I have the feeling something is still leaking.

with those last error of yours, did you check that you use the right -Dnexus.version= flag to build your plugin - which I am checking myself right now as well.

flyinprogrammer commented 11 years ago

did a clean pull from github and ran:

mvn clean install -Dmaven.test.skip -Dnexus.version=2.5.1-01

we're running 2.5.1-01 so i think i did that right?

mkristian commented 11 years ago

I did use the 2.5.1-01 plugin with 2.5.0-04 server. so I try the 2.5.1-01 now.

flyinprogrammer commented 11 years ago

ok -- and i'm running: OpenJDK 64-Bit Server VM (20.0-b12, mixed mode) on centos 6.3(final)

mkristian commented 11 years ago

well, at least the zlib error I did find and fix - please delete the *specs.4.8.gz in the sonatype-work/nexus/storage/gems (gems is the name group repo)

still unsure if I see memory leak - still observing on my side

mkristian commented 10 years ago

@flyinprogramer thanx for all the help on this. since the underlying code changed quite a bit I am closing the issue.

flyinprogrammer commented 10 years ago

sounds good to me!