search

rundeck-plugins / rundeck-ec2-nodes-plugin

Get resource node data from Amazon EC2
86 stars 45 forks source link

nodelist is stuck and not updating #34

Closed rooty0 closed 8 years ago

rooty0 commented 8 years ago

I'm facing with ec2 plugin bug. Sometimes my ec2 nodelist stops to update.

root@ml-batch01:/rundeck/var/resourceModelSourceCache/Machine-Learning# ls -la
total 204
drwxrwxr-x 2 rundeck rundeck   4096 Nov 18 14:00 .
drwxrwxr-x 3 rundeck rundeck   4096 Oct 26 17:48 ..
-rw-rw-r-- 1 rundeck rundeck 197353 Nov 17 14:15 2.aws-ec2.xml
root@ml-batch01:/rundeck/var/resourceModelSourceCache/Machine-Learning# date
Wed Nov 18 14:07:04 GMT+5 2015

I was trying to pull nodelist by visiting the node page or running any job - It doesn't resolve anything. Sometimes it's just sticking like that. Rundeck restart helps tho.

Log

WARN  LoggingResourceModelSourceCache: [ResourceModelSource: 2.aws-ec2 (AWS EC2 Resources), project: Machine-Learning] Returning cached model data
ERROR ExceptionCatchingResourceModelSource: [ResourceModelSource: 2.aws-ec2 (AWS EC2 Resources), project: Machine-Learning]
java.lang.NoClassDefFoundError: com/amazonaws/services/ec2/AmazonEC2AsyncClient
        at com.dtolabs.rundeck.plugin.resources.ec2.InstanceToNodeMapper.performQueryAsync(InstanceToNodeMapper.java:105)
        at com.dtolabs.rundeck.plugin.resources.ec2.EC2ResourceModelSource.getNodes(EC2ResourceModelSource.java:208)
        at com.dtolabs.rundeck.core.resources.ExceptionCatchingResourceModelSource.getNodes(ExceptionCatchingResourceModelSource.java:45)
        at com.dtolabs.rundeck.core.common.ProjectNodeSupport.getNodeSet(ProjectNodeSupport.java:88)
        at com.dtolabs.rundeck.core.common.FrameworkProject.getNodeSet(FrameworkProject.java:380)
        at com.dtolabs.rundeck.core.common.IRundeckProject$getNodeSet$2.call(Unknown Source)
        at rundeck.services.FrameworkService$_filterNodeSet_closure4.doCall(FrameworkService.groovy:283)
        at rundeck.services.FrameworkService$_filterNodeSet_closure4.doCall(FrameworkService.groovy)
        at sun.reflect.GeneratedMethodAccessor851.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:324)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1207)
        at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1110)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1016)
        at groovy.lang.Closure.call(Closure.java:423)
        at groovy.lang.Closure.call(Closure.java:417)
        at com.codahale.metrics.Timer.time(Timer.java:99)
        at com.codahale.metrics.Timer$time$0.call(Unknown Source)
        at MetricswebGrailsPlugin$_addDynamicMetricMethods_closure26.doCall(MetricswebGrailsPlugin.groovy:174)
        at sun.reflect.GeneratedMethodAccessor578.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:324)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1207)
        at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1110)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1016)
        at groovy.lang.Closure.call(Closure.java:423)
        at org.codehaus.groovy.runtime.metaclass.ClosureStaticMetaMethod.invoke(ClosureStaticMetaMethod.java:59)
        at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoMetaMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:248)
        at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.call(PogoMetaMethodSite.java:68)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:124)
        at rundeck.services.FrameworkService.filterNodeSet(FrameworkService.groovy:282)
        at rundeck.services.FrameworkService$filterNodeSet$24.call(Unknown Source)
        at rundeck.services.ExecutionService.createContext(ExecutionService.groovy:1061)
        at rundeck.services.ExecutionService$createContext$26.callCurrent(Unknown Source)
        at rundeck.services.ExecutionService.executeAsyncBegin(ExecutionService.groovy:794)
        at rundeck.services.ExecutionService.executeAsyncBegin(ExecutionService.groovy)
        at rundeck.services.ExecutionService$$FastClassBySpringCGLIB$$ffb73873.invoke(<generated>)
        at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:708)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
        at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:98)
        at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:262)
        at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:95)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:644)
        at rundeck.services.ExecutionService$$EnhancerBySpringCGLIB$$e0364ca7.executeAsyncBegin(<generated>)
        at rundeck.services.ExecutionService$executeAsyncBegin$15.call(Unknown Source)
        at rundeck.quartzjobs.ExecutionJob.executeCommand(ExecutionJob.groovy:290)
        at rundeck.quartzjobs.ExecutionJob$executeCommand$6.callCurrent(Unknown Source)
        at rundeck.quartzjobs.ExecutionJob.execute_internal(ExecutionJob.groovy:125)
        at sun.reflect.GeneratedMethodAccessor1281.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:324)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1207)
        at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1110)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1016)
        at groovy.lang.Closure.call(Closure.java:423)
        at groovy.lang.Closure.call(Closure.java:417)
        at com.codahale.metrics.Timer.time(Timer.java:99)
        at com.codahale.metrics.Timer$time$0.call(Unknown Source)
        at rundeck.quartzjobs.ExecutionJob.execute(ExecutionJob.groovy:73)
        at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)

discussion: https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/rundeck-discuss/7L5zlPRcaw4/FYVziqLIAgAJ

gschueler commented 8 years ago

what rundeck version are you using? what are your memory settings?

it sounds similar to this issue, fixed in 2.5: https://github.com/rundeck/rundeck/issues/808

rooty0 commented 8 years ago

I'm using Rundeck 2.6.1, latest ec2 plugin rundeck-ec2-nodes-plugin-1.5.1.jar

root@ml-batch01:~# free -m
             total       used       free     shared    buffers     cached
Mem:         15042      14374        667          3        283      12307
-/+ buffers/cache:       1784      13258
Swap:            0          0          0

-Xmx1024m -Xms256m

root@ml-batch01:~# ps auxww|grep java
root     23352  0.0  0.0  10460   936 pts/1    S+   15:06   0:00 grep --color=auto java
rundeck  32218  1.1  6.7 6584092 1036740 ?     Ssl  Nov17  16:25 /usr/bin/java -Drdeck.config=/etc/rundeck -Drdeck.base=/var/lib/rundeck -Drundeck.server.configDir=/etc/rundeck -Dserver.datastore.path=/var/lib/rundeck/data -Drundeck.server.serverDir=/var/lib/rundeck -Drdeck.projects=/rundeck/projects -Drdeck.runlogs=/rundeck/logs -Drundeck.config.location=/etc/rundeck/rundeck-config.groovy -Djava.security.auth.login.config=/etc/rundeck/jaas-auth.conf -Dloginmodule.name=authentication -Xmx1024m -Xms256m -server -cp /var/lib/rundeck/bootstrap/rundeck-jetty-server-2.6.1.jar:/var/lib/rundeck/bootstrap/jna-3.2.2.jar:/var/lib/rundeck/bootstrap/log4j-1.2.16.jar:/var/lib/rundeck/bootstrap/libpam4j-1.5.jar:/var/lib/rundeck/bootstrap/not-yet-commons-ssl-0.3.11.jar:/var/lib/rundeck/bootstrap/jetty-all-7.6.0.v20120127.jar:/var/lib/rundeck/bootstrap/servlet-api-2.5.jar: com.dtolabs.rundeck.RunServer /var/lib/rundeck

Image

rooty0 commented 8 years ago

Trying to understand why I'm getting 

ERROR ExceptionCatchingResourceModelSource: [ResourceModelSource: 2.aws-ec2 (AWS EC2 Resources), project: Machine-Learning]
java.lang.NoClassDefFoundError: com/amazonaws/services/ec2/AmazonEC2AsyncClient
        at com.dtolabs.rundeck.plugin.resources.ec2.InstanceToNodeMapper.performQueryAsync(InstanceToNodeMapper.java:105)

even the file is there and nobody touched it /rundeck/var/resourceModelSourceCache/Machine-Learning/2.aws-ec2.xml

rooty0 commented 8 years ago

@gschueler, ok, it's becomes broken right after I'm running a job, doesn't matter if it's cli or gui. Checking now if it's some specific job or all of them

rooty0 commented 8 years ago

that's interesting, seems like any job breaking this

I'm going to reproduce this at docker clean environment

gschueler commented 8 years ago

this message java.lang.NoClassDefFoundError: com/amazonaws/services/ec2/AmazonEC2AsyncClient indicates it's something to do with the jar plugin loading mechanism. the bug i mentioned before was being caused because the plugin cache (containing the expanded jar libs) was getting wiped out inadvertently, which i think might be causing this too, i'm just not sure why

rooty0 commented 8 years ago

hmm so far docker's clean environment with same AWS credentials is stable, I can't see any issues :/ that interesting

rooty0 commented 8 years ago

that's so funny, I can't reproduce this bug within docker container... also funny thing I experiencing this bug on my dev and prod environment, 2 different machines, they almost identical tho.

I noticed when I running command without running a job aka adhoc (Command tab), it's not breaks cache It happening only when I'm running any job

rooty0 commented 8 years ago

@gschueler it's really hard to debug this... I don't have this issue anymore on dev and prod environment. It just disappeared. I basically did nothing to fix it, feeling like it will happen again soon when important jobs will be running :( Any advice how to debug this?

gschueler commented 8 years ago

If it is caused by an issue with the plugin cache, you could try this: make sure no jobs are running, then remove write permissions from the files in the "pluginJar" cache for the ec2 plugin. Then try to trigger the bug. If it is caused by the cache issue, I think Rundeck is deleting those jars and replacing them out from under the jobs. If you make them non-writable maybe it will cause an error message

rooty0 commented 8 years ago

thanks, I'll try. So far I can't reproduce this bug since yesterday, any environment

gschueler commented 8 years ago

do you use the cli tools at all?

rooty0 commented 8 years ago

sometimes, all my jobs are scheduled to run repeatedly -- replacement for crontab. But some jobs actually executing commands like "/usr/bin/run -p "Machine-Learning" --job "run node discovery" " to refresh ec2 nodelist (related: https://groups.google.com/forum/#!searchin/rundeck-discuss/ec2$20refresh/rundeck-discuss/PlZRVDu2tSg/rdVQSWRikLEJ)

rooty0 commented 8 years ago

p.s. when I was trying to reproduce this bug, I was using both: CLI and GUI

rooty0 commented 8 years ago

it's already 14 days gone and I can't reproduce this issue, going to close this, will reopen this ticket if I'll get this issue again

rooty0 commented 8 years ago

hey @gschueler , I just upgraded rundeck from 2.6.1 to 2.6.2 and I'm getting this issue again... as far as I remember I got this issue at first time with the same consequences (when I upgraded to 2.6.1) so I'm wondering can it be related

rooty0 commented 8 years ago

I'm trying to restart rundeck but it doesn't help. After the restart everything is fine but right after I run a job I'm getting: java.lang.NoClassDefFoundError: com/amazonaws/services/ec2/AmazonEC2AsyncClient

rooty0 commented 8 years ago

My last issue was fixed by it's own, wondering if it's some library cache? BTW! Just recall your comment! I'm guessing issue is still there :(

rooty0 commented 8 years ago

@gschueler so I removed write permissions and ran I job, then gave back write permission and ran job again - bug disappeared

rooty0 commented 8 years ago

I'm getting now this:

WARN  EC2ResourceModelSource: Error performing query: java.lang.NoClassDefFoundError: com/amazonaws/services/ec2/model/transform/InstanceNetworkInterfaceAssociationStaxUnmarshaller
java.util.concurrent.ExecutionException: java.lang.NoClassDefFoundError: com/amazonaws/services/ec2/model/transform/InstanceNetworkInterfaceAssociationStaxUnmarshaller
    at java.util.concurrent.FutureTask.report(FutureTask.java:122)
    at java.util.concurrent.FutureTask.get(FutureTask.java:192)
...
Caused by: java.lang.NoClassDefFoundError: com/amazonaws/services/ec2/model/transform/InstanceNetworkInterfaceAssociationStaxUnmarshaller
    at com.amazonaws.services.ec2.model.transform.InstanceNetworkInterfaceStaxUnmarshaller.unmarshall(InstanceNetworkInterfaceStaxUnmarshaller.java:91)
    at com.amazonaws.services.ec2.model.transform.InstanceStaxUnmarshaller.unmarshall(InstanceStaxUnmarshaller.java:179)
rooty0 commented 8 years ago

really hard to reproduce/catch the bug. This happens in random period of time. yesterday we noticed all jobs on production server were failing because of this issue :(

gschueler commented 8 years ago

@rooty0 :( sorry, really odd. i still think it is something to do with plugin jar cache.

i'd like to figure out why. if you remove write perms for the whole libext/cache/rundeck-ec2-nodes-plugin* dir and all of the files inside it, let it run for a while, see if you can trigger any issue

rooty0 commented 8 years ago

@gschueler write permissions were removed

chown -R root:root rundeck-ec2-nodes-plugin-1.5.1

and I was able to get this:

com.dtolabs.rundeck.core.plugins.PluginException: Unable to expand plugin libs: /var/lib/rundeck/libext/cache/rundeck-ec2-nodes-plugin-1.5.1/lib/commons-beanutils-1.7.0.jar (Permission denied)
        at com.dtolabs.rundeck.core.plugins.JarPluginProviderLoader.getClassLoader(JarPluginProviderLoader.java:344)
        at com.dtolabs.rundeck.core.plugins.JarPluginProviderLoader.loadClass(JarPluginProviderLoader.java:324)
        at com.dtolabs.rundeck.core.plugins.JarPluginProviderLoader.listProviders(JarPluginProviderLoader.java:466)
        at com.dtolabs.rundeck.core.plugins.DirPluginScanner.listProviders(DirPluginScanner.java:258)
        at com.dtolabs.rundeck.core.plugins.DirPluginScanner.listProviders(DirPluginScanner.java:131)
        at com.dtolabs.rundeck.core.plugins.JarPluginScanner.listProviders(JarPluginScanner.java:38)
        at com.dtolabs.rundeck.core.plugins.FilePluginCache.listProviders(FilePluginCache.java:127)
        at com.dtolabs.rundeck.core.plugins.PluginManagerService.listProviders(PluginManagerService.java:119)
        at com.dtolabs.rundeck.core.plugins.BasePluggableProviderService.listProviders(BasePluggableProviderService.java:64)
        at com.dtolabs.rundeck.core.common.ProviderService$listProviders$0.call(Unknown Source)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:112)
        at com.dtolabs.rundeck.server.plugins.RundeckPluginRegistry.listPluginDescriptors(RundeckPluginRegistry.groovy:330)
        at com.dtolabs.rundeck.server.plugins.PluginRegistry$listPluginDescriptors$0.call(Unknown Source)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:120)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callSafe(AbstractCallSite.java:89)
        at rundeck.services.PluginService.listPlugins(PluginService.groovy:182)
        at rundeck.services.PluginService$listPlugins.call(Unknown Source)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:120)
        at rundeck.services.NotificationService.listNotificationPlugins(NotificationService.groovy:77)
        at rundeck.services.NotificationService$listNotificationPlugins.call(Unknown Source)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:112)
        at rundeck.controllers.ScheduledExecutionController$_closure7.doCall(ScheduledExecutionController.groovy:305)
        at rundeck.controllers.ScheduledExecutionController$_closure7.doCall(ScheduledExecutionController.groovy)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:324)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1207)
        at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1110)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1016)
        at groovy.lang.Closure.call(Closure.java:423)
        at groovy.lang.Closure.call(Closure.java:417)
        at rundeck.controllers.ScheduledExecutionController.show(ScheduledExecutionController.groovy)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at org.codehaus.groovy.grails.web.servlet.mvc.MixedGrailsControllerHelper.invoke(MixedGrailsControllerHelper.java:154)
        at org.codehaus.groovy.grails.web.servlet.mvc.AbstractGrailsControllerHelper.handleAction(AbstractGrailsControllerHelper.java:375)
        at org.codehaus.groovy.grails.web.servlet.mvc.AbstractGrailsControllerHelper.executeAction(AbstractGrailsControllerHelper.java:252)
        at org.codehaus.groovy.grails.web.servlet.mvc.AbstractGrailsControllerHelper.handleURI(AbstractGrailsControllerHelper.java:205)
        at org.codehaus.groovy.grails.web.servlet.mvc.AbstractGrailsControllerHelper.handleURI(AbstractGrailsControllerHelper.java:126)
        at org.codehaus.groovy.grails.web.servlet.mvc.SimpleGrailsController.handleRequest(SimpleGrailsController.java:72)
        at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:50)
        at org.codehaus.groovy.grails.web.servlet.GrailsDispatcherServlet.doDispatch(GrailsDispatcherServlet.java:347)
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:870)
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:961)
        at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:852)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:837)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:565)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1360)
        at grails.plugin.cache.web.filter.PageFragmentCachingFilter.doFilter(PageFragmentCachingFilter.java:198)
        at grails.plugin.cache.web.filter.AbstractFilter.doFilter(AbstractFilter.java:63)
        at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:344)
        at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:261)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1331)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:101)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1331)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:101)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1331)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:101)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1331)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:477)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:539)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:227)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1031)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:406)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:186)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:965)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
        at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:288)
        at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:115)
        at org.codehaus.groovy.grails.web.mapping.UrlMappingUtils.forwardRequestForUrlMappingInfo(UrlMappingUtils.java:178)
        at org.codehaus.groovy.grails.web.mapping.UrlMappingUtils.forwardRequestForUrlMappingInfo(UrlMappingUtils.java:144)
        at org.codehaus.groovy.grails.web.mapping.UrlMappingUtils.forwardRequestForUrlMappingInfo(UrlMappingUtils.java:135)
        at org.codehaus.groovy.grails.web.mapping.filter.UrlMappingsFilter.doFilterInternal(UrlMappingsFilter.java:216)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1331)
        at org.codehaus.groovy.grails.web.servlet.mvc.GrailsWebRequestFilter.doFilterInternal(GrailsWebRequestFilter.java:69)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1331)
        at org.codehaus.groovy.grails.web.filters.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:67)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1331)
        at com.codahale.metrics.servlet.AbstractInstrumentedFilter.doFilter(AbstractInstrumentedFilter.java:97)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1331)
        at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:344)
        at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:261)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1331)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:477)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:486)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:227)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1031)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:406)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:186)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:965)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111)
        at org.eclipse.jetty.server.Server.handle(Server.java:349)
        at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:449)
        at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:910)
        at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:634)
        at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:230)
        at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:76)
        at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:609)
        at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:45)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.FileNotFoundException: /var/lib/rundeck/libext/cache/rundeck-ec2-nodes-plugin-1.5.1/lib/commons-beanutils-1.7.0.jar (Permission denied)
        at java.io.FileOutputStream.open0(Native Method)
        at java.io.FileOutputStream.open(FileOutputStream.java:270)
        at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
        at java.io.FileOutputStream.<init>(FileOutputStream.java:162)
        at com.dtolabs.rundeck.core.utils.ZipUtil.extractZip(ZipUtil.java:179)
        at com.dtolabs.rundeck.core.utils.ZipUtil.extractZipFile(ZipUtil.java:79)
        at com.dtolabs.rundeck.core.plugins.JarPluginProviderLoader.extractJarContents(JarPluginProviderLoader.java:417)
        at com.dtolabs.rundeck.core.plugins.JarPluginProviderLoader.extractDependentLibs(JarPluginProviderLoader.java:388)
        at com.dtolabs.rundeck.core.plugins.JarPluginProviderLoader.getClassLoader(JarPluginProviderLoader.java:342)
        ... 124 more
com.dtolabs.rundeck.core.plugins.PluginException: Unable to expand plugin libs: /var/lib/rundeck/libext/cache/rundeck-ec2-nodes-plugin-1.5.1/lib/commons-beanutils-1.7.0.jar (Permission denied)
        at com.dtolabs.rundeck.core.plugins.JarPluginProviderLoader.getClassLoader(JarPluginProviderLoader.java:344)
        at com.dtolabs.rundeck.core.plugins.JarPluginProviderLoader.loadClass(JarPluginProviderLoader.java:324)
        at com.dtolabs.rundeck.core.plugins.JarPluginProviderLoader.listProviders(JarPluginProviderLoader.java:466)
        at com.dtolabs.rundeck.core.plugins.DirPluginScanner.listProviders(DirPluginScanner.java:258)
        at com.dtolabs.rundeck.core.plugins.DirPluginScanner.listProviders(DirPluginScanner.java:131)
        at com.dtolabs.rundeck.core.plugins.JarPluginScanner.listProviders(JarPluginScanner.java:38)
        at com.dtolabs.rundeck.core.plugins.FilePluginCache.listProviders(FilePluginCache.java:127)
        at com.dtolabs.rundeck.core.plugins.PluginManagerService.listProviders(PluginManagerService.java:119)
        at com.dtolabs.rundeck.core.plugins.BasePluggableProviderService.listProviders(BasePluggableProviderService.java:64)
        at com.dtolabs.rundeck.core.plugins.configuration.DescribableServiceUtil.listDescriptions(DescribableServiceUtil.java:49)
        at com.dtolabs.rundeck.core.plugins.configuration.DescribableServiceUtil.listDescriptions(DescribableServiceUtil.java:43)
        at com.dtolabs.rundeck.core.plugins.BasePluggableProviderService.listDescriptions(BasePluggableProviderService.java:119)
        at com.dtolabs.rundeck.core.plugins.configuration.DescribableService$listDescriptions.call(Unknown Source)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:112)
        at com.dtolabs.rundeck.server.plugins.RundeckPluginRegistry.listPluginDescriptors(RundeckPluginRegistry.groovy:347)
        at com.dtolabs.rundeck.server.plugins.PluginRegistry$listPluginDescriptors$0.call(Unknown Source)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:120)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callSafe(AbstractCallSite.java:89)
        at rundeck.services.PluginService.listPlugins(PluginService.groovy:182)
        at rundeck.services.PluginService$listPlugins.call(Unknown Source)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:120)
        at rundeck.services.NotificationService.listNotificationPlugins(NotificationService.groovy:77)
        at rundeck.services.NotificationService$listNotificationPlugins.call(Unknown Source)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:112)
        at rundeck.controllers.ScheduledExecutionController$_closure7.doCall(ScheduledExecutionController.groovy:305)
        at rundeck.controllers.ScheduledExecutionController$_closure7.doCall(ScheduledExecutionController.groovy)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:324)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1207)
        at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1110)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1016)
        at groovy.lang.Closure.call(Closure.java:423)
        at groovy.lang.Closure.call(Closure.java:417)
        at rundeck.controllers.ScheduledExecutionController.show(ScheduledExecutionController.groovy)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at org.codehaus.groovy.grails.web.servlet.mvc.MixedGrailsControllerHelper.invoke(MixedGrailsControllerHelper.java:154)
        at org.codehaus.groovy.grails.web.servlet.mvc.AbstractGrailsControllerHelper.handleAction(AbstractGrailsControllerHelper.java:375)
        at org.codehaus.groovy.grails.web.servlet.mvc.AbstractGrailsControllerHelper.executeAction(AbstractGrailsControllerHelper.java:252)
        at org.codehaus.groovy.grails.web.servlet.mvc.AbstractGrailsControllerHelper.handleURI(AbstractGrailsControllerHelper.java:205)
        at org.codehaus.groovy.grails.web.servlet.mvc.AbstractGrailsControllerHelper.handleURI(AbstractGrailsControllerHelper.java:126)
        at org.codehaus.groovy.grails.web.servlet.mvc.SimpleGrailsController.handleRequest(SimpleGrailsController.java:72)
        at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:50)
        at org.codehaus.groovy.grails.web.servlet.GrailsDispatcherServlet.doDispatch(GrailsDispatcherServlet.java:347)
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:870)
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:961)
        at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:852)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:837)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:565)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1360)
        at grails.plugin.cache.web.filter.PageFragmentCachingFilter.doFilter(PageFragmentCachingFilter.java:198)
        at grails.plugin.cache.web.filter.AbstractFilter.doFilter(AbstractFilter.java:63)
        at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:344)
        at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:261)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1331)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:101)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1331)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:101)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1331)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:101)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1331)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:477)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:539)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:227)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1031)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:406)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:186)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:965)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
        at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:288)
        at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:115)
        at org.codehaus.groovy.grails.web.mapping.UrlMappingUtils.forwardRequestForUrlMappingInfo(UrlMappingUtils.java:178)
        at org.codehaus.groovy.grails.web.mapping.UrlMappingUtils.forwardRequestForUrlMappingInfo(UrlMappingUtils.java:144)
        at org.codehaus.groovy.grails.web.mapping.UrlMappingUtils.forwardRequestForUrlMappingInfo(UrlMappingUtils.java:135)
        at org.codehaus.groovy.grails.web.mapping.filter.UrlMappingsFilter.doFilterInternal(UrlMappingsFilter.java:216)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1331)
        at org.codehaus.groovy.grails.web.servlet.mvc.GrailsWebRequestFilter.doFilterInternal(GrailsWebRequestFilter.java:69)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1331)
        at org.codehaus.groovy.grails.web.filters.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:67)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1331)
        at com.codahale.metrics.servlet.AbstractInstrumentedFilter.doFilter(AbstractInstrumentedFilter.java:97)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1331)
        at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:344)
        at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:261)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1331)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:477)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:486)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:227)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1031)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:406)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:186)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:965)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111)
        at org.eclipse.jetty.server.Server.handle(Server.java:349)
        at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:449)
        at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:910)
        at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:634)
        at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:230)
        at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:76)
        at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:609)
        at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:45)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.FileNotFoundException: /var/lib/rundeck/libext/cache/rundeck-ec2-nodes-plugin-1.5.1/lib/commons-beanutils-1.7.0.jar (Permission denied)
        at java.io.FileOutputStream.open0(Native Method)
        at java.io.FileOutputStream.open(FileOutputStream.java:270)
        at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
        at java.io.FileOutputStream.<init>(FileOutputStream.java:162)
        at com.dtolabs.rundeck.core.utils.ZipUtil.extractZip(ZipUtil.java:179)
        at com.dtolabs.rundeck.core.utils.ZipUtil.extractZipFile(ZipUtil.java:79)
        at com.dtolabs.rundeck.core.plugins.JarPluginProviderLoader.extractJarContents(JarPluginProviderLoader.java:417)
        at com.dtolabs.rundeck.core.plugins.JarPluginProviderLoader.extractDependentLibs(JarPluginProviderLoader.java:388)
        at com.dtolabs.rundeck.core.plugins.JarPluginProviderLoader.getClassLoader(JarPluginProviderLoader.java:342)
        ... 127 more
rooty0 commented 8 years ago

it's broken again, write permissions still removed:

WARN  EC2ResourceModelSource: Error performing query: java.lang.NoClassDefFoundError: com/amazonaws/services/ec2/model/transform/InstanceNetworkInterfaceAssociationStaxUnmarshaller
java.util.concurrent.ExecutionException: java.lang.NoClassDefFoundError: com/amazonaws/services/ec2/model/transform/InstanceNetworkInterfaceAssociationStaxUnmarshaller
        at java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.util.concurrent.FutureTask.get(FutureTask.java:192)
        at com.dtolabs.rundeck.plugin.resources.ec2.InstanceToNodeMapper$1.get(InstanceToNodeMapper.java:132)
        at com.dtolabs.rundeck.plugin.resources.ec2.InstanceToNodeMapper$1.get(InstanceToNodeMapper.java:117)
        at com.dtolabs.rundeck.plugin.resources.ec2.EC2ResourceModelSource.checkFuture(EC2ResourceModelSource.java:227)
        at com.dtolabs.rundeck.plugin.resources.ec2.EC2ResourceModelSource.getNodes(EC2ResourceModelSource.java:200)
        at com.dtolabs.rundeck.core.resources.ExceptionCatchingResourceModelSource.getNodes(ExceptionCatchingResourceModelSource.java:45)
        at com.dtolabs.rundeck.core.common.ProjectNodeSupport.getNodeSet(ProjectNodeSupport.java:88)
        at com.dtolabs.rundeck.core.common.FrameworkProject.getNodeSet(FrameworkProject.java:380)
        at com.dtolabs.rundeck.core.common.IRundeckProject$getNodeSet$1.call(Unknown Source)
        at rundeck.controllers.FrameworkController.nodesdata(FrameworkController.groovy:296)
        at rundeck.controllers.FrameworkController$nodesdata$1.callCurrent(Unknown Source)
        at rundeck.controllers.FrameworkController.nodesFragment(FrameworkController.groovy:526)
        at rundeck.controllers.FrameworkController.nodesFragment(FrameworkController.groovy)
        at sun.reflect.GeneratedMethodAccessor1021.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at org.codehaus.groovy.grails.web.servlet.mvc.MixedGrailsControllerHelper.invoke(MixedGrailsControllerHelper.java:154)
        at org.codehaus.groovy.grails.web.servlet.mvc.AbstractGrailsControllerHelper.handleAction(AbstractGrailsControllerHelper.java:375)
        at org.codehaus.groovy.grails.web.servlet.mvc.AbstractGrailsControllerHelper.executeAction(AbstractGrailsControllerHelper.java:252)
        at org.codehaus.groovy.grails.web.servlet.mvc.AbstractGrailsControllerHelper.handleURI(AbstractGrailsControllerHelper.java:205)
        at org.codehaus.groovy.grails.web.servlet.mvc.AbstractGrailsControllerHelper.handleURI(AbstractGrailsControllerHelper.java:126)
        at org.codehaus.groovy.grails.web.servlet.mvc.SimpleGrailsController.handleRequest(SimpleGrailsController.java:72)
        at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:50)
        at org.codehaus.groovy.grails.web.servlet.GrailsDispatcherServlet.doDispatch(GrailsDispatcherServlet.java:347)
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:870)
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:961)
        at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:852)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:837)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:565)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1360)
        at grails.plugin.cache.web.filter.PageFragmentCachingFilter.doFilter(PageFragmentCachingFilter.java:198)
        at grails.plugin.cache.web.filter.AbstractFilter.doFilter(AbstractFilter.java:63)
        at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:344)
        at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:261)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1331)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:101)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1331)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:101)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1331)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:101)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1331)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:477)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:539)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:227)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1031)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:406)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:186)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:965)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
        at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:288)
        at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:115)
        at org.codehaus.groovy.grails.web.mapping.UrlMappingUtils.forwardRequestForUrlMappingInfo(UrlMappingUtils.java:178)
        at org.codehaus.groovy.grails.web.mapping.UrlMappingUtils.forwardRequestForUrlMappingInfo(UrlMappingUtils.java:144)
        at org.codehaus.groovy.grails.web.mapping.UrlMappingUtils.forwardRequestForUrlMappingInfo(UrlMappingUtils.java:135)
        at org.codehaus.groovy.grails.web.mapping.filter.UrlMappingsFilter.doFilterInternal(UrlMappingsFilter.java:216)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1331)
        at org.codehaus.groovy.grails.web.servlet.mvc.GrailsWebRequestFilter.doFilterInternal(GrailsWebRequestFilter.java:69)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1331)
        at org.codehaus.groovy.grails.web.filters.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:67)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1331)
        at com.codahale.metrics.servlet.AbstractInstrumentedFilter.doFilter(AbstractInstrumentedFilter.java:97)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1331)
        at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:344)
        at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:261)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1331)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:477)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:486)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:227)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1031)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:406)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:186)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:965)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111)
        at org.eclipse.jetty.server.Server.handle(Server.java:349)
        at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:449)
        at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:910)
        at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:634)
        at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:230)
        at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:76)
        at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:609)
        at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:45)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NoClassDefFoundError: com/amazonaws/services/ec2/model/transform/InstanceNetworkInterfaceAssociationStaxUnmarshaller
        at com.amazonaws.services.ec2.model.transform.InstanceNetworkInterfaceStaxUnmarshaller.unmarshall(InstanceNetworkInterfaceStaxUnmarshaller.java:91)
        at com.amazonaws.services.ec2.model.transform.InstanceStaxUnmarshaller.unmarshall(InstanceStaxUnmarshaller.java:179)
        at com.amazonaws.services.ec2.model.transform.ReservationStaxUnmarshaller.unmarshall(ReservationStaxUnmarshaller.java:67)
        at com.amazonaws.services.ec2.model.transform.DescribeInstancesResultStaxUnmarshaller.unmarshall(DescribeInstancesResultStaxUnmarshaller.java:47)
        at com.amazonaws.services.ec2.model.transform.DescribeInstancesResultStaxUnmarshaller.unmarshall(DescribeInstancesResultStaxUnmarshaller.java:32)
        at com.amazonaws.http.StaxResponseHandler.handle(StaxResponseHandler.java:99)
        at com.amazonaws.http.StaxResponseHandler.handle(StaxResponseHandler.java:42)
        at com.amazonaws.http.AmazonHttpClient.handleResponse(AmazonHttpClient.java:929)
        at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:663)
        at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:441)
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:292)
        at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:9225)
        at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:4541)
        at com.amazonaws.services.ec2.AmazonEC2AsyncClient$173.call(AmazonEC2AsyncClient.java:8126)
        at com.amazonaws.services.ec2.AmazonEC2AsyncClient$173.call(AmazonEC2AsyncClient.java:8124)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        ... 1 more
Likeyn commented 8 years ago

Hey, just wanna let you guys know I ran into a similar issue and landed here while searching what was wrong, so it might help others. Mine was due to a time desynchronisation issue; if you spot the following line in the logs:

com.amazonaws.AmazonServiceException: AWS was not able to validate the provided access credentials (Service: AmazonEC2; Status Code: 401; Error Code: AuthFailure; Request ID: feed-da-deadbeef)

but your credentials are okay, chances are you have the same issue I had. Apparently Amazon rejects the authorisation request if the difference between your time and their time is too big; updating my local time fixed it for me.

Source: http://stackoverflow.com/a/33296525/1752536

Jinkxed commented 8 years ago

Believe it has to do with: https://github.com/rundeck/rundeck/issues/1725

Using Rundeck 2.6.4-1 With rundeck-ec2-nodes-plugin-1.5.2.jar Not using docker. Amazon Linux on EC2.

Node list isn't updating unless I click save on the configuration page of the plugin.

I am using a custom mapping of: 

description.default=EC2 node instance
editUrl.default=https://console.aws.amazon.com/ec2/home#s=Instances&selectInstance=${node.instanceId}
hostname.selector=privateIpAddress
sshport.default=22
sshport.selector=tags/ssh_config_Port
instanceId.selector=instanceId
nodename.selector=tags/Name,instanceId
osArch.selector=architecture
osFamily.default=unix
osFamily.selector=platform
osName.default=Linux
osName.selector=platform
privateDnsName.selector=privateDnsName
privateIpAddress.selector=privateIpAddress
state.selector=state.name
tag.pending.selector=state.name=pending
tag.running.selector=state.name=running
tag.shutting-down.selector=state.name=shutting-down
tag.stopped.selector=state.name=stopped
tag.stopping.selector=state.name=stopping
tag.terminated.selector=state.name=terminated
tags.default=ec2
tags.selector=tags/Environment|tags/Product|tags/Role
username.default=rd
username.selector=tags/Rundeck-User

Not sure if that makes any difference?

rooty0 commented 8 years ago

@Likeyn, thanks for heads up, never had this issue, just double checked with grep across all logs since 2015, haven't found this. Seems like this is not my case :(

At some point I just gave up and I'm not using this plugin anymore as it was a huge mess on production.

Jinkxed commented 8 years ago

@rooty0 What did you move to for your ec2 nodes?

rooty0 commented 8 years ago

@chad-upton, I rewrote my jobs so they don't need aws ec2 resources anymore. There is other simple solution which I didn't have time to work on yet: just to generate a list (every one minute for example) of all ec2 instances with awscli or any aws sdk and redirect output to some file which you can use as "Resource Model Source"

burdandrei commented 8 years ago

Hi everyone! I received the bug after upgrading to 2.6.4 Thanks to @chad-upton i don't need to restart it, save button work. but this is really strange. Nothing in logs @rooty0, really nice hack, but i don't want to do that

gschueler commented 8 years ago

hi all, please try recent 2.6.6 rundeck release, I think it may fix this issue

burdandrei commented 8 years ago

thanks @gschueler, updated today. will report if this will reoccur :+1:

rooty0 commented 8 years ago

I'll test in dev. Thanks

burdandrei commented 8 years ago

Looks like 2.6.6 solves the issue! :star:

gschueler commented 8 years ago

great!