search

rundeck-plugins / rundeck-s3-log-plugin

Store execution logs in Amazon S3 buckets
Apache License 2.0
32 stars 26 forks source link

S3 Plugin gets Java stacktrace on attempt to perform duties #19

Closed isuftin closed 7 years ago

isuftin commented 7 years ago

Plugin version: 1.0.3 Rundeck version: 2.9.3-1 Running on AWS in us-west-2 on CentOS 7

The config:

framework.properties:

framework.plugin.ExecutionFileStorage.org.rundeck.amazon-s3.bucket=my-logging-bucket
framework.plugin.ExecutionFileStorage.org.rundeck.amazon-s3.forceSigV4=true
framework.plugin.ExecutionFileStorage.org.rundeck.amazon-s3.path=rundeck/${job.project}/${job.id}/${job.execid}.log
framework.plugin.ExecutionFileStorage.org.rundeck.amazon-s3.region=us-west-2

rundeck-config.properties:

rundeck.execution.logs.fileStoragePlugin=org.rundeck.amazon-s3

The trace:

2017-09-12 14:31:33,716 [qtp81628611-19] INFO  grails.app.services.rundeck.services.ProjectManagerService - Loading project definition for PROD_OWI...
2017-09-12 14:31:33,729 [qtp81628611-19] INFO  grails.app.services.rundeck.services.ProjectManagerService - Loaded project PROD_OWI in 13ms
2017-09-12 14:31:33,732 [qtp81628611-19] INFO  grails.app.services.rundeck.services.ProjectManagerService - Loading project definition for DEV_OWI...
2017-09-12 14:31:33,743 [qtp81628611-19] INFO  grails.app.services.rundeck.services.ProjectManagerService - Loaded project DEV_OWI in 11ms
2017-09-12 14:31:38,786 [quartzScheduler_Worker-3] INFO  grails.app.services.rundeck.services.ExecutionUtilService - Execution successful: 465 in project Helpers
2017-09-12 14:31:38,858 [LogFileStorage1] ERROR rundeck.services.TaskRunner - An error occured while processing a task: java.io.IOException: No such file or directory
java.io.IOException: No such file or directory
        at java.io.UnixFileSystem.createFileExclusively(Native Method)
        at java.io.File.createTempFile(File.java:2024)
        at java.io.File.createTempFile(File.java:2070)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
        at org.codehaus.groovy.runtime.callsite.StaticMetaMethodSite$StaticMetaMethodSiteNoUnwrap.invoke(StaticMetaMethodSite.java:133)
        at org.codehaus.groovy.runtime.callsite.StaticMetaMethodSite.call(StaticMetaMethodSite.java:91)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:133)
        at rundeck.services.ProjectService.getExecutionXmlFileForExecution(ProjectService.groovy:147)
        at rundeck.services.ProjectService$getExecutionXmlFileForExecution.callCurrent(Unknown Source)
        at rundeck.services.ProjectService.produceStorageFileForExecution(ProjectService.groovy:136)
        at rundeck.services.logging.ExecutionFileProducer$produceStorageFileForExecution.call(Unknown Source)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
        at rundeck.services.logging.ExecutionFileProducer$produceStorageFileForExecution.call(Unknown Source)
        at rundeck.services.LogFileStorageService$_getExecutionFiles_closure21.doCall(LogFileStorageService.groovy:1123)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1218)
        at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1125)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1027)
        at groovy.lang.Closure.call(Closure.java:414)
        at groovy.lang.Closure.call(Closure.java:430)
        at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:2030)
        at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:2015)
        at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:2044)
        at org.codehaus.groovy.runtime.dgm$160.invoke(Unknown Source)
        at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:274)
        at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:56)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callSafe(AbstractCallSite.java:87)
        at rundeck.services.LogFileStorageService.getExecutionFiles(LogFileStorageService.groovy:1122)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1218)
        at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1125)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1027)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1163)
        at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1125)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1027)
        at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:69)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:174)
        at rundeck.services.LogFileStorageService$_runStorageRequest_closure3.doCall(LogFileStorageService.groovy:189)
        at rundeck.services.LogFileStorageService$_runStorageRequest_closure3$doCall.call(Unknown Source)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
        at rundeck.services.ExecutionService$_updateScheduledExecStatistics_closure65$doCall.call(Unknown Source)
        at org.codehaus.groovy.grails.orm.hibernate.HibernateGormStaticApi$6.doInHibernate(HibernateGormStaticApi.groovy:517)
        at org.codehaus.groovy.grails.orm.hibernate.GrailsHibernateTemplate.doExecute(GrailsHibernateTemplate.java:188)
        at org.codehaus.groovy.grails.orm.hibernate.GrailsHibernateTemplate.execute(GrailsHibernateTemplate.java:132)
        at org.codehaus.groovy.grails.orm.hibernate.GrailsHibernateTemplate$execute$0.call(Unknown Source)
        at org.codehaus.groovy.grails.orm.hibernate.HibernateGormStaticApi.withNewSession(HibernateGormStaticApi.groovy:515)
        at rundeck.LogFileStorageRequest.withNewSession(LogFileStorageRequest.groovy)
        at rundeck.LogFileStorageRequest$withNewSession$2.call(Unknown Source)
        at rundeck.services.LogFileStorageService.runStorageRequest(LogFileStorageService.groovy:186)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1218)
        at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1125)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1027)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1163)
        at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1125)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1027)
        at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:69)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:166)
        at rundeck.services.LogFileStorageService$_afterPropertiesSet_closure1.doCall(LogFileStorageService.groovy:125)
        at rundeck.services.LogFileStorageService$_afterPropertiesSet_closure1$doCall.call(Unknown Source)
        at rundeck.services.TaskRunner.run(TaskRunner.groovy:44)
        at org.springframework.core.task.SimpleAsyncTaskExecutor$ConcurrencyThrottlingRunnable.run(SimpleAsyncTaskExecutor.java:251)
        at java.lang.Thread.run(Thread.java:748)
isuftin commented 7 years ago

Also I should mention that the EC2 instance has an IAM role attached that allows writing to the S3 bucket, so credentials aren't configured at this level.

gschueler commented 7 years ago

the stacktrace traces to this line: at rundeck.services.ProjectService.getExecutionXmlFileForExecution(ProjectService.groovy:147)

Which is using Java's File.createTempFile which will fail if the Temp directory that is defined does not exist or is not writable.

Make sure your JVM properties have a -Djava.io.tmpdir value pointing at a valid temp dir. For Debian/RPM installs, you can define the RUNDECK_TEMPDIR in /etc/default/rundeckd or /etc/sysconfig/rundeckd respectively.

export RUNDECK_TEMPDIR=/var/tmp/rundeck

If you are using the Launcher, you will have to add -Djava.io.tmpdir=/tmp/dir to your commandline

isuftin commented 7 years ago

@gschueler Adding the -Djava.io.tmpdir option fixes this issue. I wonder if this is something that should be in the documentation?

gschueler commented 7 years ago

@isuftin yes, should be in the docs. Also, perhaps Rundeck should test the value at startup and echo a warning if the temp dir is not valid/writeable.