Issue retrieving s3 logs

[jennyfountain]

Here is my error: Log file could not be retrieved: Failed to move temp file to location: /var/lib/rundeck/logs/rundeck/project1/job/df753429-a4b0-449f-8d64-60657bb16fb0/logs/2274735.state.json (via plugin "org.rundeck.amazon-s3")

During my search, I do see the 2274735.log.state.json and 2274735.log.rdlog.

I moved to s3 since webdav did the same exact thing. This is getting quite frustrating. Here are the s3 config lines in my configs:

framework: framework.plugin.ExecutionFileStorage.org.rundeck.amazon-s3.AWSCredentialsFile=aws.file

name of the bucket

framework.plugin.ExecutionFileStorage.org.rundeck.amazon-s3.bucket=rundeck

path to store the logs

framework.plugin.ExecutionFileStorage.org.rundeck.amazon-s3.path=logs/${job.project}/${job.execid}

rundeck: rundeck.execution.logs.fileStoragePlugin=org.rundeck.amazon-s3

Can someone please tell me what may be the issue?

Thanks

[gschueler]

After retrieving the file, rundeck does "rename" to move it from a temp dir to the correct path. Based on the error message, I think it is failing at that point. This might fail if the temp dir is on a different filesystem perhaps, or Rundeck can't create the parent directories for some reason.

We should improve the way the file gets renamed. In the mean time you might try changing your java.io.tmpdir system property to point to a different temp path, or verify that rundeck should be able to create the parent directories if they do not exist

[jennyfountain]

Spoke too soon. One rundeck box works but the other does not.

-XX:MaxPermSize=256m -server -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Addresses -Duser.timezone=America/NewYork -Djava.io.tmpdir=/tmp/rundeck

[root@box rundeck]# ll /tmp/rundeck/ total 0 -rw-rw-r-- 1 rundeck rundeck 0 Feb 22 16:36 org.hibernate.cache.internal.StandardQueryCache.data -rw-rw-r-- 1 rundeck rundeck 0 Feb 22 16:36 org.hibernate.cache.spi.UpdateTimestampsCache.data

[gschueler]

we should file an issue to have rundeck rename the temp file in a more resilient way.

I'm not sure what your paste is showing: do you mean you set the tmpdir, but still have the problem?

[jennyfountain]

Yes, I am still having the issue - the tmp directory is created and it appears rundeck is using it but still seeing the issue.

[gschueler]

are the temp dir and the /var/lib/rundeck/logs dir on different file systems?

[jennyfountain]

They were - I just moved the tmp directory to the same file system and it seems to work. Thanks!

[gschueler]

:cool: I will file an issue in @rundeck/rundeck about fixing the "renameTo" to something better

[diwu1989]

hitting this issue too, would be great if we can have tmp directory in different FS

[jamieps]

rundeck/rundeck#1702 addressed via PR rundeck/rundeck#1866 should have resolved this, Rundeck no longer uses Files.renameTo

@diwu1989 what version of Rundeck are you using?

[silver8ack]

FYI, I'm having the same issue with rundexk 2.9 and the WebDAV-logstore. It's the same problem though. I changed the java.io.tmpdir to the same fs and it works fine.

[hjdr4]

The issue still exists using Rundeck 2.7.1.

[gschueler]

@hjdr4 can you paste your exact error message? is your tmp dir on the same filesystem?

[joker1007]

I have same issue too.

Log file could not be retrieved: Failed to move temp file to location: /var/lib/rundeck/logs/rundeck/MyProject/job/36b9f22d-6280-4b1f-be44-bf4341d8fb56/logs/654.state.json (via plugin "org.rundeck.amazon-s3")

I'm using Rundeck-2.7.2 and launch on Docker container. I ensured that /var/lib/rundeck/logs and java.io.tmpdir are on same filesystem. A container can read logs after the container executes job. But other new container cannot retrieve the log files from S3. Of course, the container is not read only.

[gschueler]

this should be fixed in Rundeck 2.8.3 and later