search

rundeck-plugins / rundeck-s3-log-plugin

Store execution logs in Amazon S3 buckets
Apache License 2.0
32 stars 26 forks source link

job logs stopped being uploaded to S3 #7

Open makered opened 8 years ago

makered commented 8 years ago

I have a two node Rundeck cluster and using the S3 plugin. Recently I noticed that as of about 10 days ago logs stopped being delivered to S3 and one of the Rundeck nodes cannot access logs in the UI (logs are possibly buffered of the local filesystem of the other node).

Our users started scheduling more jobs lately, some running fairly frequently (every minute) and I am wondering if this might cause a inflow of execution logs that the S3 plugin cannot keep up with.

Once I restarted one of the nodes, it spends a long time (over 3 hrs) initially re-queueing logs with messages similar to:

2016-03-19 01:15:47,794 [main] INFO grails.app.services.rundeck.services.LogFileStorageService - re-queueing incomplete log storage request for execution 546710

I checked Rundeck logs and there do not seem to be any clues there as to why this is occurring. Are there any recommended steps I should follow to troubleshoot the plugin?

Thank you

lorenzoaiello commented 7 years ago

Any updates on this? We are running into the same issue.

gschueler commented 7 years ago

what rundeck version are you using?

lorenzoaiello commented 7 years ago

Rundeck 2.6.9-1

tkant commented 4 years ago

Hi @gschueler, wondering if we have any updates on this issue? A user of RUNDECK ENTERPRISE 3.1.2-20190927 version.