openshift / origin-server

OpenShift 2 (deprecated)
889 stars 516 forks source link

Make cron max run time configurable #6356

Closed sebastien-fauvel closed 8 years ago

sebastien-fauvel commented 8 years ago

It would be helpful to make the max run time of cron jobs configurable at the application level. Maybe by changing the access rights on ~/cron/configuration/limits. Currently, the timeout is 20 minutes for each bunch of cron jobs of a given frequency. For long cron jobs that run daily/weekly/monthly, this might be too short. It would be great to have the possibility to override the defaults at the application level, or maybe just to increase the timeout for daily/weekly/monthly cron jobs. Hard limits can already be set in /etc/openshift/cron/limits, but it looks like this isn't being used on OpenShift Online for now.

dinhxuanvu commented 8 years ago

Hi Sebastien,

Unfortunately, for OpenShift Online since it's multi-tenant environment, increasing the max_time for cron jobs is not supported at the moment as it may potentially impact other gears that are sharing the same environment in term of shared resource consumption. You can go to this link to see the full article (https://access.redhat.com/solutions/1326593).

For that reason, you may need to do some workaround to avoid reaching that limit. I found a post on stackoverflow (http://stackoverflow.com/questions/29435435/openshift-warning-weekly-cron-run-terminated-as-it-exceeded-max-run-time) and I think you can take a look to see if it's helpful for your situation.

Thanks, Vu

sebastien-fauvel commented 8 years ago

Thank you, that's already what I've done as a workaround and it works fine. I've moved the original cron scripts to .openshift/cron_nohup, and call them with nohup e.g. from .openshift/cron/daily/nohup.sh:

#!/bin/bash
nohup ${OPENSHIFT_REPO_DIR}/.openshift/cron_nohup/daily/script.sh < /dev/null &>> ${OPENSHIFT_LOG_DIR}/cron_daily_nohup.log &

It runs daily for 50 minutes without being killed and the logs are being put into a separate file.