kraiz / django-crontab

dead simple crontab powered job scheduling for django.
Other
846 stars 113 forks source link

django-crontab can't access environment variables used in settings.py #88

Open calcutec opened 6 years ago

calcutec commented 6 years ago

Environment & Versions

Settings

Details

no

The traceback is below. The issue is that crontab doesn't recognize the os.environ variables i use in settings.py for db connections, aws secret keys, etc. I don't know how to make these available to a cron job and hoped that django-crontab would have a baked-in solution.

The same function works fine when I am running it from the django project itself.

Thanks!

Message 18: From billburton@Jimmys-iMac.local Fri Aug 10 15:05:00 2018 X-Original-To: billburton Delivered-To: billburton@Jimmys-iMac.local From: billburton@Jimmys-iMac.local (Cron Daemon) To: billburton@Jimmys-iMac.local Subject: Cron billburton@Jimmys-iMac /Users/billburton/Projects/djangodigital/dpmvenv/bin/python /Users/billburton/Projects/djangodigital/manage.py crontab run 79da9099e73f903adeec54d63e2acd67 # django-cronjobs for djangodigital X-Cron-Env: X-Cron-Env: X-Cron-Env: X-Cron-Env: X-Cron-Env: Date: Fri, 10 Aug 2018 15:05:00 -0400 (EDT)

Traceback (most recent call last): File "/Users/billburton/Projects/djangodigital/manage.py", line 22, in execute_from_command_line(sys.argv) File "/Users/billburton/Projects/djangodigital/dpmvenv/lib/python3.6/site-packages/django/core/management/init.py", line 371, in execute_from_command_line utility.execute() File "/Users/billburton/Projects/djangodigital/dpmvenv/lib/python3.6/site-packages/django/core/management/init.py", line 317, in execute settings.INSTALLED_APPS File "/Users/billburton/Projects/djangodigital/dpmvenv/lib/python3.6/site-packages/django/conf/init.py", line 56, in getattr self._setup(name) File "/Users/billburton/Projects/djangodigital/dpmvenv/lib/python3.6/site-packages/django/conf/init.py", line 43, in _setup self._wrapped = Settings(settings_module) File "/Users/billburton/Projects/djangodigital/dpmvenv/lib/python3.6/site-packages/django/conf/init.py", line 106, in init mod = importlib.import_module(self.SETTINGS_MODULE) File "/Users/billburton/Projects/djangodigital/dpmvenv/lib/python3.6/importlib/init.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 994, in _gcd_import File "", line 971, in _find_and_load File "", line 955, in _find_and_load_unlocked File "", line 665, in _load_unlocked File "", line 678, in exec_module File "", line 219, in _call_with_frames_removed File "/Users/billburton/Projects/djangodigital/djangodigital/settings.py", line 94, in 'HOST': os.environ['RDS_DPM_MAIN_HOSTNAME'], File "/Users/billburton/Projects/djangodigital/dpmvenv/bin/../lib/python3.6/os.py", line 669, in getitem raise KeyError(key) from None KeyError: 'RDS_DPM_MAIN_HOSTNAME'

calcutec commented 6 years ago

Figured out how to inform the crontab of environment variables. Pointed it to the .bash_profile where they are set.

CRONJOBS = [ (' * source $HOME/.bash_profile;', 'publicity.cron.my_scheduled_job') ]

Works great

kraiz commented 6 years ago

Have a look at the README.There's an example for the setting CRONTAB_COMMAND_PREFIX. You should either define your needed values there or place them manually on the top of crontab via crontab -e.

calcutec commented 6 years ago

I glossed over that. Thanks.

Great project!

txemac commented 5 years ago

I solved it editing the crontab, add this at first line:

SHELL=/bin/bash

And change the line created by crontab library with the command:

* * * * * source /Users/billburton/Projects/djangodigital/dpmvenv/bin/activate && /Users/billburton/Projects/djangodigital/manage.py crontab run 79da9099e73f903adeec54d63e2acd67 # django-cronjobs for djangodigital
johnyoonh commented 4 years ago

Yeah, it seems like I have to add that line SHELL=/bin/bash manually in crontab. Setting it as a prefix doesn't make it work with or without export. One solution to wrap the command after specifying the shell in the settings.py https://unix.stackexchange.com/a/430478/16814

luzhongyang commented 4 years ago

If use in docker or k8s, SHELL=/bin/bash can not work, my solution is this:

  1. export all env in docker cmd or entrypoint.sh
    export > /opt/env
  2. load env on django-crontab
    CRONJOBS = [
    ('*/5 * * * * (source /opt/env || true) &&', 'app.crontab.crontab')
    ]
Gradd commented 4 years ago

None of the above worked for me, but it pointed me in the right direction.

carestad's answer in this link was what did the trick for me https://stackoverflow.com/questions/2229825/where-can-i-set-environment-variables-that-crontab-will-use

basically, the crontab doesn't have the env variables in scope, so to do that, we can output our env into the crontab, that'll make them available for the scripts that need to be executed

in my entrypoint.sh:

if [[ "$RUN_CRON" == *"YES"* ]]; then
  service rsyslog start
  echo "$(env ; crontab -l)" | crontab -
  /etc/init.d/cron start
  python /code/manage.py crontab add
fi

then in the settings.py

CRONJOBS = [
    ('0 0 * * *', 'app.management.commands.test_command.test_command', '>> /code/cron_nightly.logs')
]
yousufctec commented 3 years ago

@kraiz is right, you can use the CRONTAB_COMMAND_PREFIXin your settings.py as below.

envs = []
for envkey in os.environ.keys():
    envs.append(envkey + '=' + os.environ[envkey])
CRONTAB_COMMAND_PREFIX = 'env $(echo ' + '\n'.join(envs) + ' | xargs)'

This is working, given the environment variables were not modified/added during runtime.

Sajzad commented 3 years ago

If you are using Ubuntu >=18.0, try with following changes. I am assuming you have successfully integrated django-crontab with your django application. Now, access to your server via ssh and type "crontab -e", You will see your cronjobs are defined here. Add Your Environment Variables just above your cronjob and save the file using CTRL+o & CTRL+x. See the following piece of lines-

DB_NAME = "**" DB_USER = "**" DB_PASSWORD = "" SECRET_KEY = "***"

/1 * /home/sms/sms/env/bin/python /home/sms/sms/sms-backend/website/manage.py crontab run 446aeeebac8c8f82d9e01bb662dc9b21 >> /tmp/scheduled_job.log # django>

Nota Bene: For me, I have 4 variables, Add all your environment variables as it is mentioned and replace all asterisk alpahbet with your information.

cjaca commented 2 years ago

For docker/k8s case:

  1. Export all env set previously in docker cmd in entrypoint.sh or any start script to .env file:
    printenv > /app_folder/.env
  2. Install additional plugin python-dotenv==0.20.0 and add it to requirements.txt
    pip install python-dotenv
  3. Import python-dotenv in settings.py and import values:
    
    from pathlib import Path
    import os
    from dotenv import dotenv_values

BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(file))) ENV_FILE=str(Path(BASE_DIR)) + '/.env' myvars=dotenv_values(ENV_FILE)

4. Reuse .env values across all Django project:
```python
DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql',
        'NAME': myvars['POSTGRES_NAME'],
        'USER': myvars['POSTGRES_USER'],
        'PASSWORD': myvars['POSTGRES_PASSWORD'],
        'HOST': myvars['POSTGRES_HOST'],
        'PORT': 5432,
    }
}
  1. No changes required in CRONJOBS
ayudmin commented 2 years ago

Had a similar issue with a project but of course i figured it out just about the environment variables you just need to add the CRONTAB_COMMAND_PREFIX before the list of your crontabs. OK basically that tells crontab where to find the environment it needs to work with ie:

CRONTAB_COMMAND_PREFIX = (
    'STAGE={whether production or development}'
)

CRONJOBS = [
    ('{time for cron}', '{your crontab view}'),

]
Sajzad commented 2 years ago

Solved. Thank you very much for your reply.

On Tue, 20 Sep 2022, 7:13 pm Ayume Francis, @.***> wrote:

Had a similar issue with a project but of course i figured it out just about the environment variables you just need to add the CRONTAB_COMMAND_PREFIX before the list of your crontabs. OK basically that tells crontab where to find the environment it needs to work with ie:

`CRONTAB_COMMAND_PREFIX = ( 'STAGE={whether production or development}' )

CRONJOBS = [ (' *', '{your crontab view}'),

]`

— Reply to this email directly, view it on GitHub https://github.com/kraiz/django-crontab/issues/88#issuecomment-1252336107, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKECZZLT7HWOBDVVQ5YEVMTV7GZ7XANCNFSM4FPCXTDA . You are receiving this because you commented.Message ID: @.***>

cesarbonadio commented 1 year ago

For docker/k8s case:

  1. Export all env set previously in docker cmd in entrypoint.sh or any start script to .env file:
printenv > /app_folder/.env
  1. Install additional plugin python-dotenv==0.20.0 and add it to requirements.txt
pip install python-dotenv
  1. Import python-dotenv in settings.py and import values:
from pathlib import Path
import os
from dotenv import dotenv_values

BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
ENV_FILE=str(Path(BASE_DIR)) + '/.env'
myvars=dotenv_values(ENV_FILE)
  1. Reuse .env values across all Django project:
DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql',
        'NAME': myvars['POSTGRES_NAME'],
        'USER': myvars['POSTGRES_USER'],
        'PASSWORD': myvars['POSTGRES_PASSWORD'],
        'HOST': myvars['POSTGRES_HOST'],
        'PORT': 5432,
    }
}
  1. No changes required in CRONJOBS

This a good approach and working solution for more complex variables. In my case, I don't use docker but I needed to pass the database configurations, which happens to work unexpectedly using the default approach of CRONTAB_COMMAND_PREFIX. This approach worked for me

yangwe1 commented 3 months ago

echo "$(env ; crontab -l)" | crontab -

I used this command in the docker console, and it really saved my day.