search

datamade / django-councilmatic

:heartpulse: Django app providing core functions for *.councilmatic.org
http://councilmatic.org
MIT License
26 stars 16 forks source link

Remove update and send commands from import #169

Closed reginafcompton closed 6 years ago

reginafcompton commented 6 years ago

This PR handles issue #168.

What does this mean for Councilmatics that use notifications?

Cron needs to execute two commands: (1) python manage.py update_index (note: each Councilmatic may have different args, but we do not want to hard code the age=24 arg, as we did previously) (2) python manage.py send_notifications

These commands should fire in sequence. So, for NYC Councilmatic we'd have:

10,25,40,55 * * * * datamade /usr/bin/flock -n /tmp/nyc_dataload.lock -c 'cd /home/datamade/nyc-council-councilmatic && /home/datamade/.virtualenvs/nyc-council-councilmatic/bin/python manage.py import_data >> /tmp/nyc-council-councilmatic-loaddata.log 2>&1 && /home/datamade/.virtualenvs/nyc-council-councilmatic/bin/python manage.py update_index --batch-size=50 >> /tmp/nyc-council-councilmatic-updateindex.log 2>&1 && /home/datamade/.virtualenvs/nyc-council-councilmatic/bin/python manage.py update_index --batch-size=50 >> /tmp/nyc-council-councilmatic-updateindex.log 2>&1 && /home/datamade/.virtualenvs/nyc-council-councilmatic/bin/python manage.py convert_rtf >> /tmp/convert_rtf.log 2>&1'

hancush commented 6 years ago

I wonder if it'd be worth setting these long commands as variables (I read you can't set arbitrary variables in a crontab, but you can use environmental variables) or alias them, so they're a bit more readable. I don't have initial thoughts about how best to segment them, but just a thought on readability.

reginafcompton commented 6 years ago

I did manage to set arbitrary variables...per a 2014 discussion and man 5 cron. It works with a test cron. So, something like this should execute as expected: 

# /etc/cron.d/nyc-council-councilmatic-crontasks
APPDIR=/home/datamade/nyc-council-councilmatic
PYTHONDIR=/home/datamade/.virtualenvs/nyc-council-councilmatic/bin/python
10,25,40,55 * * * * datamade /usr/bin/flock -n /tmp/nyc_dataload.lock -c 'cd $APPDIR && $PYTHONDIR manage.py import_data >> /tmp/nyc-council-councilmatic-loaddata.log 2>&1 && $PYTHONDIR manage.py update_index --batch-size=50 >> /tmp/nyc-council-councilmatic-updateindex.log 2>&1 && $PYTHONDIR manage.py send_notifications >> /tmp/nyc-council-councilmatic-sendnotifications.log 2>&1 && $PYTHONDIR manage.py convert_rtf >> /tmp/convert_rtf.log 2>&1'